Log analysis method, device, equipment, storage medium and program product

Abstract

This application discloses a log parsing method, apparatus, device, storage medium, and program product. The method includes: acquiring multiple logs to be parsed; replacing fields in each log that match a preset field type with a field type identifier corresponding to the preset field type to obtain a string corresponding to each log; hashing and binning the multiple logs based on the string corresponding to each log to be parsed to obtain at least one log bucket; each log bucket includes at least one log to be parsed, and the strings corresponding to each log to be parsed in the same log bucket are the same; inputting at least one log to be parsed from each log bucket into a large language model, so that the large language model performs comparative parsing processing on at least one log to be parsed in each log bucket to obtain a log template corresponding to each log to be parsed in each log bucket. Using this application embodiment can improve log parsing efficiency and the accuracy of log template extraction.

Description

Technical Field

[0001] This application relates to the field of Internet technology, and in particular to a log parsing method, apparatus, device, storage medium, and program product. Background Technology

[0002] Log parsing refers to the process of extracting fixed templates from unstructured log text, identifying unchanging parts as constants and changing parts as variables. Currently, log parsing methods are mainly divided into two categories: rule-based or clustering-based methods, and large language model-based methods. Rule-based or clustering-based methods (such as Drain3) group logs and extract templates through predefined rules or text similarity calculations. In scenarios with massive log volumes, these methods require numerous pairwise similarity comparisons, resulting in high computational complexity and low parsing efficiency. Large language model-based methods input logs into a large language model for semantic parsing and template extraction. However, when faced with a single log entry, the large language model struggles to determine whether a field is a variable or a constant, leading to ambiguity in variable identification and affecting the accuracy of log template extraction. Summary of the Invention

[0003] The technical problem to be solved by the embodiments of this application is to provide a log parsing method, apparatus, device, storage medium and program product that can improve log parsing efficiency and log template extraction accuracy.

[0004] On the one hand, embodiments of this application provide a log parsing method, the method including: Retrieve multiple log entries to be parsed; Replace the field in each log to be parsed that matches the preset field type with the field type identifier corresponding to the preset field type to obtain the string corresponding to each log to be parsed; Based on the string corresponding to each log to be parsed, the multiple logs to be parsed are hashed and bucketed to obtain at least one log bucket; a log bucket includes at least one log to be parsed, and the strings corresponding to each log to be parsed in the same log bucket are the same; At least one log entry to be parsed from each log bucket is input into the large language model, so that the large language model performs comparative parsing processing on at least one log entry to be parsed in each log bucket to obtain the log template corresponding to each log entry to be parsed in each log bucket.

[0005] On the other hand, embodiments of this application provide a log parsing apparatus, which includes: The acquisition unit is used to retrieve multiple log entries to be parsed. The replacement unit is used to replace the field in each log to be parsed that matches the preset field type with the field type identifier corresponding to the preset field type, so as to obtain the string corresponding to each log to be parsed; The processing unit is used to perform hash bucketing on the multiple logs to be parsed based on the string corresponding to each log to be parsed, to obtain at least one log bucket; a log bucket includes at least one log to be parsed, and the strings corresponding to each log to be parsed in the same log bucket are the same; The processing unit is further configured to input at least one log to be parsed from each log bucket into the large language model, so that the large language model performs comparative parsing processing on at least one log to be parsed in each log bucket to obtain a log template corresponding to each log to be parsed in each log bucket.

[0006] In one implementation, the processing unit performs hash bucketing on the multiple logs to be parsed based on the string corresponding to each log to be parsed, and after obtaining at least one log bucket, it can also be used for: The target log to be parsed is determined from each log bucket, and the semantic vector of each target log is encoded to obtain the semantic vector of each target log; the target log is determined based on the field length and / or the proportion of variables. Based on the semantic vector of each target log to be parsed, the semantic similarity between each pair of target logs to be parsed is determined, and the log buckets corresponding to the target logs whose semantic similarity reaches the semantic similarity threshold are merged to obtain at least one log group. The processing unit inputs at least one log entry to be parsed from each log bucket into the large language model, so that the large language model performs comparative parsing processing on at least one log entry to be parsed in each log bucket, obtaining a log template corresponding to each log entry to be parsed in each log bucket, which can be used for: Multiple logs to be parsed from each log group are input into the large language model, so that the large language model performs comparative parsing processing on the multiple logs to be parsed from each log group to obtain the log template corresponding to each log to be parsed from each log group.

[0007] In one implementation, the processing unit performs semantic vector encoding on each target log to be parsed, obtaining a semantic vector for each target log to be parsed, which can be used for: Semantic segmentation is performed on any target log to be parsed to obtain multiple semantic segments corresponding to the target log to be parsed; Determine the semantic segment weight of each semantic segment corresponding to any target log to be parsed; the semantic segment weight is determined based on at least one of the following: semantic segment position, semantic segment length, and semantic segment keywords; Each semantic segment is semantically vector-encoded to obtain a semantic segment vector corresponding to each semantic segment; Based on the semantic segment vector corresponding to each semantic segment and the semantic segment weight of each semantic segment, the semantic vector of any target log to be parsed is obtained.

[0008] In one implementation, the replacement unit replaces the fields in each log entry to be parsed that match a preset field type with the field type identifier corresponding to the preset field type, obtaining a string corresponding to each log entry to be parsed, which can be used for: Each log entry to be parsed is semantically segmented to obtain multiple semantic segments corresponding to each log entry to be parsed; For any log to be parsed, a target semantic segment is determined from the multiple semantic segments of the log to be parsed. Replace the field in the target semantic segment corresponding to each log to be parsed that matches the preset field type with the field type identifier corresponding to the preset field type to obtain the string corresponding to each target semantic segment; The processing unit performs hash bucketing on the multiple log entries to be parsed based on the string corresponding to each log entry, obtaining at least one log bucket, which can be used for: Based on the string corresponding to each target semantic segment, the multiple logs to be parsed are hashed and bucketed to obtain at least one log bucket.

[0009] In one implementation, the processing unit performs hash bucketing on the multiple logs to be parsed based on the string corresponding to each log to be parsed, and after obtaining at least one log bucket, it can also be used for: Based on the number of logs to be parsed in each log bucket, the logs to be parsed in each log bucket are clustered to obtain at least one log unit bucket corresponding to each log bucket. The processing unit inputs at least one log entry to be parsed from each log bucket into the large language model, so that the large language model performs comparative parsing processing on at least one log entry to be parsed in each log bucket, obtaining a log template corresponding to each log entry to be parsed in each log bucket, which can be used for: At least one log entry to be parsed from each log unit bucket is input into the large language model, so that the large language model performs comparative parsing processing on at least one log entry to be parsed from each log unit bucket to obtain the log template corresponding to each log entry to be parsed from each log unit bucket.

[0010] In one implementation, the processing unit inputs at least one log entry to be parsed from each log bucket into a large language model, so that the large language model performs comparative parsing processing on at least one log entry to be parsed in each log bucket to obtain a log template corresponding to each log entry to be parsed in each log bucket, which can be used for: Select any log to be parsed from each log bucket, and determine the semantic vector of any log to be parsed corresponding to each log bucket; Based on the semantic vector of any log to be parsed corresponding to each log bucket, semantic similarity retrieval is performed in the prompt word database to obtain at least one historical log corresponding to each log bucket and a historical log template corresponding to the at least one historical log. The at least one historical log corresponding to each log bucket and the historical log template corresponding to the at least one historical log are used as log samples. At least one log entry to be parsed from each log bucket, and a log sample corresponding to each log bucket, are input into the large language model. The large language model then performs comparative parsing on at least one log entry to be parsed in each log bucket based on the log sample corresponding to each log bucket, thereby obtaining a log template corresponding to each log entry to be parsed in each log bucket.

[0011] In one implementation, before the processing unit inputs at least one log entry to be parsed from each log bucket into the large language model, it can also be used for: The target log to be parsed is determined from each log bucket, and the constant field sequence corresponding to each target log is extracted; the constant field sequence is composed of fields in the target log that do not match the preset field type; Query the log template library to see if there is a historical log template that matches the sequence of constant fields corresponding to any target log to be parsed; If a historical log template exists that matches the sequence of constant fields corresponding to any of the target logs to be parsed, then the historical log template that matches the sequence of constant fields corresponding to any of the target logs to be parsed will be used as the log template for each log to be parsed in the log bucket corresponding to any of the target logs to be parsed. If no historical log template matches the sequence of constant fields corresponding to any of the target logs to be parsed, then the step of inputting at least one log from each log bucket into the large language model is performed.

[0012] In one embodiment, the processing unit can also be used for: The historical parsing logs, the corresponding historical log templates, and the semantic vectors of the historical parsing logs are associated and stored in the prompt word database; and / or, Extract the constant field sequence corresponding to the historical parsing log, and associate and store the constant field sequence corresponding to the historical parsing log and the historical log template corresponding to the historical parsing log in the log template library.

[0013] On the other hand, embodiments of this application provide a computer device, which includes a memory, a communication interface, and a processor, wherein the memory, the communication interface, and the processor are interconnected; the memory stores a computer program, and the processor calls the computer program stored in the memory to implement the above-mentioned log parsing method.

[0014] On the other hand, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described log parsing method.

[0015] On the other hand, embodiments of this application provide a computer program product, which includes a computer program stored in a computer storage medium; the processor of a computer device reads the computer program from the computer storage medium and executes the computer program, causing the computer device to perform the above-described log parsing method.

[0016] In this embodiment, multiple logs to be parsed are obtained; the fields in each log that match a preset field type are replaced with field type identifiers corresponding to the preset field type, resulting in a string corresponding to each log; based on the string corresponding to each log, the multiple logs are hash-bucketed to obtain at least one log bucket; each log bucket includes at least one log to be parsed, and the strings corresponding to each log in the same log bucket are identical; at least one log in each log bucket is input into a large language model, so that the large language model performs comparative parsing processing on at least one log in each log bucket, resulting in a log template corresponding to each log in each log bucket. By replacing the fields in each log that match a preset field type with field type identifiers corresponding to the preset field type, and performing hash-bucketing based on the replacement results, the complexity of log grouping is reduced, and the log parsing efficiency is improved; furthermore, by inputting at least one log in the same log bucket into the large language model for comparative parsing processing, the large language model can accurately identify variables and constants by utilizing the differences between the logs in each bucket, thus improving the accuracy of log template extraction. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the background art, the accompanying drawings used in the embodiments of this application or the background art will be described below.

[0018] Figure 1This is a logical schematic diagram of a log parsing method provided in an embodiment of this application; Figure 2 This is a flowchart illustrating a log parsing method provided in an embodiment of this application; Figure 3 This is a flowchart illustrating another log parsing method provided in an embodiment of this application; Figure 4 This is a flowchart illustrating another log parsing method provided in an embodiment of this application; Figure 5 This is a flowchart illustrating another log parsing method provided in an embodiment of this application; Figure 6 This is a schematic diagram of the structure of a log parsing device provided in an embodiment of this application; Figure 7 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0019] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0020] Furthermore, in the description of the embodiments of this application, the terms "first," "second," etc., are used only for distinguishing descriptions and should not be construed as indicating or implying relative importance. Therefore, a feature specified as "first" or "second" may explicitly or implicitly include one or more features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0021] In the specific implementation of this application, user-related data, such as user logs, are involved. When the embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with local laws, regulations and standards.

[0022] Big data computing platforms (such as Flink, Ray, and StarRocks) generate massive amounts of unstructured logs during operation. These logs are characterized by: enormous volume, with a single task failure generating tens of thousands of lines of logs; high repetition, with the same type of log event appearing repeatedly on a large number of nodes, for example, hundreds of worker nodes generating the same error pattern; and variable format, with different versions and components having different log formats, making it difficult to cover them with fixed rules.

[0023] When these logs are fed into the Large Language Model (LLM) for analysis, two core issues are faced: First, the volume problem, with tens of thousands of logs far exceeding the context window limit of a single Large Language Model, requiring compression; second, the cost problem, with the latency and application programming interface (API) call costs being unacceptable if the Large Language Model is called to parse each log.

[0024] Log parsing refers to the process of extracting a fixed template from unstructured log text, identifying the unchanging parts of the log as constants and the changing parts as variables. The goal of log template extraction is to group similar logs into a single template, where the unchanging parts (constants) are preserved in their original form, and the changing parts (variables) are replaced with placeholders <*>.

[0025] For example, the following three log entries: "Container [task_001] killed by the system:exitCode=137", "Container [task_002] killed by the system: exitCode=137", "Container [task_003] killed by the system: exitCode=143".

[0026] After log parsing, the template "Container [<*>] killed by the system:exitCode=<*>" can be extracted. Here, "Container [" and "] killed by the system:exitCode=" are constants that remain unchanged across similar logs; the task identifiers (task_001, task_002, task_003) and exit codes (137, 137, 143) are variables that may change across similar logs, so they are replaced with placeholders <*>. By merging similar logs into a single template, log volume can be significantly reduced, providing a structured data foundation for subsequent log analysis, anomaly detection, and other tasks.

[0027] Currently, log parsing methods are mainly divided into two categories: rule-based or clustering-based methods, and methods based on large language models.

[0028] Rule-based or clustering-based methods (such as Drain3) group logs and extract templates through predefined rules or text similarity calculations. In scenarios with massive log volumes, these methods require numerous pairwise similarity comparisons, resulting in high computational complexity and low parsing efficiency. Furthermore, rule-based methods rely on manually maintained rule bases, making it difficult to adapt to iterative changes in log formats, leading to a significant decrease in parsing accuracy for new log formats.

[0029] The large language model-based approach inputs logs into a large language model for semantic parsing and template extraction, demonstrating strong semantic understanding capabilities. However, when faced with a single log entry, the large language model struggles to determine whether a field is a variable or a constant due to the lack of comparative references, leading to ambiguity in variable identification and impacting the accuracy of log template extraction. Furthermore, parsing each log entry using the large language model incurs high inference costs in large-scale log scenarios, making it difficult to meet the requirements of online real-time parsing in terms of processing efficiency.

[0030] Based on this, this application provides an adaptive evolution log parsing method (Adaptive Parser, AdaParse) based on a large language model. Specifically, multiple logs to be parsed can be obtained; the fields in each log that match a preset field type are replaced with field type identifiers corresponding to the preset field type to obtain a string corresponding to each log; based on the string corresponding to each log, the multiple logs to be parsed are hashed and bucketed to obtain at least one log bucket; each log bucket includes at least one log to be parsed, and the strings corresponding to each log to be parsed in the same log bucket are the same; at least one log to be parsed in each log bucket is input into a large language model, so that the large language model performs comparative parsing processing on at least one log to be parsed in each log bucket to obtain a log template corresponding to each log to be parsed in each log bucket. By replacing the fields in each log entry that match the preset field type with the corresponding field type identifier, and performing hash bucketing based on the replacement results, the complexity of log grouping is reduced and the efficiency of log parsing is improved. Furthermore, by inputting at least one log entry from the same log bucket into the large language model for comparison and parsing, the large language model can accurately identify variables and constants by utilizing the differences between the log entries in the bucket, thereby improving the accuracy of log template extraction.

[0031] Based on the above description, please refer to Figure 1 , Figure 1 This is a logical diagram illustrating a log parsing method provided in an embodiment of this application. Specifically: Logs to be parsed. Big data computing platforms generate numerous logs to be parsed during operation. These logs are typically unstructured text, characterized by their massive volume, high repetition, and varied formats. For example, a single task failure can generate tens of thousands of log lines; the same type of log event may repeatedly occur on a large number of nodes; and different versions and components may have different log formats.

[0032] Multi-stage adaptive clustering (Log Grouper). This process performs multi-stage clustering on multiple input log entries to group structurally similar logs into the same group, providing a comparative context for subsequent steps. Specifically, this multi-stage clustering includes: (1) Structured semantic segmentation: The long log is divided into multiple semantic segments according to the preset structural boundaries, and each semantic segment is assigned a semantic segment weight so that subsequent processing can perceive the structural hierarchy of the log.

[0033] (2) String hash-based bucketing: Replace the fields in each log to be parsed that match the preset field type with the corresponding field type identifier to obtain the string corresponding to each log to be parsed; based on the string corresponding to each log to be parsed, perform hash bucketing on multiple logs to be parsed to obtain at least one log bucket, achieving coarse-grained clustering with O(1) complexity. A log bucket includes at least one log to be parsed, and the strings corresponding to each log to be parsed in the same log bucket are the same.

[0034] (3) Adaptive fine clustering: Within each hash bucket, a clustering strategy is adaptively selected based on the number of logs in the bucket to further subdivide the data, resulting in at least one log unit bucket corresponding to each log bucket.

[0035] (4) Semantic vector cluster merging: Determine the target log to be parsed from each log bucket, and encode the semantic vector of each target log to be parsed to obtain the semantic vector of each target log to be parsed; Based on the semantic vector of each target log to be parsed, determine the semantic similarity between each two target logs to be parsed, and merge the log buckets corresponding to the target logs to be parsed whose semantic similarity reaches the semantic similarity threshold to obtain at least one log group.

[0036] After the above four stages, the logs to be parsed within the same log group remain consistent in the constant part, but show differences in the variable part, providing a basis for comparison for template inference of the subsequent large language model.

[0037] Local cache hit check (Log Hitter). Before inputting the logs to be parsed into the large language model, the target logs to be parsed are determined from each log group, and the constant field sequence corresponding to each target log is extracted. The log template library is then queried to see if there is a historical log template that matches the constant field sequence corresponding to any target log. If a match is found, the historical log template is used as the log template for each log in the log group corresponding to the target log, and the parsing result is directly output without calling the large language model. If no match is found, the logs to be parsed in the current group are determined to be a new log pattern, and subsequent processing is performed.

[0038] Prompt Database Retrieval. For log groups that do not hit the cache, select any log to be parsed from each log group and determine the semantic vector of that log. Based on the semantic vector of that log, perform semantic similarity retrieval in the prompt database to obtain at least one historical log and its corresponding historical log template, which serve as log samples.

[0039] Joint template inference within a large language model group. Multiple log entries to be parsed from the same log group, along with retrieved log samples, are input into the large language model. Based on this log sample, the large language model performs comparative parsing by utilizing the differences between the log entries within the same log group, identifying which fields are variables and which are constants, and generating the corresponding log template for that log group.

[0040] Self-Evolution: Extract the constant field sequence corresponding to the target log to be parsed, associate this constant field sequence with the newly generated log template and store it in the log template library; associate and store multiple logs to be parsed in the log group, the newly generated log template, and the semantic vectors of these multiple logs to be parsed in the prompt word database.

[0041] Finally, the output includes the log template string, variable value list, and hit count for each log group, completing the parsing of the input logs.

[0042] Based on the above description, please refer to Figure 2 , Figure 2 This is a flowchart illustrating a log parsing method provided in an embodiment of this application, as shown below. Figure 2 The log parsing method shown includes, but is not limited to, steps S201~S204, wherein: S201, retrieve multiple logs to be parsed.

[0043] Unparsed logs refer to unstructured log text generated during the operation of a big data computing platform that has not yet undergone template extraction processing. These unparsed logs are characterized by: massive volume (a single task failure can generate tens of thousands of lines of logs); high repetition (the same type of log event appears repeatedly on a large number of nodes, for example, hundreds of worker nodes generate the same pattern of errors); and variable format (different versions and components have different log formats, making it difficult to cover them with fixed rules).

[0044] For example, the logs to be parsed mentioned above can be the raw logs output by big data computing platforms such as Flink, Ray, and StarRocks during task execution. For example, the following is a partial log of a Flink task failure: "2026-04-02 18:08:11,203 WARN Task - Container [baichuan-353-24191-taskmanager-1-0] killed by the system: exitCode=137", "2026-04-02 18:08:11,451 WARN Task - Container [baichuan-353-24191-taskmanager-1-1] killed by the system: exitCode=137", "2026-04-02 18:08:12,103 WARN Task - Container[baichuan-353-24191-taskmanager-1-2] killed by the system: exitCode=143".

[0045] In the example above, each log entry to be parsed contains a timestamp, log level, component name, and specific log message content. Fields such as timestamp (e.g., "2026-04-02 18:08:11,203"), task identifier (e.g., "taskmanager-1-0"), and exit code (e.g., "137") may change in different log instances and are considered variables in the log. Fields such as log level (e.g., "WARN"), component name (e.g., "Task"), and fixed log message structure (e.g., "Container [ ] killed by the system: exitCode=") remain unchanged in the same type of log and are considered constants in the log.

[0046] S202, replace the fields in each log to be parsed that match the preset field type with the field type identifier corresponding to the preset field type, and obtain the string corresponding to each log to be parsed.

[0047] The preset field type refers to the pre-defined field type used to identify variable fields in the log to be parsed. In practical applications, the log to be parsed contains a large number of fields with different specific values, such as IP addresses, numbers, universally unique identifiers, timestamps, and file paths. These fields have different values ​​in different instances of the same type of log, and belong to the variable part of the log. If the raw log text is directly used for subsequent grouping processing, the differences in the values ​​of these variable fields will cause logs that originally belonged to the same template to be scattered into different groups, affecting the accuracy of grouping.

[0048] Therefore, this embodiment of the application can eliminate the impact of differences in variable field values ​​on grouping by replacing variable fields with a unified field type identifier. Specifically, for each log entry to be parsed, each field is checked to see if it matches a preset field type; if it matches, the field is replaced with the field type identifier corresponding to that preset field type. The field type identifier is a unified mark used to represent a certain type of field, and all fields of the same preset field type are replaced with the same field type identifier.

[0049] For example, the correspondence between preset field types and field type identifiers can be shown in Table (1) below: Table (1)

[0050] After the above replacement process, the fields belonging to variables in each log entry to be parsed are uniformly replaced with the corresponding field type identifiers, while fields in the log that do not belong to any preset field type (i.e., the constant parts of the log, such as log level, component name, fixed log message structure, etc.) are retained. Therefore, each log entry to be parsed generates a corresponding string, which is composed of the retained constant fields and the replaced field type identifiers concatenated in the original log order.

[0051] Taking the log to be parsed, “Lost connection to JobManager akka: / / flink@10.92.80.57:6123. Reconnecting…”, as an example, “10.92.80.57” matches the preset field type “IP address” and is replaced with the field type identifier “ <ip>"6123" matches the default field type "Number" and is replaced with the field type identifier. <num>The remaining fields, such as "Lost", "connection", "to", "JobManager", "akka", "flink", and "Reconnecting", do not belong to any preset field type and are retained. The resulting string after replacement is "Lostconnection to JobManager akka flink". <ip> <num>Reconnecting.

[0052] For multiple logs that belong to the same log template but differ only in variable values, after the above replacement process, the generated strings for each log will be the same.

[0053] Optionally, the specific form of the field type identifier is not limited to the angle bracket format in the example above. Other identifiers that can distinguish different preset field types can also be used, such as numeric codes, specific prefix markers, etc. The specific types and number of preset field types can also be configured according to the characteristics of the variable fields in the logs to be parsed in the actual application scenario. For example, it can also include field types such as email addresses and Uniform Resource Locators.

[0054] In one optional implementation, each log to be parsed can be semantically segmented to obtain multiple semantic segments corresponding to each log to be parsed; for any log to be parsed, a target semantic segment is determined from the multiple semantic segments of any log to be parsed; the fields in the target semantic segment corresponding to each log to be parsed that match the preset field type are replaced with the field type identifier corresponding to the preset field type to obtain the string corresponding to each target semantic segment.

[0055] Specifically, each log entry to be parsed can be semantically segmented according to preset structural boundaries, resulting in multiple semantic segments corresponding to each log entry. The preset structural boundaries refer to the identifiers used to divide different structural levels in the log text. These preset structural boundaries can include at least one of the following: line breaks, indentation levels, stack frame identifiers, JSON nesting levels, and XML nesting levels. For example, for Java stack logs, stack frame identifiers can be prefixes such as "at org.apache." or "atcom.xhs."; for Python stack logs, stack frame identifiers can be prefixes such as "File". After semantic segmentation, the originally continuous long text of a log entry is split into multiple structurally independent semantic segments, each corresponding to a structural level in the log. For example, an exception log containing a Java stack can be segmented into the following three semantic segments: the first semantic segment contains exception type and level information, the middle semantic segment contains stack frame information, and the last semantic segment contains root cause description information.

[0056] After semantic segmentation is completed, for any log to be parsed, the target semantic segment is determined from the multiple semantic segments. The target semantic segment is the semantic segment used to perform subsequent field replacement and hash bucketing operations among the multiple semantic segments. Optionally, the target semantic segment can be the semantic segment that is at the beginning of any log to be parsed (i.e., the first line semantic segment). The first line semantic segment usually contains discriminative information such as log level and exception type, and it is the fastest and most direct basis for distinguishing different categories of logs. In other possible implementations, the target semantic segment can also be a semantic segment in other structural positions according to the actual application scenario, such as selecting the last line semantic segment containing root cause descriptions.

[0057] After determining the target semantic segment, the fields in the target semantic segment corresponding to each log entry to be parsed that match the preset field type are replaced with the field type identifier corresponding to the preset field type, resulting in a string corresponding to each target semantic segment. This replacement process is consistent with the field replacement process described above, except that the replacement scope is narrowed down from the entire log entry to the target semantic segment. For example, for an exception log containing a Java stack trace, if the target semantic segment is the first line semantic segment, only the variable fields in the first line semantic segment are replaced; the fields in the middle stack frame segments and the last line root cause description segment are not involved in the replacement or subsequent hash bucketing operations.

[0058] S203, based on the string corresponding to each log to be parsed, perform hash bucketing on multiple logs to be parsed to obtain at least one log bucket; a log bucket includes at least one log to be parsed, and the strings corresponding to each log to be parsed in the same log bucket are the same.

[0059] Each log entry to be parsed corresponds to a string. The string for any log entry is constructed by concatenating the retained constant fields and the replaced field type identifiers in the original log order. For multiple log entries that differ only in variable values, their corresponding strings are identical.

[0060] Hash bucketing refers to a data processing method that uses the string corresponding to each log entry to be parsed as input to a hash function to calculate a hash value, and groups log entries with the same hash value into the same log bucket. Specifically, for each log entry to be parsed, its corresponding string is input into a preset hash function to obtain a hash value; this hash value is used as an index to assign the log entry to the corresponding log bucket. If a log bucket corresponding to the hash value does not yet exist, a new log bucket is created and the log entry is placed in it; if a log bucket corresponding to the hash value already exists, the log entry is appended to that log bucket. Thus, after hash bucketing, at least one log bucket is obtained, each log bucket contains at least one log entry to be parsed, and all log entries within the same log bucket have the same string corresponding to them.

[0061] In this embodiment, log grouping is achieved through hash bucketing. Each log entry only requires one hash calculation to determine its bucket, eliminating the need for individual similarity comparisons with other log entries. Therefore, the time complexity of log grouping is O(n), where n is the total number of logs to be parsed. Compared to clustering methods based on pairwise text similarity comparisons (whose time complexity is typically O(n²)), this improves grouping efficiency and meets the requirements for online real-time processing in large-scale log scenarios. Furthermore, since the field replacement step uniformly replaces variable fields with field type identifiers, logs with the same structure but different variable values ​​generate the same string, thus accurately being assigned to the same log bucket, balancing efficiency and accuracy in grouping.

[0062] In one alternative implementation, multiple log entries to be parsed can be hashed and bucketed based on the string corresponding to each target semantic segment to obtain at least one log bucket.

[0063] Specifically, in S202, each log entry to be parsed has been semantically segmented. Target semantic segments are determined from the multiple semantic segments of each log entry, and fields in the target semantic segments that match preset field types are replaced with their corresponding field type identifiers, resulting in a string corresponding to each target semantic segment. Based on this, the string used for hash bucketing in this step is the string corresponding to each of these target semantic segments, not the string generated after field replacement of the entire log entry to be parsed.

[0064] Based on the string corresponding to each target semantic segment, multiple log entries to be parsed are hash-bucketed to obtain at least one log bucket. The specific implementation of hash bucketing is consistent with the method of hash bucketing based on the entire log string described above, that is: the string corresponding to each target semantic segment is used as the input of the hash function to calculate the hash value; the hash value is used as an index to allocate the corresponding log entry to the appropriate log bucket; log entries to be parsed with the same hash value (i.e., the strings corresponding to the target semantic segments are the same) are grouped into the same log bucket.

[0065] For example, consider the following two log entries to be parsed: 1. "[ERROR] NullPointerException in UserService at com.xhs.UserService.getUser(UserService.java:42) at com.xhs.Controller.handle(Controller.java:18) Caused by: user_id is null".

[0066] 2. "[ERROR] NullPointerException in UserService at com.xhs.UserService.getUser(UserService.java:42) at com.xhs.Controller.handle(Controller.java:29) Caused by: user_id is null".

[0067] After semantic segmentation, each log entry to be parsed is divided into three semantic segments: the first semantic segment contains exception type and level information, the middle semantic segment contains stack frame information, and the last semantic segment contains root cause description information. If the target semantic segment is determined to be the first semantic segment, only the variable fields in that first semantic segment are replaced with field type identifiers. The first line of both log entries is "[ERROR] NullPointerException in UserService", where "UserService" is identified as a variable field and replaced with the corresponding field type identifier; the rest is retained, generating the string corresponding to the first semantic segment. Since the first line content of the two log entries is exactly the same, the first semantic segment string generated after field replacement is also the same. Therefore, during hash bucketing, the two log entries are grouped into the same log bucket.

[0068] In this embodiment, since the string used for hash bucketing comes only from the target semantic segment (such as the first line semantic segment), the string length is greatly reduced, the amount of input data for hash calculation is smaller, and the computational overhead is further reduced. Since the target semantic segment is usually selected from the most structurally distinctive part of the log (such as the log level and exception type of the first line), it can effectively avoid interference from other semantic segments in the log (such as differences in line numbers in the middle stack frame and differences in the specific variable values ​​in the last line) on the bucketing results, thereby improving the accuracy of bucketing.

[0069] S204. Input at least one log to be parsed from each log bucket into the large language model so that the large language model can perform comparison and parsing processing on at least one log to be parsed in each log bucket to obtain the log template corresponding to each log to be parsed in each log bucket.

[0070] In this process, logs with the same structure are grouped into the same log bucket after hash bucketing. Logs within the same bucket maintain consistency in their constant portions, differing only in their variable portions. At least one log from the same bucket is then input into the large language model. The model automatically identifies which fields are variables and which are constants by comparing the differences between the logs within the bucket, and generates a log template for each log in that bucket.

[0071] Large language models refer to neural network models with natural language understanding and reasoning capabilities, such as GPT, Claude, and Qwen. Large language models can understand the semantics of log text and determine the constant or variable attributes of each field based on the comparison relationships between multiple input log entries.

[0072] Specifically, for each log bucket, at least one log entry to be parsed from that bucket is input into the large language model. Optionally, multiple log entries to be parsed from the same log bucket can be input into the large language model together, allowing the large language model to perform comparative parsing processing on each log entry within that bucket. Comparative parsing processing refers to the large language model comparing multiple log entries to be parsed from the same log bucket when generating the log template, comparing whether the values ​​of the same field position are the same in each log entry: if the values ​​of the same field position are the same in all log entries, the field is identified as a constant and the original text is retained in the template; if the values ​​of the same field position differ in all log entries, the field is identified as a variable and replaced with a placeholder in the template.

[0073] For example, for a log bucket obtained after hashing, the bucket contains the following three logs to be parsed: "Container [task_001] killed by the system: exitCode=137", "Container [task_002] killed by the system: exitCode=137", "Container [task_003] killed by the system: exitCode=143".

[0074] After the three log entries to be parsed are input into the large language model, the model performs comparative parsing on them. Through this process, the field "Container [" is the same in all three log entries and is identified as a constant; the fields "task_001", "task_002", and "task_003" are different in all three log entries and are identified as variables; the field "] killed by the system: exitCode=" is the same in all three log entries and is identified as a constant; the fields "137", "137", and "143" differ in all three log entries (the first two are 137, the third is 143) and are identified as variables. Therefore, the large language model generates the log template for this log bucket as: "Container [<*>] killed by the system: exitCode=<*>".

[0075] This log template is the log template corresponding to each log entry to be parsed in this log bucket. The "<*>" in the template are variable placeholders, indicating that the position is a variable field in the original log, and its specific value can be different.

[0076] After generating the log template, the large language model outputs the log template. For each log entry to be parsed in the log bucket, the corresponding log template is the above template. Simultaneously, the specific field values ​​corresponding to the variable placeholders in the template for each log entry to be parsed can be output as a list of variable values ​​for subsequent processing.

[0077] In this embodiment, when only a single log entry is input to the large language model (e.g., only "Container [task_001] killed by the system: exitCode=137"), due to the lack of comparison, it is difficult to determine whether "task_001" is a dynamically changing task identifier or a fixed task name; it is also difficult to determine whether "137" is a fixed status code or a potentially changing exit code. However, by inputting multiple log entries from the same log bucket, the large language model can accurately determine the constant or variable attribute of a field by comparing the differences in values ​​at the same field position in each log entry, eliminating the ambiguity of variable identification during single log parsing and improving the accuracy of log template extraction. A single call to the large language model can process all log entries to be parsed in a log bucket, instead of calling the large language model separately for each log entry, thus processing more log entries with the same number of calls. In large-scale log scenarios, since a large number of logs belong to the same structural pattern (such as hundreds of worker nodes generating the same error pattern), they are classified into a limited number of log buckets after hashing. The number of calls to the large language model is much less than the total number of logs to be parsed, which improves log parsing efficiency and reduces inference costs and latency.

[0078] In one optional implementation, before inputting at least one log entry from each log bucket into the large language model, target log entries can be determined from each log bucket, and a sequence of constant fields corresponding to each target log entry can be extracted. The sequence of constant fields consists of fields in the target log entries that do not match a preset field type. A query is performed in the log template library to see if a historical log template matches the sequence of constant fields corresponding to any target log entry. If a historical log template matches the sequence of constant fields corresponding to any target log entry, it is used as the log template for each log entry in the corresponding log bucket. If no historical log template matches the sequence of constant fields corresponding to any target log entry, the step of inputting at least one log entry from each log bucket into the large language model is performed.

[0079] In this embodiment, a log template library can be maintained. This library can be persistently stored in CSV file format and loaded and organized as a memory dictionary during system runtime for fast querying. The log template library is a data set used to store the mapping relationship between constant field sequences of parsed logs and their corresponding log templates. Optionally, the log template library can be maintained locally, on a server, or in the cloud. The memory dictionary is a key-value pair storage structure, where the key is a constant field sequence and the value is the log template corresponding to that constant field sequence. The constant field sequence is an ordered sequence of fields retained after regularization of the original log text to be parsed. Variable fields belonging to a preset field type are removed, and constant fields in the log are retained. It is a highly stable structural fingerprint of the same type of log; the constant field sequence can also be called a skeleton token sequence. The log template is a complete template corresponding to the constant field sequence, inferred from the history of a large language model, where variable positions are identified by placeholders (such as <*>). The log template library is loaded from the CSV file into the memory dictionary all at once during system startup. Each time a new log bucket is obtained through the above hash bucketing, the target log to be parsed is determined from the log bucket, and the constant field sequence corresponding to the target log to be parsed is extracted as the query key. An exact match is then performed in the memory dictionary. The matching of the constant field sequence has a time complexity of O(1), which can complete the query in a finite amount of time.

[0080] Specifically, for each log bucket obtained through hash bucketing, a target log to be parsed is determined from each log bucket. The target log to be parsed refers to a single log entry selected from the log bucket that represents the overall log characteristics within that bucket. Optionally, the first log entry in the log bucket can be determined as the target log; alternatively, the target log can be determined based on field length and / or variable proportion. For a detailed implementation of determining the target log based on field length and / or variable proportion, please refer to the relevant description below, which will not be repeated here. After determining the target log, the constant field sequence corresponding to each target log is extracted. The constant field sequence is an ordered sequence composed of fields in the target log that do not match the preset field type. That is, during the above field replacement process, fields matching the preset field type are replaced with field type identifiers, while the constant field sequence is formed by concatenating those fields that were not replaced and retained their original text in the order they appeared in the original log. A sequence of constant fields is a highly stable structural fingerprint of the same type of log. For logs to be parsed that belong to the same log template, even if the values ​​of the variable fields are different, their sequence of constant fields will be the same.

[0081] After extracting the constant field sequence, the log template library is queried to see if a historical log template exists that matches the constant field sequence corresponding to any target log to be parsed. The log template library can store multiple historical parsing records, each containing a constant field sequence for a historical log and the corresponding log template. During the query, the extracted constant field sequence is used as the query key and matched against the log template library. If a historical record in the log template library has a constant field sequence that is exactly the same as the query key, it is considered a match, meaning a historical log template exists that matches the constant field sequence corresponding to any target log to be parsed.

[0082] If a historical log template matching the constant field sequence exists, it indicates that the logs in the current log bucket belong to a previously parsed log type, and there is no need to call the large language model again for repeated parsing. In this case, the matched historical log template is used as the log template for each log in the log bucket corresponding to the target log to be parsed, and the log template is directly output as the parsing result, skipping the subsequent vector retrieval and large language model call steps.

[0083] If no historical log template matches the sequence of constant fields, it means that the logs in the current log bucket belong to a new log pattern and have not yet been parsed. In this case, the step of inputting at least one log entry to be parsed from the log bucket into the large language model is executed, and the large language model performs comparison and parsing processing to generate a new log template.

[0084] For example, take a log bucket containing the following three logs to be parsed after hashing: "Container [task_001] killed by the system: exitCode=137", "Container [task_002] killed by the system: exitCode=137", "Container [task_003] killed by the system: exitCode=143".

[0085] The first log entry in the log bucket is identified as the target log to be parsed, and its constant field sequence is extracted. "task_001" matches a preset field type (e.g., task identifier), and "137" matches a preset field type (e.g., number); both are variable fields and are not included in the constant field sequence. The remaining fields, such as "Container," "killed," "by," "the," "system," and "exitCode," do not match any preset field type and are constant fields. These are preserved in their original text and concatenated in their original order to obtain the constant field sequence corresponding to the target log entry. The log template library is then queried to see if a historical log template matching this constant field sequence exists. If the log template corresponding to this log bucket has been previously parsed and stored in the log template library, a match is found, and the historical template is reused directly. If this log bucket represents a new pattern appearing for the first time, a match is not found, and a large language model is called for comparative parsing.

[0086] In this embodiment, since most of the logs generated by the big data computing platform are repetitive types that have already appeared, by querying the log template library, most of the logs to be parsed in the log buckets can be directly reused using historical templates through constant field sequence matching, skipping the vector retrieval and large language model calling steps. The constant field sequence matching is an exact match with O(1) time complexity, and the hit path latency is extremely low, enabling the output of parsing results in milliseconds, meeting the latency requirements of online real-time processing. At the same time, exact matching ensures the security of reuse, preventing the misuse of historical templates that are similar but belong to different log types. Thus, the high-cost reasoning capability of the large language model is precisely used for the new format logs, avoiding redundant calls and reducing the number of calls to the large language model by an order of magnitude.

[0087] In one optional implementation, any log to be parsed can be selected from each log bucket, and the semantic vector of any log to be parsed corresponding to each log bucket can be determined. Based on the semantic vector of any log to be parsed corresponding to each log bucket, semantic similarity retrieval is performed in the prompt word database to obtain at least one historical log and at least one historical log template corresponding to each log bucket. The at least one historical log and at least one historical log template corresponding to each log bucket are used as log samples. At least one log to be parsed in each log bucket and the log sample corresponding to each log bucket are input into a large language model, so that the large language model performs comparative parsing processing on at least one log to be parsed in each log bucket based on the log sample corresponding to each log bucket, to obtain the log template corresponding to each log to be parsed in each log bucket.

[0088] In this embodiment, a prompt word database can be maintained. The prompt word database is a vector database used to store historical parsing examples. Specifically, the prompt word database stores multiple historical parsing records. Each record is organized in the form of a triple, including: the original text of the historical log to be parsed, the historical log template corresponding to the historical log to be parsed, and the embedding vector of the historical log to be parsed. The embedding vector refers to the semantic vector obtained by encoding the original log text through a pre-trained embedding model. This semantic vector can represent the semantic features of the log text, making logs with similar semantics closer together in the vector space.

[0089] Specifically, for each log bucket obtained through hash bucketing, any log to be parsed can be selected from each log bucket. Optionally, for each log bucket obtained through hash bucketing, if no historical log template matching the constant field sequence corresponding to any target log to be parsed is found in the log template library, any log to be parsed can be selected from each log bucket. The selected log to be parsed is used to represent the overall semantic features of that log bucket. Then, the semantic vector of the log to be parsed corresponding to each log bucket is determined. Specifically, a pre-trained embedding model can be used to encode the log to be parsed into a semantic vector. The pre-trained embedding model can be a pre-trained language model that can convert unstructured log text into a fixed-dimensional semantic vector representation.

[0090] After obtaining the semantic vector, semantic similarity retrieval is performed in the prompt word database based on the semantic vector of any log to be parsed corresponding to each log bucket. Semantic similarity retrieval refers to using the semantic vector as the query vector, performing vector nearest neighbor retrieval in the prompt word database, calculating the similarity (e.g., cosine similarity) between the query vector and the embedding vectors of each historical parsed record in the prompt word database, and returning the K historical parsed records with the highest similarity, where K is a positive integer. The retrieved results include: at least one historical log, and the historical log template corresponding to that at least one historical log. The retrieved at least one historical log and its corresponding historical log template are used as log samples. These log samples serve as few-shot examples for the large language model, used to demonstrate template extraction paradigms for similar logs to the large language model, guiding it to perform template inference according to the format and logic of the examples.

[0091] After obtaining log samples, at least one log entry to be parsed from each log bucket, along with the corresponding log sample for that bucket, is input into the large language model. Upon receiving this input, the large language model, based on the log samples, performs comparative parsing on each log entry to be parsed within that log bucket, identifying which fields are variables and which are constants, and generating the corresponding log template for that log bucket. The log samples provide the large language model with a reference to historical parsing experience, enabling it to learn from semantically similar parsing examples in the past when processing new log formats it has never handled before, rather than inferring from scratch, thereby improving the accuracy of template extraction.

[0092] For example, take a log bucket obtained after hashing and bucketing as an example. This log bucket contains the following three logs to be parsed: "Lost connection to JobManager akka: / / flink@10.92.80.57Reconnecting...", "Lost connection to JobManager akka: / / flink@10.92.80.58Reconnecting...", and "Lost connection to JobManager akka: / / flink@10.92.80.59Reconnecting...".

[0093] Select any log entry to be parsed (e.g., the first log entry) from the log bucket and encode it into a semantic vector using a pre-trained embedding model. Use this semantic vector as the query vector to perform a semantic similarity search in the prompt word database. Retrieve the most semantically similar historical log entry and its corresponding historical log template as log samples. For example, the retrieved log samples may include: Historical log entry: "Lost connection to ResourceManager akka: / / flink@10.0.0.1 Reconnecting...", Historical log template: "Lost connection to ResourceManagerakka: / / flink@<*>. Reconnecting...".

[0094] The above log sample, along with three log entries to be parsed from this log bucket, are input into the large language model. Based on the log sample, the large language model performs comparative parsing on these three log entries. Through comparison, the IP address field values ​​in the three log entries are different (10.92.80.57, 10.92.80.58, 10.92.80.59), and are identified as variables; the other fields (such as "Lost connection to JobManager akka: / / flink@" and "Reconnecting...") are the same in all three log entries and are identified as constants. Simultaneously, the large language model referenced historical log templates by replacing IP addresses with placeholders in the log sample, ultimately generating the log template corresponding to this log bucket as: "Lost connection to JobManager akka: / / flink@<*>. Reconnecting...".

[0095] In this embodiment, semantic similarity retrieval of the prompt word database provides historical parsing samples as a reference for each large language model call. This allows the large language model to draw on historical parsing experience with semantic similarity when processing new format logs it has seen before, rather than inferring from scratch, reducing template inference errors caused by lack of context. Since the retrieval in the prompt word database is based on semantic vector similarity rather than exact string matching, even if historical logs and the current log to be parsed differ in specific words or formats, as long as they are semantically similar, historical parsing experience can still be recalled and used to assist in template inference of the current log, achieving cross-format and cross-version experience reuse.

[0096] Optionally, after obtaining the log template, you can also output the log template, a list of variable values, and a hit count.

[0097] The variable value list corresponds to the log template and records the specific values ​​of the variable fields in each log entry to be parsed. For each log entry within the same log bucket, the original field values ​​at the corresponding positions of the variable placeholders in the log template are extracted sequentially to form the variable value sequence for that log entry. The variable value sequences of all log entries to be parsed constitute the variable value list for that log bucket. The variable value list is used to record the variable information in the original logs, ensuring that the logs after template compression can still completely reconstruct the variable values ​​of the original logs, providing a structured data foundation for subsequent processing.

[0098] The hit count records the number of times a historical log template matching the constant field sequence of the target log file is found in the log template library during the current parsing process. For log buckets where a reused historical log template is found through the log template library, the hit count is incremented by one; for log buckets where a reused historical log template is not found through the log template library, the hit count is zero. As the system runs longer and the log template library becomes richer, the proportion of the hit count to the total number of log buckets continuously increases, the number of large language model calls decreases accordingly, and the system can achieve self-evolution under unsupervised conditions.

[0099] In one optional implementation, the historical parsing log, the historical log template corresponding to the historical parsing log, and the semantic vector of the historical parsing log can also be associated and stored in the prompt word database; and / or, the constant field sequence corresponding to the historical parsing log can be extracted, and the constant field sequence corresponding to the historical parsing log and the historical log template corresponding to the historical parsing log can be associated and stored in the log template library.

[0100] Specifically, the constant field sequence corresponding to the historical parsing log can be extracted, and the constant field sequence corresponding to the historical parsing log, along with the historical log template corresponding to the historical parsing log, can be stored in the log template library. The historical parsing log refers to the log to be parsed that has already been parsed before the current parsing round; the historical log template refers to the log template already generated for the historical parsing log; the method for extracting the constant field sequence is described above and will not be repeated here.

[0101] After associating and storing the constant field sequence of historical parsed logs with historical log templates in the log template library, when the current parsing process queries the log template library to see if a historical log template matching the constant field sequence of any target log to be parsed exists, it can also query the log template library to see if a historical log template matching the constant field sequence of the target log to be parsed in the current log bucket exists. If it exists, the historical log template is directly reused, eliminating the need to call the large language model, improving log parsing efficiency, avoiding redundant calls, and reducing the number of calls to the large language model by an order of magnitude.

[0102] For example, prior to this parsing, the system has already processed historical parsing logs and generated corresponding log templates. The constant field sequence corresponding to the historical parsing log "Lost connection to JobManager akka: / / flink@10.92.80.57Reconnecting..." is associated with the historical log template "Lost connection to JobManager akka: / / flink@<*>. Reconnecting..." and stored as a key-value pair in the log template library. Specifically, this key-value pair can be stored in an in-memory dictionary and synchronously persisted to a CSV file. During this parsing process, when the system encounters a new log bucket with the same constant field sequence, it can directly retrieve the historical log template when querying the log template library, without needing to call the large language model again.

[0103] Specifically, historical parsing logs, their corresponding historical log templates, and their semantic vectors are associated and stored in the prompt word database. The storage format is a triple: the first element is the original text of the historical parsing log, the second is the corresponding historical log template, and the third is the semantic vector of the historical parsing log. The generation method of the semantic vectors is described above and will not be repeated here.

[0104] After storing the triples of historical parsing logs in the prompt word database, when the prompt word database is searched during the current parsing process, semantic similarity can be searched in the prompt word database based on the semantic vector of any log to be parsed in the current log bucket. This retrieves semantically similar historical parsing logs and their corresponding historical log templates as log samples, providing a small number of sample examples for the large language model and guiding the large language model to quickly and accurately infer new log templates.

[0105] For example, the following triples are associated and stored in the prompt word database: the first is the original historical parsing log text "Lost connection to JobManager akka: / / flink@10.92.80.57 Reconnecting...", the second is the corresponding historical log template "Lost connection to JobManager akka: / / flink@<*>.Reconnecting...", and the third is the semantic vector of the log. During this parsing process, when the system retrieves logs with similar semantics but different fields (e.g., "Lost connection to ResourceManager akka: / / flink@10.0.0.1Reconnecting..."), it can retrieve the above triples as log samples from the prompt word database by semantic similarity retrieval. This assists the large language model in quickly inferring the template of new logs, improving log parsing efficiency and the accuracy of log template extraction.

[0106] In one optional implementation, a target log to be parsed can be determined from each log bucket, and a sequence of constant fields corresponding to each target log to be parsed can be extracted. Each sequence of constant fields is associated with a log template corresponding to each target log to be parsed and stored in a log template library; and / or, multiple logs to be parsed in each log bucket, the log templates corresponding to the multiple logs to be parsed, and the semantic vectors of the multiple logs to be parsed are associated and stored in a prompt word database.

[0107] The method for determining the target log to be parsed is described in the relevant description below, and the method for extracting the constant field sequence is also described in the relevant description above, and will not be repeated here. Each constant field sequence is associated with the log template corresponding to each target log to be parsed and stored in the log template library; that is, each constant field sequence and the log template corresponding to each log to be parsed are written back to the log template library.

[0108] After the log template library is written back, when the system subsequently processes logs with the same constant field sequence to be parsed, it can query the log template that was written back in the log template library, realizing precise matching and reuse with O(1) time complexity, without having to call the large language model again.

[0109] For example, after the large language model generates a log template "Lost connection to JobManager akka: / / flink@<*>. Reconnecting..." for multiple logs to be parsed in a certain log bucket, it identifies the target log to be parsed from that log bucket and extracts its constant field sequence. This constant field sequence is then associated with the aforementioned log template as a key-value pair and stored in the log template library. Specifically, this key-value pair can be appended to an in-memory dictionary and simultaneously flushed to a CSV file for persistent storage. When the system subsequently processes logs with the same constant field sequence, it can query the log template library to find the matching log template, completing the parsing instantly with zero calls to the large language model.

[0110] Specifically, multiple log entries to be parsed from each log bucket, their corresponding log templates, and their semantic vectors are associated and stored in the prompt word database. This involves writing back these log entries, templates, and semantic vectors from each log bucket to the prompt word database. The storage format is a triple: the first entry is the original text of the desired log entry, the second is the corresponding log template, and the third is the semantic vector. The generation method for the semantic vectors is described above and will not be repeated here.

[0111] After the prompt word database is written back, the candidate sample pool of the prompt word database is expanded. When the system subsequently processes new logs that are semantically similar but have slightly different formats, it can retrieve the log to be parsed and its log template from this write-back as log samples in the prompt word database retrieval step. This provides high-quality few-sample examples for the large language model and improves the template inference accuracy of new format logs.

[0112] For example, the following triples are associated and stored in the prompt word database: the first is the original text of any log to be parsed, "Lost connection to JobManager akka: / / flink@10.92.80.57 Reconnecting...", the second is the corresponding log template, "Lost connection to JobManager akka: / / flink@<*>.Reconnecting...", and the third is the semantic vector of the log. When the system subsequently encounters logs with similar semantics but different fields (e.g., "Lost connection to ResourceManager akka: / / flink@10.0.0.1Reconnecting..."), it can retrieve the above triples from the prompt word database using semantic similarity as log samples, assisting the large language model in quickly inferring the template of the new log.

[0113] In this embodiment, through the aforementioned two-layer write-back mechanism, the newly generated log template in this parsing becomes the historical log template, and the log to be parsed in this parsing becomes the historical parsing log for subsequent parsing. The longer the system runs, the richer the log template library becomes, the higher the log template library hit rate, and the fewer calls to the large language model; the richer the prompt word database, the higher the quality of few-sample retrieval of new format logs, and the continuously improving cold-start parsing accuracy. No manual annotation is required throughout the process; the system automatically adapts to changes in log format under unsupervised conditions, forming a self-reinforcing closed loop of "processing → write-back → better processing".

[0114] In this embodiment, multiple logs to be parsed are obtained; fields matching a preset field type in each log are replaced with field type identifiers corresponding to the preset field type, resulting in a string corresponding to each log; based on the string corresponding to each log, the multiple logs are hash-bucketed to obtain at least one log bucket; each log bucket includes at least one log to be parsed, and the strings corresponding to each log in the same log bucket are identical; at least one log in each log bucket is input into a large language model, so that the large language model performs comparative parsing processing on at least one log in each log bucket, resulting in a log template corresponding to each log in each log bucket. By replacing fields matching a preset field type in each log to be parsed with field type identifiers corresponding to the preset field type, and performing hash-bucketing based on the replacement results, the complexity of log grouping is reduced, and the log parsing efficiency is improved; furthermore, by inputting at least one log in the same log bucket into the large language model for comparative parsing processing, the large language model can accurately identify variables and constants by utilizing the differences between the logs in each bucket, thus improving the accuracy of log template extraction.

[0115] Based on the above description, please refer to Figure 3 , Figure 3 This is a flowchart illustrating another log parsing method provided in an embodiment of this application, as shown below. Figure 3 The log parsing method shown includes, but is not limited to, steps S301 to S306, wherein: S301, retrieve multiple logs to be parsed.

[0116] For a detailed description of step S301 in this embodiment, please refer to the description of step S201 above. This embodiment will not repeat the details here.

[0117] S302, replace the fields in each log to be parsed that match the preset field type with the field type identifier corresponding to the preset field type, and obtain the string corresponding to each log to be parsed.

[0118] For a detailed description of step S302 in this embodiment, please refer to the description of step S202 above. This embodiment will not repeat the details here.

[0119] S303, based on the string corresponding to each log to be parsed, perform hash bucketing on multiple logs to be parsed to obtain at least one log bucket; a log bucket includes at least one log to be parsed, and the strings corresponding to each log to be parsed in the same log bucket are the same.

[0120] For a detailed description of step S303 in this application embodiment, please refer to the description of step S203 above. This application embodiment will not repeat the details here.

[0121] S304. Determine the target log to be parsed from each log bucket, and encode the semantic vector of each target log to be parsed to obtain the semantic vector of each target log to be parsed; the target log to be parsed is determined based on the field length and / or the proportion of variables.

[0122] The process involves hash bucketing to obtain at least one log bucket, with each bucket containing multiple log entries with the same structure to be parsed. Optionally, hash bucketing can be performed based on the target semantic segment (such as the first line of the semantic segment). However, since hash bucketing is performed only based on the target semantic segment, log entries with slightly different first lines but the same overall semantic meaning may be assigned to different log buckets. Therefore, a representative semantic vector can be generated for each log bucket, providing a basis for subsequent inter-bucket merging based on semantic similarity.

[0123] Specifically, the first step is to determine the target logs to be parsed from each log bucket. A target log is one or more logs selected from a log bucket that represent the overall semantic characteristics of that log bucket. The selection method for target logs can be determined based on field length and / or variable proportion.

[0124] The variable percentage refers to the proportion of fields that are variables in the log to be parsed to the total number of fields in that log. Fields that are variables are those in the log that match a preset field type. A lower variable percentage indicates that the log entry contains more constant fields and fewer variable fields, better reflecting the structural characteristics of the log template corresponding to that log bucket. Optionally, the log entry with the lowest variable percentage in the log bucket can be selected as the target log entry; alternatively, log entries with a variable percentage less than or equal to a variable percentage threshold can be selected as the target log entry.

[0125] Here, field length refers to the total number of fields contained in the log to be parsed. Optionally, logs with a field length that is in the middle of the log bucket can be selected as target logs to be parsed. A middle field length means that the number of fields in the log to be parsed is closest to the average number of fields in all logs to be parsed in that log bucket. Selecting logs with a middle field length as target logs avoids selecting abnormal logs with insufficient semantic information due to too few fields, and also avoids selecting abnormal logs with too many fields containing excessive noise, making the encoded semantic vector more representative and robust.

[0126] Optionally, both the variable percentage and field length can be considered to determine the target log to be parsed. For example, you can first filter out several logs with the lowest variable percentage, and then select the log with the field length closest to the average field length of the log bucket as the target log to be parsed; or, you can first filter out several logs with average field length, and then select the log with the lowest variable percentage as the target log to be parsed.

[0127] After identifying the target logs to be parsed, semantic vector encoding is performed on each target log to obtain a semantic vector for each target log. Semantic vector encoding refers to the process of converting the text of the target log into a fixed-dimensional numerical vector using a pre-trained embedding model. This numerical vector can represent the semantic features of the log text, making logs with similar semantics closer together in the vector space.

[0128] In one optional implementation, any target log to be parsed can be semantically segmented to obtain multiple semantic segments corresponding to the target log to be parsed; the semantic segment weight of each semantic segment corresponding to the target log to be parsed can be determined; the semantic segment weight is determined according to at least one of the following: semantic segment position, semantic segment length, and semantic segment keywords; each semantic segment is semantically vector encoded to obtain a semantic segment vector corresponding to each semantic segment; based on the semantic segment vector corresponding to each semantic segment and the semantic segment weight of each semantic segment, the semantic vector of the target log to be parsed is obtained.

[0129] For long logs containing multi-line stack traces, nested JSON structures, etc., treating them as flat text for overall encoding results in the semantic information at different structural levels being treated equally. This leads to the semantic features of the core error message being diluted by secondary information such as intermediate stack frames, resulting in the complete loss of structural semantics. Therefore, a better approach is to first segment the log into semantic segments and assign weights to each segment, then encode each segment separately and perform a weighted fusion. This restores the hierarchical structure to a weighted sequence of semantic units, enabling the generated semantic vectors to perceive the log's structural hierarchy and more accurately reflect its core semantic features.

[0130] Specifically, any target log to be parsed can be semantically segmented to obtain multiple semantic segments corresponding to that target log. Semantic segmentation can also be performed on any target log according to preset structural boundaries, resulting in multiple semantic segments corresponding to that target log. The preset structural boundaries refer to the identifiers used to divide different structural levels in the log text. Preset structural boundaries can include at least one of the following: line breaks, indentation levels, stack frame identifiers, JSON nesting levels, and XML nesting levels. For example, for Java stack logs, stack frame identifiers can be prefixes such as "atorg.apache." or "at com.xhs."; for Python stack logs, stack frame identifiers can be prefixes such as "File". After semantic segmentation, the originally continuous long text of any target log is split into multiple structurally independent semantic segments, each corresponding to a structural level in the log. For example, an exception log containing a Java stack can be segmented into the following three semantic segments: the first semantic segment contains exception type and level information, the middle semantic segment contains stack frame information, and the last semantic segment contains root cause description information.

[0131] Determine the semantic segment weight for each semantic segment corresponding to any given target log to be parsed. The semantic segment weight characterizes the importance of that semantic segment within the entire log; a higher weight indicates a greater contribution of the semantic information of that segment to the final generated semantic vector. The semantic segment weight can be determined based on at least one of the following: semantic segment position, semantic segment length, and semantic segment keywords. The specific determination method is as follows: Semantic segment position refers to the structural location of a semantic segment within the log to be parsed, such as the first line, the last line, or a middle line. The semantic segment position can be characterized by a position factor, which is determined based on the segment's structural location. The first line is the core basis for log category identification; the last line carries root cause information and has strong distinguishing value; the line numbers of middle stack frames may differ slightly among logs of the same type, so assigning them low weights can avoid noise interference. Therefore, the first line (the segment containing the exception type / level) has the highest weight, and its position factor can be set to, for example, 1.0; the last line (the segment containing the root cause description) has the second highest weight, and its position factor can be set to, for example, 0.7; and the middle layer (the segment containing the stack frame) has the lowest weight, and its position factor can be set to, for example, 0.2.

[0132] Semantic segment length refers to the number of fields contained in a semantic segment. Semantic segment length can be characterized by a length factor, which is determined based on the number of fields contained in the semantic segment. Specifically, the length factor is the smaller of the ratio between the number of fields in the semantic segment and the average number of fields in all semantic segments in the target log to be parsed, and 1.0. Semantic segments with more information have higher length factors; setting an upper limit of 1.0 can prevent excessively long stack frames from dominating the weighting.

[0133] Semantic segment keywords refer to the preset exception keywords contained in the semantic segment. Preset exception keywords are specific terms pre-defined to identify exception information in the log, and may include at least one of the following: ERROR, Exception, Caused by, exitCode, FATAL. Semantic segment keywords can be characterized by keyword factors, which are determined based on whether the semantic segment contains preset exception keywords. If the semantic segment contains the above exception keywords, the keyword factor is amplified to highlight the contribution of the exception information to the generation of the semantic vector; if the semantic segment does not contain any preset exception keywords, the keyword factor is 1.0, i.e., no amplification is applied. For example, the amplification factor can be set to 1.5.

[0134] Optionally, after determining the above three factors, the semantic segment weight can be calculated according to the following formula: Semantic segment weight = (position factor × α + length factor × β) × keyword factor. Where α and β are preset coefficients, α + β = 1. For example, α can be 0.7 and β can be 0.3, allowing the position factor to dominate the weight calculation, while the length factor plays a supporting role.

[0135] Semantic vector encoding is performed on each semantic segment to obtain a corresponding semantic segment vector. A semantic segment vector is a fixed-dimensional numerical vector obtained by encoding the text of a semantic segment individually using a pre-trained embedding model. Each semantic segment vector independently represents the semantic features of that semantic segment.

[0136] Based on the semantic segment vector corresponding to each semantic segment and the semantic segment weight of each semantic segment, a weighted fusion is performed to obtain the semantic vector of any target log to be parsed. Weighted fusion refers to multiplying each semantic segment vector by its corresponding semantic segment weight and then summing the results to obtain a comprehensive semantic vector. In this comprehensive semantic vector, the semantic features of high-weight semantic segments are amplified, while the semantic features of low-weight semantic segments are suppressed, thus making the final generated semantic vector more accurately reflect the core semantics of the log.

[0137] In this embodiment, by restoring the hierarchical structure to a weighted sequence of semantic units, core semantic segments such as the first-line exception type and the last-line root cause description dominate the semantic vector, while noise interference from secondary semantic segments such as the intermediate stack frames is effectively suppressed, resulting in a more accurate representation of the semantic vector. Because the semantic vector more accurately reflects the core semantic features of the log, it can accurately merge log buckets with the same semantics but slightly different first-line descriptions, thus improving the clustering recall rate.

[0138] S305, based on the semantic vector of each target log to be parsed, determine the semantic similarity between every two target logs to be parsed, and merge the log buckets corresponding to the target logs to be parsed whose semantic similarity reaches the semantic similarity threshold to obtain at least one log group.

[0139] Specifically, based on the semantic vector of each target log to be parsed, the semantic similarity between any two target logs is determined. Semantic similarity is a measure of the similarity between the semantic vectors of two target logs, used to characterize the degree of semantic closeness between the two logs. Optionally, the cosine similarity between any two target logs can be determined. Cosine similarity is the cosine of the angle between two vectors, typically ranging from -1 to 1. The closer the value is to 1, the closer the directions of the two vectors are, meaning the two logs are more semantically similar. For example, the cosine similarity between the semantic vectors of two target logs can be calculated as the semantic similarity between those two target logs.

[0140] The semantic similarity threshold is a pre-set minimum similarity value used to determine whether two logs belong to the same semantic category. If the semantic similarity between two target logs reaches this threshold, it indicates that the two log buckets corresponding to the two target logs are highly similar semantically and should be merged into the same log group; if the semantic similarity threshold is not reached, it indicates that the two log buckets corresponding to the two target logs are significantly different semantically and should remain independent log groups.

[0141] Log buckets corresponding to target logs whose semantic similarity reaches the semantic similarity threshold are merged to obtain at least one log group. Each log group contains one or more merged log buckets, and the logs to be parsed within the same log group are semantically highly similar. For log buckets whose semantic similarity does not reach the threshold, each becomes an independent log group.

[0142] In this embodiment, hash bucketing can only perform exact matching based on the target semantic segment (such as the first line semantic segment), and cannot identify logs with different first line expressions but the same overall semantics. Semantic vector cluster merging uses the comprehensive semantic vector of the entire log for similarity judgment, which can re-aggregate log buckets with the same semantics but slightly different first line expressions, improving the recall rate of clustering. System version iterations may change the way the first line of the log is expressed, but the overall semantic framework remains unchanged. Semantic vector cluster merging can overcome the limitations of exact string matching, accurately grouping the above logs and providing a more complete comparative context for subsequent template inference.

[0143] S306, input multiple logs to be parsed from each log group into the large language model, so that the large language model can compare and parse the multiple logs to be parsed from each log group to obtain the log template corresponding to each log to be parsed from each log group.

[0144] In this process, semantically similar log buckets are merged into the same log group through semantic vector cluster merging, ensuring semantic consistency among the logs to be parsed within each log group. Multiple logs to be parsed within the same log group are then input into the large language model. The large language model automatically identifies which fields are variables and which are constants by comparing the differences between the logs within the group, and generates a log template for each log to be parsed in each log group.

[0145] For a detailed description of step S306 in this embodiment, please refer to the description of step S204 above. This embodiment will not repeat the details here.

[0146] In one optional implementation, before inputting multiple logs to be parsed from each log group into the large language model, a target log to be parsed can be determined from each log group, and a constant field sequence corresponding to each target log to be parsed can be extracted. The constant field sequence consists of fields in the target logs that do not match the preset field type. The log template library is queried to see if there is a historical log template that matches the constant field sequence corresponding to any target log to be parsed. If there is a historical log template that matches the constant field sequence corresponding to any target log to be parsed, then the historical log template that matches the constant field sequence corresponding to any target log to be parsed is used as the log template corresponding to each log in the log group corresponding to any target log to be parsed. If there is no historical log template that matches the constant field sequence corresponding to any target log to be parsed, then the step of inputting multiple logs to be parsed from each log group into the large language model is performed.

[0147] For detailed implementation of the embodiments of this application, please refer to the above-mentioned descriptions. The embodiments of this application will not be repeated here.

[0148] In one optional implementation, any log to be parsed can be selected from each log group, and the semantic vector of any log to be parsed corresponding to each log bucket can be determined. Based on the semantic vector of any log to be parsed corresponding to each log group, semantic similarity retrieval is performed in the prompt word database to obtain at least one historical log and at least one historical log template corresponding to each log group. The at least one historical log and at least one historical log template corresponding to each log group are used as log samples. Multiple logs to be parsed in each log group and the log samples corresponding to each log group are input into a large language model, so that the large language model performs comparative parsing processing on multiple logs to be parsed in each log group based on the log samples corresponding to each log group, and obtains the log template corresponding to each log to be parsed in each log group.

[0149] For detailed implementation of the embodiments of this application, please refer to the above-mentioned descriptions. The embodiments of this application will not be repeated here.

[0150] In one optional implementation, the historical parsing log, the historical log template corresponding to the historical parsing log, and the semantic vector of the historical parsing log can also be associated and stored in the prompt word database; and / or, the constant field sequence corresponding to the historical parsing log can be extracted, and the constant field sequence corresponding to the historical parsing log and the historical log template corresponding to the historical parsing log can be associated and stored in the log template library.

[0151] For detailed implementation of the embodiments of this application, please refer to the above-mentioned descriptions. The embodiments of this application will not be repeated here.

[0152] In one optional implementation, a target log to be parsed can be determined from each log group, and a sequence of constant fields corresponding to each target log to be parsed can be extracted. Each sequence of constant fields is associated with a log template corresponding to each target log to be parsed and stored in a log template library; and / or, multiple logs to be parsed in each log group, the log templates corresponding to the multiple logs to be parsed, and the semantic vectors of the multiple logs to be parsed are associated and stored in a prompt word database.

[0153] For detailed implementation of the embodiments of this application, please refer to the above-mentioned descriptions. The embodiments of this application will not be repeated here.

[0154] In this embodiment, multiple logs to be parsed are obtained; fields matching a preset field type in each log are replaced with field type identifiers corresponding to the preset field type, resulting in a string corresponding to each log; based on the string corresponding to each log, the multiple logs to be parsed are hashed and bucketed to obtain at least one log bucket; each log bucket includes at least one log to be parsed, and the strings corresponding to each log to be parsed in the same log bucket are the same; target logs to be parsed are determined from each log bucket, and semantic vector encoding is performed on each target log to be parsed to obtain a semantic vector for each target log; the target logs to be parsed are determined based on field length and / or variable proportion; based on the semantic vector of each target log to be parsed, the semantic similarity between every two target logs to be parsed is determined, and the log buckets corresponding to target logs whose semantic similarity reaches a semantic similarity threshold are merged to obtain at least one log group; multiple logs to be parsed in each log group are input into a large language model, so that the large language model performs comparative parsing processing on multiple logs to be parsed in each log group to obtain a log template corresponding to each log to be parsed in each log group. By replacing fields in each log entry that match a preset field type with the corresponding field type identifier, and performing hash bucketing based on the replacement results, the complexity of log grouping is reduced and the efficiency of log parsing is improved. Through semantic vector encoding and inter-cluster merging, log buckets with the same overall semantics are re-aggregated into the same log group, improving the recall rate of clustering. The merged log group is input into a large language model for comparative parsing, enabling the large language model to accurately identify variables and constants by utilizing the differences between each log entry in the log group, thus improving the accuracy of log template extraction.

[0155] Based on the above description, please refer to Figure 4 , Figure 4 This is a flowchart illustrating another log parsing method provided in an embodiment of this application, as shown below. Figure 4 The log parsing method shown includes, but is not limited to, steps S401 to S405, wherein: S401, retrieve multiple logs to be parsed.

[0156] For a detailed description of step S401 in this embodiment, please refer to the description of step S201 above. This embodiment will not repeat the details here.

[0157] S402, replace the fields in each log to be parsed that match the preset field type with the field type identifier corresponding to the preset field type, and obtain the string corresponding to each log to be parsed.

[0158] For a detailed description of step S402 in this embodiment, please refer to the description of step S202 above. This embodiment will not repeat the details here.

[0159] S403, based on the string corresponding to each log to be parsed, perform hash bucketing on multiple logs to be parsed to obtain at least one log bucket; a log bucket includes at least one log to be parsed, and the strings corresponding to each log to be parsed in the same log bucket are the same.

[0160] For a detailed description of step S403 in this application embodiment, please refer to the description of step S203 above. This application embodiment will not repeat the details here.

[0161] S404: Based on the number of logs to be parsed in each log bucket, perform clustering processing on the logs to be parsed in each log bucket to obtain at least one log unit bucket corresponding to each log bucket.

[0162] Specifically, for each log bucket obtained through hash bucketing, the number of logs to be parsed within that log bucket is obtained, and clustering is performed on the logs to be parsed within that log bucket based on this number. Clustering refers to the process of grouping similar logs into the same log unit bucket and dissimilar logs into different log unit buckets based on the similarity between the logs to be parsed.

[0163] Optionally, the clustering strategy used in the clustering process is determined based on the number of logs in the log bucket. Specifically, the number of logs in the log bucket can be compared with a preset log count threshold, and different clustering strategies can be selected based on the comparison result. Optionally, when the number of logs in a log bucket is less than a preset log number threshold, indicating a small number of logs in the bucket, a first clustering strategy can be adopted. The first clustering strategy is a precision-prioritizing clustering strategy. For example, it can be based on text similarity (such as Jaccard similarity) to directly merge the logs to be parsed in the bucket pairwise, grouping logs with similarity reaching the merging threshold into the same log unit bucket. When the number of logs in a log bucket is greater than or equal to the preset log number threshold, indicating a large number of logs in the bucket, the computational complexity of pairwise comparison is high, and a second clustering strategy can be adopted. The second clustering strategy is an efficiency-prioritizing clustering strategy. For example, it can be an approximate clustering strategy based on Locality Sensitive Hashing (LSH), which can quickly group a large number of logs in the bucket in O(n) time complexity, where n is the number of logs to be parsed in the log bucket.

[0164] Through the clustering process described above, each log bucket is further divided into at least one log unit bucket. A log unit bucket is a subgroup within a log bucket. Logs within the same log unit bucket exhibit a high degree of consistency in overall structure, while logs between different log unit buckets differ in local structure. Subsequent processing will input the log unit buckets into the large language model for comparative parsing, rather than the entire log bucket, thereby improving the precision of template extraction.

[0165] In this embodiment, for small buckets with a small number of logs, a precision-first clustering strategy is adopted to ensure grouping accuracy; for large buckets with a large number of logs, an efficiency-first clustering strategy is adopted to avoid the computational overhead of pairwise comparisons, meeting the real-time processing requirements of large-scale log scenarios. A single log bucket may contain multiple similar but slightly different log sub-patterns. Fine-grained clustering within the bucket separates these sub-patterns, making each group of logs input to the large language model more structurally consistent and the comparison between variables and constants clearer, thereby improving the accuracy of template extraction.

[0166] S405, input at least one log to be parsed from each log unit bucket into the large language model, so that the large language model performs comparison and parsing processing on at least one log to be parsed from each log unit bucket, and obtains the log template corresponding to each log to be parsed from each log unit bucket.

[0167] For each log unit bucket, at least one log entry to be parsed from that bucket is input into the large language model. Optionally, multiple log entries from the same log unit bucket can be input into the large language model together. The large language model performs comparative parsing on each log entry within the log unit bucket: it compares the values ​​of the same field position in each log entry within the same group. If the values ​​of the same field position are the same in all log entries, the field is identified as a constant and retained in the template; if the values ​​of the same field position differ in all log entries, the field is identified as a variable and replaced with a placeholder in the template. The large language model outputs the log template corresponding to each log entry in the log unit bucket.

[0168] For a detailed description of step S405 in this application embodiment, please refer to the description of step S204 above. This application embodiment will not repeat the details here.

[0169] In one optional implementation, before inputting at least one log entry to be parsed from each log unit bucket into the large language model, target log entries to be parsed can be determined from each log unit bucket, and a sequence of constant fields corresponding to each target log entry to be parsed can be extracted. The sequence of constant fields consists of fields in the target log entries to be parsed that do not match a preset field type. The log template library is queried to see if there is a historical log template that matches the sequence of constant fields corresponding to any target log entry to be parsed. If there is a historical log template that matches the sequence of constant fields corresponding to any target log entry to be parsed, then the historical log template that matches the sequence of constant fields corresponding to any target log entry to be parsed is used as the log template corresponding to each log entry to be parsed in the log unit bucket corresponding to any target log entry to be parsed. If there is no historical log template that matches the sequence of constant fields corresponding to any target log entry to be parsed, then the step of inputting at least one log entry to be parsed from each log unit bucket into the large language model is performed.

[0170] For detailed implementation of the embodiments of this application, please refer to the above-mentioned descriptions. The embodiments of this application will not be repeated here.

[0171] In one optional implementation, any log to be parsed can be selected from each log unit bucket, and the semantic vector of any log to be parsed corresponding to each log unit bucket can be determined. Based on the semantic vector of any log to be parsed corresponding to each log unit bucket, semantic similarity retrieval is performed in the prompt word database to obtain at least one historical log and at least one historical log template corresponding to each log unit bucket. The at least one historical log and at least one historical log template corresponding to each log unit bucket are used as log samples. Multiple logs to be parsed in each log unit bucket and the log samples corresponding to each log unit bucket are input into a large language model, so that the large language model performs comparative parsing processing on multiple logs to be parsed in each log unit bucket based on the log samples corresponding to each log unit bucket, to obtain the log template corresponding to each log to be parsed in each log unit bucket.

[0172] For detailed implementation of the embodiments of this application, please refer to the above-mentioned descriptions. The embodiments of this application will not be repeated here.

[0173] In one optional implementation, the historical parsing log, the historical log template corresponding to the historical parsing log, and the semantic vector of the historical parsing log can also be associated and stored in the prompt word database; and / or, the constant field sequence corresponding to the historical parsing log can be extracted, and the constant field sequence corresponding to the historical parsing log and the historical log template corresponding to the historical parsing log can be associated and stored in the log template library.

[0174] For detailed implementation of the embodiments of this application, please refer to the above-mentioned descriptions. The embodiments of this application will not be repeated here.

[0175] In one optional implementation, a target log to be parsed can be determined from each log unit bucket, and a sequence of constant fields corresponding to each target log to be parsed can be extracted. Each sequence of constant fields is associated with a log template corresponding to each target log to be parsed and stored in a log template library. And / or, multiple logs to be parsed in each log unit bucket, the log templates corresponding to the multiple logs to be parsed, and the semantic vectors of the multiple logs to be parsed are associated and stored in a prompt word database.

[0176] For detailed implementation of the embodiments of this application, please refer to the above-mentioned descriptions. The embodiments of this application will not be repeated here.

[0177] In this embodiment, multiple logs to be parsed are obtained; the fields in each log that match a preset field type are replaced with field type identifiers corresponding to the preset field type, resulting in a string corresponding to each log; based on the string corresponding to each log, the multiple logs are hashed and bucketed to obtain at least one log bucket; each log bucket includes at least one log to be parsed, and the strings corresponding to each log to be parsed in the same log bucket are the same; based on the number of logs to be parsed in each log bucket, the logs to be parsed in each log bucket are clustered to obtain at least one log unit bucket corresponding to each log bucket; at least one log to be parsed in each log unit bucket is input into a large language model, so that the large language model performs comparative parsing processing on at least one log to be parsed in each log unit bucket to obtain a log template corresponding to each log to be parsed in each log unit bucket. By replacing fields in each log entry that match a preset field type with the corresponding field type identifier, and performing hash bucketing based on the replacement results, the complexity of log grouping is reduced, and log parsing efficiency is improved. A single log bucket may contain multiple similar but slightly different log sub-patterns. Fine-grained clustering within the bucket separates these sub-patterns, resulting in at least one log unit bucket. This makes each group of logs input to the large language model more structurally consistent, and the comparison between variables and constants clearer, thereby improving the accuracy of template extraction. Furthermore, the large language model can accurately identify variables and constants by utilizing the differences between log entries within each log unit bucket, further improving the accuracy of log template extraction.

[0178] Based on the above description, please refer to Figure 5 , Figure 5 This is a flowchart illustrating another log parsing method provided in an embodiment of this application, as shown below. Figure 5 The log parsing method shown includes, but is not limited to, steps S501 to S507, wherein: S501, retrieve multiple logs to be parsed.

[0179] For a detailed description of step S501 in this embodiment, please refer to the description of step S201 above. This embodiment will not repeat the details here.

[0180] S502, replace the fields in each log to be parsed that match the preset field type with the field type identifier corresponding to the preset field type, and obtain the string corresponding to each log to be parsed.

[0181] For a detailed description of step S502 in this embodiment, please refer to the description of step S202 above. This embodiment will not repeat the details here.

[0182] S503, based on the string corresponding to each log to be parsed, perform hash bucketing on multiple logs to be parsed to obtain at least one log bucket; a log bucket includes at least one log to be parsed, and the strings corresponding to each log to be parsed in the same log bucket are the same.

[0183] For a detailed description of step S503 in this application embodiment, please refer to the description of step S203 above. This application embodiment will not repeat the details here.

[0184] S504. Based on the number of logs to be parsed in each log bucket, perform clustering processing on the logs to be parsed in each log bucket to obtain at least one log unit bucket corresponding to each log bucket.

[0185] For a detailed description of step S504 in this application embodiment, please refer to the description of step S404 above. This application embodiment will not repeat the details here.

[0186] S505: Determine the target log to be parsed from each log unit bucket, and encode the semantic vector of each target log to be parsed to obtain the semantic vector of each target log to be parsed; the target log to be parsed is determined based on the field length and / or the proportion of variables.

[0187] For a detailed description of step S505 in this embodiment, please refer to the description of step S304 above. This embodiment will not repeat the details here.

[0188] S506. Based on the semantic vector of each target log to be parsed, determine the semantic similarity between every two target logs to be parsed, and merge the log buckets corresponding to the target logs whose semantic similarity reaches the semantic similarity threshold to obtain at least one log group.

[0189] For a detailed description of step S506 in this embodiment, please refer to the description of step S305 above. This embodiment will not repeat the details here.

[0190] S507: Input multiple logs to be parsed from each log group into the large language model so that the large language model can perform comparative parsing processing on the multiple logs to be parsed from each log group to obtain the log template corresponding to each log to be parsed from each log group.

[0191] For a detailed description of step S507 in this embodiment, please refer to the description of step S306 above. This embodiment will not repeat the details here.

[0192] In this embodiment, multiple logs to be parsed are obtained; the fields in each log that match a preset field type are replaced with field type identifiers corresponding to the preset field type, resulting in a string corresponding to each log; based on the string corresponding to each log, the multiple logs are hash-bucketed to obtain at least one log bucket; each log bucket includes at least one log to be parsed, and the strings corresponding to each log to be parsed in the same log bucket are the same; based on the number of logs to be parsed in each log bucket, the logs to be parsed in each log bucket are clustered to obtain at least one log unit bucket corresponding to each log bucket; and the target logs are determined from each log unit bucket. The logs to be parsed are labeled, and semantic vector encoding is performed on each target log to obtain a semantic vector for each target log. The target logs to be parsed are determined based on field length and / or variable proportion. Based on the semantic vector of each target log, the semantic similarity between every two target logs is determined, and the log buckets corresponding to the target logs whose semantic similarity reaches the semantic similarity threshold are merged to obtain at least one log group. Multiple logs to be parsed in each log group are input into a large language model, so that the large language model can perform comparative parsing processing on multiple logs to be parsed in each log group to obtain a log template corresponding to each log to be parsed in each log group. By replacing fields matching the preset field type in each log entry with the corresponding field type identifier, and performing hash binning based on the replacement results, the complexity of log grouping is reduced, and log parsing efficiency is improved. A single log bucket may contain multiple similar but slightly different log sub-patterns; these sub-patterns are separated through fine-grained clustering within the bucket, resulting in at least one log unit bucket. This makes each group of logs input to the large language model more structurally consistent, and the comparison between variables and constants clearer, thereby improving the accuracy of template extraction. Through semantic vector encoding and inter-cluster merging, log unit buckets with the same overall semantics are re-aggregated into the same log group, improving the recall rate of clustering. The merged log group is then input into the large language model for comparative parsing, enabling the large language model to accurately identify variables and constants based on the differences between the log entries within the log group, further improving the accuracy of log template extraction.

[0193] Based on the above description, this application uses a batch of Flink exception logs containing multiple lines of Java stack structures as an example to illustrate in detail the complete processing flow from log acquisition to template output: Input: Raw exception logs. Multiple logs to be parsed are obtained. This example uses the following three Flink exception logs containing multi-line Java stack traces as examples: Log 1: "[ERROR] NullPointerException in UserService at com.xhs.UserService.getUser(UserService.java:42) at com.xhs.Controller.handle(Controller.java:18) at com.xhs.Filter.doFilter(Filter.java:91) Caused by: user_id is null, exitCode=1".

[0194] Log 2: "[ERROR] NullPointerException in UserService at com.xhs.UserService.getUser(UserService.java:42) at com.xhs.Controller.handle(Controller.java:29) at com.xhs.Filter.doFilter(Filter.java:91) Caused by: user_id is null, exitCode=1".

[0195] Log 3: "[ERROR] NullPointerException in OrderService at com.xhs.OrderService.getOrder(OrderService.java:67) at com.xhs.Controller.handle(Controller.java:18) at com.xhs.Filter.doFilter(Filter.java:91) Caused by: order_id is null, exitCode=1".

[0196] All three log entries contain a first line (exception type and level), a middle section (Java stack frame), and a last line (root cause description). The overall structure of the three log entries is consistent, but there are differences in the values ​​of specific fields: the exception type for log 1 and log 2 is "NullPointerException in UserService," while log 3 is "NullPointerException in OrderService"; the stack frame line numbers for log 1 and log 2 differ (Controller.java:18 and Controller.java:29); the root cause description for log 1 and log 2 is "user_id is null," while log 3 is "order_id is null."

[0197] Step 1: Multi-stage adaptive clustering: S1: Structured semantic segmentation and semantic segment weights.

[0198] Each log entry to be parsed is semantically segmented according to preset structural boundaries. These boundaries include newline characters and stack frame identifiers (e.g., "at com.xhs."). After segmentation, each log entry is divided into three semantic segments: Segment 1 (first line) contains the exception type and level; Segment 2 (middle) contains the Java stack frame (containing three lines of stack frame information); and Segment 3 (last line) contains the root cause description.

[0199] Determine the weight of each semantic segment for each log entry to be parsed. The semantic segment weight is determined based on at least one of the following: semantic segment position, semantic segment length, and semantic segment keyword. The semantic segment position corresponds to a position factor, the semantic segment length corresponds to a length factor, and the semantic segment keyword corresponds to a keyword factor, where the weight coefficient α for the position factor is 0.7, and the weight coefficient β for the length factor is 0.3. When a semantic segment contains a preset abnormal keyword, the keyword factor is amplified by a factor of 1.5; when the semantic segment does not contain a preset abnormal keyword, the keyword factor is 1.0. The final weight calculation formula is: Final Weight = (Position Factor × 0.7 + Length Factor × 0.3) × Keyword Factor.

[0200] Taking log 1 as an example, the weights of each semantic segment are calculated as follows: Segment 1 (first line): The content is "[ERROR] NullPointerException in UserService". This segment is located at the first line of the log, with a position factor of 1.0; the number of fields in this segment is close to the average number of fields in all segments, with a length factor of approximately 1.0; this segment contains the exception keyword "ERROR", with a keyword factor of 1.5. Substituting into the formula, the final weight is (1.0 × 0.7 + 1.0 × 0.3) × 1.5 = 1.50.

[0201] Segment 2 (Middle): This segment contains three lines of stack frame information. Located in the middle of the log, its position factor is 0.2; its length factor is approximately 1.0; and it does not contain any exception keywords, so its keyword factor is 1.0. Substituting these values ​​into the formula, the final weight is (0.2 × 0.7 + 1.0 × 0.3) × 1.0 = 0.44 (approximately 0.44).

[0202] Segment 3 (tail line): The content is "Caused by: user_id is null, exitCode=1". This segment is located at the tail line of the log, with a position factor of 0.7 and a length factor of approximately 1.0. This segment contains the exception keywords "Caused by" and "exitCode", with a keyword factor of 1.5. Substituting into the formula, the final weight is (0.7 × 0.7 + 1.0 × 0.3) × 1.5 = 1.19 (approximately 1.19).

[0203] The weights of each semantic segment in logs 2 and 3 are calculated in the same way as in log 1, and will not be repeated here.

[0204] The first line contains the exception level (ERROR) and exception type (NullPointerException), which are the core criteria for determining the log category and are given the highest weight. The last line contains the root cause description (Caused by, exitCode), which has strong distinguishing value and is given the second highest weight. The line numbers of the middle stack frames may differ slightly among logs of the same type, so they are given low weight to avoid noise interfering with the clustering results. By segmenting semantic segments and assigning weights to them, the hierarchical structure of long logs is restored to a weighted sequence of semantic units, enabling subsequent similarity calculations to be aware of the structural hierarchy.

[0205] S2: Field type identifier replacement and hash bucketing.

[0206] In this step, hash bucketing is performed only based on the first line semantic segment (target semantic segment). Fields in the first line semantic segment of each log entry that match a preset field type are replaced with their corresponding field type identifiers, resulting in a string corresponding to each first line semantic segment; hash bucketing is then performed based on this string.

[0207] For the three log entries mentioned above, "UserService" and "OrderService" in the first line semantic segment are identified as variable fields and replaced with their corresponding field type identifiers; the remaining fields (such as "[ERROR] NullPointerExceptionin") are retained. Since the string structures obtained after replacing the variable fields in the first line semantic segment of the three log entries are completely identical (variable fields are all replaced with the same field type identifier, and constant fields are retained and identical), the strings obtained after replacing the first line of the three log entries are exactly the same. Through hash bucketing, the three log entries are all assigned to the same log bucket (bucket A). The bucket stores the complete original log (including intermediate stack traces and the last line).

[0208] S3: In-bucket adaptive fine clustering.

[0209] Within bucket A, a clustering strategy is adaptively selected based on the number of logs in the bucket for fine-grained clustering. Bucket A contains 3 logs, and the number of logs is less than the log number threshold (e.g., the log number threshold is set to 10). A direct pairwise merging strategy based on Jaccard similarity is adopted. The Jaccard similarity between log 1 and log 2 is calculated. The two logs are identical in most fields except for the stack frame line number, and the similarity is high (e.g., about 0.91). Compared with log 1 and log 2, log 3 differs in the Service name (UserService and OrderService) in the first line and the field name (user_id and order_id) in the last line, but the overall structure is consistent, and the similarity is still high (e.g., about 0.73), reaching the merging threshold (e.g., the threshold is set to 0.65). Finally, all 3 logs are grouped into the same log unit bucket.

[0210] S4: Weighted inter-cluster merging based on semantic vectors.

[0211] This step performs semantic vector encoding and inter-cluster merging on each preliminary cluster. First, the target log to be parsed is determined from the log unit bucket, and a semantic vector is generated for the target log using segmented weighted encoding (the specific calculation process has been explained in detail in S1 above). Since all three logs have been grouped into the same log unit bucket after S3, only the semantic vector of the target log to be parsed in that log unit bucket needs to be generated. After calculating the comprehensive semantic vector, since there is currently only one cluster, no inter-cluster merging operation is needed; the log unit bucket is directly treated as a log group. Final clustering result: 3 logs are grouped into 1 log group.

[0212] Step 2: Local cache hit check: Before inputting the logs to be parsed from the log group into the large language model, a local cache hit check is performed. The target log to be parsed is determined from the log bucket corresponding to the log group, its constant field sequence is extracted, and a query is performed in the log template library to check if a matching historical log template exists. This embodiment assumes that the system is processing this type of abnormal log for the first time, and there is no matching historical log template in the log template library; therefore, the cache is not hit, and the process proceeds to the next step.

[0213] Step 3: Search the keyword database: Select any log entry to be parsed from the log group and encode it into a semantic vector using a pre-trained embedding model. Use this semantic vector as the query vector to perform a semantic similarity search in the prompt word database, recalling several historical logs with the most similar semantics and their corresponding historical log templates as log samples (i.e., few-sample examples). The retrieved log samples may include historical parsing records related to the "NullPointerException" exception, providing a reference example for large language models.

[0214] Step 4: Joint template inference within the large language model group: The three log entries to be parsed from this log group, along with the log sample retrieved in step three, are input into the large language model. The large language model performs comparative parsing on these three log entries based on the log sample. By comparing the differences in values ​​at the same field positions in the three log entries, the large language model identifies the following: "UserService" and "OrderService" in the first line have different values, thus identified as variables; "user_id" and "order_id" in the last line have different values, thus identified as variables; the line numbers in the stack frames (UserService.java:42, Controller.java:18 / 29, Filter.java:91) are partially the same and partially different, thus identified as variables; the remaining fields (such as "[ERROR] NullPointerException in", "Caused by:", "is null, exitCode=") are all the same in the three log entries, thus identified as constants. Specifically, the "exitCode" value is 1 in all three log entries, thus identified as a constant.

[0215] The large language model outputs the following log template for this log group: "[ERROR] NullPointerException in<*> at<*> at<*> at<*> Caused by:<*>is null, exitCode=<*>”.

[0216] In this log template, "<*>" is a variable placeholder, indicating that the position is a variable field in the original log.

[0217] Step 5: Adaptive Evolution Write-Back: The newly generated log template from this parsing is written back. The constant field sequence of the target log to be parsed is extracted and associated with the aforementioned log template, stored in the log template library (appended to the in-memory dictionary and simultaneously flushed to a CSV file for persistent storage). Multiple logs to be parsed in this log group, the aforementioned log template, and the semantic vectors of each log to be parsed are associated and stored in the prompt word database. After the write-back, the log template generated in this parsing becomes the historical log template, for use in parsing subsequent identical or similar logs.

[0218] Final output: After the above steps, the three original exception logs were parsed into one log template and a corresponding list of variable values, as follows: Log template (appears 3 times): "[ERROR] NullPointerException in<*> at<*> at<*> at<*> Caused by:<*>is null, exitCode=<*>”.

[0219] List of variable values: Service variable values: [UserService, UserService, OrderService]; Stack frame line number variable values: [UserService.java:42, Controller.java:18, Filter.java:91; UserService.java:42, Controller.java:29, Filter.java:91; OrderService.java:67, Controller.java:18, Filter.java:91]; Field variable values: [user_id, user_id, order_id]; ExitCode: 1 (constant).

[0220] Compression Results: Three original log entries (approximately 480 characters) were compressed into one log template and a few variable value examples (approximately 120 characters), achieving a compression ratio of approximately 4:1. During this parsing process, a query in the log template library failed, triggering one call to the large language model. After adaptive evolution and write-back, when the system subsequently processes similar log entries with the same constant field sequence, it can directly reuse the generated log template by querying the log template library, eliminating the need to call the large language model again.

[0221] This application also provides a computer storage medium storing program instructions, which, when executed, are used to implement the corresponding methods described in the above embodiments.

[0222] This application provides a computer program product, which includes a computer program stored in a computer storage medium. The processor of a computer device reads the computer program from the computer storage medium and executes the computer program, causing the computer device to perform the corresponding methods described in the above embodiments.

[0223] See also Figure 6 , Figure 6 This is a schematic diagram of the structure of a log parsing device provided in an embodiment of this application.

[0224] In one implementation of the log parsing device according to an embodiment of this application, the log parsing device includes the following structure.

[0225] Unit 601 is used to retrieve multiple log entries to be parsed. Replacement unit 602 is used to replace the field in each log to be parsed that matches the preset field type with the field type identifier corresponding to the preset field type, so as to obtain the string corresponding to each log to be parsed; Processing unit 603 is used to perform hash bucketing on the multiple logs to be parsed based on the string corresponding to each log to be parsed, to obtain at least one log bucket; a log bucket includes at least one log to be parsed, and the strings corresponding to each log to be parsed in the same log bucket are the same; The processing unit 603 is further configured to input at least one log to be parsed from each log bucket into the large language model, so that the large language model performs comparative parsing processing on at least one log to be parsed in each log bucket to obtain a log template corresponding to each log to be parsed in each log bucket.

[0226] In one implementation, the processing unit 603 performs hash bucketing on the multiple logs to be parsed based on the string corresponding to each log to be parsed, and after obtaining at least one log bucket, it can also be used for: The target log to be parsed is determined from each log bucket, and the semantic vector of each target log is encoded to obtain the semantic vector of each target log; the target log is determined based on the field length and / or the proportion of variables. Based on the semantic vector of each target log to be parsed, the semantic similarity between each pair of target logs to be parsed is determined, and the log buckets corresponding to the target logs whose semantic similarity reaches the semantic similarity threshold are merged to obtain at least one log group. The processing unit 603 inputs at least one log entry to be parsed from each log bucket into the large language model, so that the large language model performs comparative parsing processing on at least one log entry to be parsed in each log bucket, and obtains a log template corresponding to each log entry to be parsed in each log bucket, which can be used for: Multiple logs to be parsed from each log group are input into the large language model, so that the large language model performs comparative parsing processing on the multiple logs to be parsed from each log group to obtain the log template corresponding to each log to be parsed from each log group.

[0227] In one implementation, the processing unit 603 performs semantic vector encoding on each target log to be parsed to obtain a semantic vector for each target log to be parsed, which can be used for: Semantic segmentation is performed on any target log to be parsed to obtain multiple semantic segments corresponding to the target log to be parsed; Determine the semantic segment weight of each semantic segment corresponding to any target log to be parsed; the semantic segment weight is determined based on at least one of the following: semantic segment position, semantic segment length, and semantic segment keywords; Each semantic segment is semantically vector-encoded to obtain a semantic segment vector corresponding to each semantic segment; Based on the semantic segment vector corresponding to each semantic segment and the semantic segment weight of each semantic segment, the semantic vector of any target log to be parsed is obtained.

[0228] In one implementation, the replacement unit 602 replaces the fields in each log entry to be parsed that match a preset field type with the field type identifier corresponding to the preset field type, thereby obtaining a string corresponding to each log entry to be parsed, which can be used for: Each log entry to be parsed is semantically segmented to obtain multiple semantic segments corresponding to each log entry to be parsed; For any log to be parsed, a target semantic segment is determined from the multiple semantic segments of the log to be parsed. Replace the field in the target semantic segment corresponding to each log to be parsed that matches the preset field type with the field type identifier corresponding to the preset field type to obtain the string corresponding to each target semantic segment; The processing unit 603 performs hash bucketing on the multiple logs to be parsed based on the string corresponding to each log to be parsed, obtaining at least one log bucket, which can be used for: Based on the string corresponding to each target semantic segment, the multiple logs to be parsed are hashed and bucketed to obtain at least one log bucket.

[0229] In one implementation, the processing unit 603 performs hash bucketing on the multiple logs to be parsed based on the string corresponding to each log to be parsed, and after obtaining at least one log bucket, it can also be used for: Based on the number of logs to be parsed in each log bucket, the logs to be parsed in each log bucket are clustered to obtain at least one log unit bucket corresponding to each log bucket. The processing unit 603 inputs at least one log entry to be parsed from each log bucket into the large language model, so that the large language model performs comparative parsing processing on at least one log entry to be parsed in each log bucket, and obtains a log template corresponding to each log entry to be parsed in each log bucket, which can be used for: At least one log entry to be parsed from each log unit bucket is input into the large language model, so that the large language model performs comparative parsing processing on at least one log entry to be parsed from each log unit bucket to obtain the log template corresponding to each log entry to be parsed from each log unit bucket.

[0230] In one implementation, the processing unit 603 inputs at least one log entry to be parsed from each log bucket into a large language model, so that the large language model performs comparative parsing processing on at least one log entry to be parsed in each log bucket to obtain a log template corresponding to each log entry to be parsed in each log bucket, which can be used for: Select any log to be parsed from each log bucket, and determine the semantic vector of any log to be parsed corresponding to each log bucket; Based on the semantic vector of any log to be parsed corresponding to each log bucket, semantic similarity retrieval is performed in the prompt word database to obtain at least one historical log corresponding to each log bucket and a historical log template corresponding to the at least one historical log. The at least one historical log corresponding to each log bucket and the historical log template corresponding to the at least one historical log are used as log samples. At least one log entry to be parsed from each log bucket, and a log sample corresponding to each log bucket, are input into the large language model. The large language model then performs comparative parsing on at least one log entry to be parsed in each log bucket based on the log sample corresponding to each log bucket, thereby obtaining a log template corresponding to each log entry to be parsed in each log bucket.

[0231] In one implementation, before the processing unit 603 inputs at least one log entry to be parsed from each log bucket into the large language model, it can also be used for: The target log to be parsed is determined from each log bucket, and the constant field sequence corresponding to each target log is extracted; the constant field sequence is composed of fields in the target log that do not match the preset field type; Query the log template library to see if there is a historical log template that matches the sequence of constant fields corresponding to any target log to be parsed; If a historical log template exists that matches the sequence of constant fields corresponding to any of the target logs to be parsed, then the historical log template that matches the sequence of constant fields corresponding to any of the target logs to be parsed will be used as the log template for each log to be parsed in the log bucket corresponding to any of the target logs to be parsed. If no historical log template matches the sequence of constant fields corresponding to any of the target logs to be parsed, then the step of inputting at least one log from each log bucket into the large language model is performed.

[0232] In one embodiment, the processing unit 603 can also be used for: The historical parsing logs, the corresponding historical log templates, and the semantic vectors of the historical parsing logs are associated and stored in the prompt word database; and / or, Extract the constant field sequence corresponding to the historical parsing log, and associate and store the constant field sequence corresponding to the historical parsing log and the historical log template corresponding to the historical parsing log in the log template library.

[0233] In this embodiment, the acquisition unit 601 acquires multiple logs to be parsed; the replacement unit 602 replaces the fields in each log to be parsed that match a preset field type with the field type identifier corresponding to the preset field type, thereby obtaining a string corresponding to each log to be parsed; the processing unit 603 performs hash bucketing on the multiple logs to be parsed based on the string corresponding to each log to be parsed, thereby obtaining at least one log bucket; each log bucket includes at least one log to be parsed, and the strings corresponding to each log to be parsed in the same log bucket are the same; at least one log to be parsed in each log bucket is input into a large language model, so that the large language model performs comparative parsing processing on at least one log to be parsed in each log bucket, thereby obtaining a log template corresponding to each log to be parsed in each log bucket. By replacing the fields in each log entry that match the preset field type with the corresponding field type identifier, and performing hash bucketing based on the replacement results, the complexity of log grouping is reduced and the efficiency of log parsing is improved. Furthermore, by inputting at least one log entry from the same log bucket into the large language model for comparison and parsing, the large language model can accurately identify variables and constants by utilizing the differences between the log entries in the bucket, thereby improving the accuracy of log template extraction.

[0234] See also Figure 7 , Figure 7 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. The computer device in this embodiment includes a power supply module and other structures, and includes a processor 701, a memory 702, and a communication interface 703. The processor 701, the memory 702, and the communication interface 703 can exchange data, and the processor 701 implements the corresponding log parsing method.

[0235] The memory 702 may include volatile memory, such as random-access memory (RAM); the memory 702 may also include non-volatile memory, such as flash memory, solid-state drive (SSD), etc.; the memory 702 may also include a combination of the above types of memory.

[0236] Processor 701 may be a central processing unit (CPU). Processor 701 may also be a combination of CPU and GPU. In a computer device, multiple CPUs and GPUs may be included as needed for corresponding log parsing. In one embodiment, memory 702 is used to store program instructions. Processor 701 can invoke program instructions to implement the various methods described above in the embodiments of this application.

[0237] The communication interface 703 may include a display screen, microphone, or speaker, etc.

[0238] In one possible implementation, the processor 701 of the computer device calls program instructions stored in the memory 702 to obtain multiple logs to be parsed; replaces the fields in each log that match a preset field type with the field type identifier corresponding to the preset field type to obtain a string corresponding to each log; based on the string corresponding to each log, hash-bins the multiple logs to be parsed to obtain at least one log bucket; each log bucket includes at least one log to be parsed, and the strings corresponding to each log to be parsed in the same log bucket are the same; at least one log to be parsed in each log bucket is input into a large language model, so that the large language model performs comparative parsing processing on at least one log to be parsed in each log bucket to obtain a log template corresponding to each log to be parsed in each log bucket.

[0239] In one implementation, after the processor 701 performs hash bucketing on the multiple logs to be parsed based on the string corresponding to each log to be parsed, and obtains at least one log bucket, it can also perform the following operations: The target log to be parsed is determined from each log bucket, and the semantic vector of each target log is encoded to obtain the semantic vector of each target log; the target log is determined based on the field length and / or the proportion of variables. Based on the semantic vector of each target log to be parsed, the semantic similarity between each pair of target logs to be parsed is determined, and the log buckets corresponding to the target logs whose semantic similarity reaches the semantic similarity threshold are merged to obtain at least one log group. The processor 701 inputs at least one log entry to be parsed from each log bucket into the large language model, so that the large language model performs comparative parsing processing on at least one log entry to be parsed in each log bucket to obtain a log template corresponding to each log entry to be parsed in each log bucket, and can perform the following operations: Multiple logs to be parsed from each log group are input into the large language model, so that the large language model performs comparative parsing processing on the multiple logs to be parsed from each log group to obtain the log template corresponding to each log to be parsed from each log group.

[0240] In one implementation, the processor 701 performs semantic vector encoding on each target log to be parsed to obtain a semantic vector for each target log to be parsed, and can perform the following operations: Semantic segmentation is performed on any target log to be parsed to obtain multiple semantic segments corresponding to the target log to be parsed; Determine the semantic segment weight of each semantic segment corresponding to any target log to be parsed; the semantic segment weight is determined based on at least one of the following: semantic segment position, semantic segment length, and semantic segment keywords; Each semantic segment is semantically vector-encoded to obtain a semantic segment vector corresponding to each semantic segment; Based on the semantic segment vector corresponding to each semantic segment and the semantic segment weight of each semantic segment, the semantic vector of any target log to be parsed is obtained.

[0241] In one implementation, the processor 701 replaces the fields in each log entry that match a preset field type with the field type identifier corresponding to the preset field type, thereby obtaining a string corresponding to each log entry. The processor can then perform the following operations: Each log entry to be parsed is semantically segmented to obtain multiple semantic segments corresponding to each log entry to be parsed; For any log to be parsed, a target semantic segment is determined from the multiple semantic segments of the log to be parsed. Replace the field in the target semantic segment corresponding to each log to be parsed that matches the preset field type with the field type identifier corresponding to the preset field type to obtain the string corresponding to each target semantic segment; The processor 701 performs hash bucketing on the multiple logs to be parsed based on the string corresponding to each log to be parsed, obtaining at least one log bucket, and can perform the following operations: Based on the string corresponding to each target semantic segment, the multiple logs to be parsed are hashed and bucketed to obtain at least one log bucket.

[0242] In one implementation, after the processor 701 performs hash bucketing on the multiple logs to be parsed based on the string corresponding to each log to be parsed, and obtains at least one log bucket, it can also perform the following operations: Based on the number of logs to be parsed in each log bucket, the logs to be parsed in each log bucket are clustered to obtain at least one log unit bucket corresponding to each log bucket. The processor 701 inputs at least one log entry to be parsed from each log bucket into the large language model, so that the large language model performs comparative parsing processing on at least one log entry to be parsed in each log bucket to obtain a log template corresponding to each log entry to be parsed in each log bucket, and can perform the following operations: At least one log entry to be parsed from each log unit bucket is input into the large language model, so that the large language model performs comparative parsing processing on at least one log entry to be parsed from each log unit bucket to obtain the log template corresponding to each log entry to be parsed from each log unit bucket.

[0243] In one implementation, the processor 701 inputs at least one log entry to be parsed from each log bucket into a large language model, so that the large language model performs comparative parsing processing on at least one log entry to be parsed in each log bucket to obtain a log template corresponding to each log entry to be parsed in each log bucket, and can perform the following operations: Select any log to be parsed from each log bucket, and determine the semantic vector of any log to be parsed corresponding to each log bucket; Based on the semantic vector of any log to be parsed corresponding to each log bucket, semantic similarity retrieval is performed in the prompt word database to obtain at least one historical log corresponding to each log bucket and a historical log template corresponding to the at least one historical log. The at least one historical log corresponding to each log bucket and the historical log template corresponding to the at least one historical log are used as log samples. At least one log entry to be parsed from each log bucket, and a log sample corresponding to each log bucket, are input into the large language model. The large language model then performs comparative parsing on at least one log entry to be parsed in each log bucket based on the log sample corresponding to each log bucket, thereby obtaining a log template corresponding to each log entry to be parsed in each log bucket.

[0244] In one implementation, before the processor 701 inputs at least one log entry to be parsed from each log bucket into the large language model, it may also perform the following operations: The target log to be parsed is determined from each log bucket, and the constant field sequence corresponding to each target log is extracted; the constant field sequence is composed of fields in the target log that do not match the preset field type; Query the log template library to see if there is a historical log template that matches the sequence of constant fields corresponding to any target log to be parsed; If a historical log template exists that matches the sequence of constant fields corresponding to any of the target logs to be parsed, then the historical log template that matches the sequence of constant fields corresponding to any of the target logs to be parsed will be used as the log template for each log to be parsed in the log bucket corresponding to any of the target logs to be parsed. If no historical log template matches the sequence of constant fields corresponding to any of the target logs to be parsed, then the step of inputting at least one log from each log bucket into the large language model is performed.

[0245] In one embodiment, the processor 701 may also perform the following operations: The historical parsing logs, the corresponding historical log templates, and the semantic vectors of the historical parsing logs are associated and stored in the prompt word database; and / or, Extract the constant field sequence corresponding to the historical parsing log, and associate and store the constant field sequence corresponding to the historical parsing log and the historical log template corresponding to the historical parsing log in the log template library.

[0246] In this embodiment, processor 701 acquires multiple logs to be parsed; replaces the fields in each log that match a preset field type with the field type identifier corresponding to the preset field type, obtaining a string corresponding to each log; based on the string corresponding to each log, hash-bins the multiple logs to be parsed, obtaining at least one log bucket; each log bucket includes at least one log to be parsed, and the strings corresponding to each log to be parsed in the same log bucket are the same; at least one log to be parsed in each log bucket is input into a large language model, so that the large language model performs comparative parsing processing on at least one log to be parsed in each log bucket, obtaining a log template corresponding to each log to be parsed in each log bucket. By replacing the fields in each log to be parsed that match a preset field type with the field type identifier corresponding to the preset field type, and performing hash-binning based on the replacement results, the complexity of log grouping is reduced, and the log parsing efficiency is improved; and by inputting at least one log to be parsed in the same log bucket into the large language model for comparative parsing processing, the large language model can accurately identify variables and constants by utilizing the differences between each log to be parsed in the bucket, improving the accuracy of log template extraction.

[0247] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

[0248] The above-disclosed embodiments are merely some of the embodiments of this application, and should not be construed as limiting the scope of this application. Those skilled in the art can understand that all or part of the processes for implementing the above embodiments, and equivalent changes made in accordance with the claims of this application, still fall within the scope of this application.< / num> < / ip> < / num> < / ip>

Claims

1. A log parsing method, characterized in that, The method includes: Retrieve multiple log entries to be parsed; Replace the field in each log to be parsed that matches the preset field type with the field type identifier corresponding to the preset field type to obtain the string corresponding to each log to be parsed; Based on the string corresponding to each log to be parsed, the multiple logs to be parsed are hashed and bucketed to obtain at least one log bucket; a log bucket includes at least one log to be parsed, and the strings corresponding to each log to be parsed in the same log bucket are the same; At least one log entry to be parsed from each log bucket is input into the large language model, so that the large language model performs comparative parsing processing on at least one log entry to be parsed in each log bucket to obtain the log template corresponding to each log entry to be parsed in each log bucket.

2. The method as described in claim 1, characterized in that, After hashing and bucketing the multiple log files based on the string corresponding to each log file to be parsed, to obtain at least one log bucket, the process further includes: The target log to be parsed is determined from each log bucket, and the semantic vector of each target log is encoded to obtain the semantic vector of each target log; the target log is determined based on the field length and / or the proportion of variables. Based on the semantic vector of each target log to be parsed, the semantic similarity between each pair of target logs to be parsed is determined, and the log buckets corresponding to the target logs whose semantic similarity reaches the semantic similarity threshold are merged to obtain at least one log group. The step involves inputting at least one log entry to be parsed from each log bucket into a large language model, so that the large language model performs comparative parsing processing on at least one log entry to be parsed in each log bucket to obtain a log template corresponding to each log entry to be parsed in each log bucket, including: Multiple logs to be parsed from each log group are input into the large language model, so that the large language model performs comparative parsing processing on the multiple logs to be parsed from each log group to obtain the log template corresponding to each log to be parsed from each log group.

3. The method as described in claim 2, characterized in that, The step of semantically encoding each target log to be parsed to obtain a semantic vector for each target log to be parsed includes: Semantic segmentation is performed on any target log to be parsed to obtain multiple semantic segments corresponding to the target log to be parsed; Determine the semantic segment weight of each semantic segment corresponding to any target log to be parsed; the semantic segment weight is determined based on at least one of the following: semantic segment position, semantic segment length, and semantic segment keywords; Each semantic segment is semantically vector-encoded to obtain a semantic segment vector corresponding to each semantic segment; Based on the semantic segment vector corresponding to each semantic segment and the semantic segment weight of each semantic segment, the semantic vector of any target log to be parsed is obtained.

4. The method as described in claim 1, characterized in that, The step of replacing the fields in each log entry that match a preset field type with the field type identifier corresponding to the preset field type to obtain the string corresponding to each log entry includes: Each log entry to be parsed is semantically segmented to obtain multiple semantic segments corresponding to each log entry to be parsed; For any log to be parsed, a target semantic segment is determined from the multiple semantic segments of the log to be parsed. Replace the field in the target semantic segment corresponding to each log to be parsed that matches the preset field type with the field type identifier corresponding to the preset field type to obtain the string corresponding to each target semantic segment; The step involves hashing and bucketing the multiple log entries based on the string corresponding to each log entry to obtain at least one log bucket, including: Based on the string corresponding to each target semantic segment, the multiple logs to be parsed are hashed and bucketed to obtain at least one log bucket.

5. The method as described in claim 1, characterized in that, After hashing and bucketing the multiple log files based on the string corresponding to each log file to be parsed, to obtain at least one log bucket, the process further includes: Based on the number of logs to be parsed in each log bucket, the logs to be parsed in each log bucket are clustered to obtain at least one log unit bucket corresponding to each log bucket. The step involves inputting at least one log entry to be parsed from each log bucket into a large language model, so that the large language model performs comparative parsing processing on at least one log entry to be parsed in each log bucket to obtain a log template corresponding to each log entry to be parsed in each log bucket, including: At least one log entry to be parsed from each log unit bucket is input into the large language model, so that the large language model performs comparative parsing processing on at least one log entry to be parsed from each log unit bucket to obtain the log template corresponding to each log entry to be parsed from each log unit bucket.

6. The method as described in claim 1, characterized in that, The step involves inputting at least one log entry to be parsed from each log bucket into a large language model, so that the large language model performs comparative parsing processing on at least one log entry to be parsed in each log bucket to obtain a log template corresponding to each log entry to be parsed in each log bucket, including: Select any log to be parsed from each log bucket, and determine the semantic vector of any log to be parsed corresponding to each log bucket; Based on the semantic vector of any log to be parsed corresponding to each log bucket, semantic similarity retrieval is performed in the prompt word database to obtain at least one historical log corresponding to each log bucket and a historical log template corresponding to the at least one historical log. The at least one historical log corresponding to each log bucket and the historical log template corresponding to the at least one historical log are used as log samples. At least one log entry to be parsed from each log bucket, and a log sample corresponding to each log bucket, are input into the large language model. The large language model then performs comparative parsing on at least one log entry to be parsed in each log bucket based on the log sample corresponding to each log bucket, thereby obtaining a log template corresponding to each log entry to be parsed in each log bucket.

7. The method as described in claim 1, characterized in that, Before inputting at least one log entry to be parsed from each log bucket into the large language model, the process also includes: The target log to be parsed is determined from each log bucket, and the constant field sequence corresponding to each target log is extracted; the constant field sequence is composed of fields in the target log that do not match the preset field type; Query the log template library to see if there is a historical log template that matches the sequence of constant fields corresponding to any target log to be parsed; If a historical log template exists that matches the sequence of constant fields corresponding to any of the target logs to be parsed, then the historical log template that matches the sequence of constant fields corresponding to any of the target logs to be parsed will be used as the log template for each log to be parsed in the log bucket corresponding to any of the target logs to be parsed. If no historical log template matches the sequence of constant fields corresponding to any of the target logs to be parsed, then the step of inputting at least one log from each log bucket into the large language model is performed.

8. The method as described in claim 6 or 7, characterized in that, The method further includes: The historical parsing logs, the corresponding historical log templates, and the semantic vectors of the historical parsing logs are associated and stored in the prompt word database; and / or, Extract the constant field sequence corresponding to the historical parsing log, and associate and store the constant field sequence corresponding to the historical parsing log and the historical log template corresponding to the historical parsing log in the log template library.

9. A log parsing device, characterized in that, The device includes: The acquisition unit is used to retrieve multiple log entries to be parsed. The replacement unit is used to replace the field in each log to be parsed that matches the preset field type with the field type identifier corresponding to the preset field type, so as to obtain the string corresponding to each log to be parsed; The processing unit is used to perform hash bucketing on the multiple logs to be parsed based on the string corresponding to each log to be parsed, to obtain at least one log bucket; a log bucket includes at least one log to be parsed, and the strings corresponding to each log to be parsed in the same log bucket are the same; The processing unit is further configured to input at least one log to be parsed from each log bucket into the large language model, so that the large language model performs comparative parsing processing on at least one log to be parsed in each log bucket to obtain a log template corresponding to each log to be parsed in each log bucket.

10. A computer device, characterized in that, The computer device includes a memory, a communication interface, and a processor, wherein the memory, the communication interface, and the processor are interconnected; the memory stores a computer program, and the processor calls the computer program stored in the memory to implement the log parsing method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the log parsing method as described in any one of claims 1 to 8.

12. A computer program product, characterized in that, The computer program product includes a computer program stored in a computer storage medium; the processor of the computer device reads the computer program from the computer storage medium, and the processor executes the computer program, causing the computer device to perform the log parsing method as described in any one of claims 1 to 8.

Images