Methods, systems, storage media, and program products for determining log templates

Abstract

This application relates to the field of computers, and more specifically to a method for determining a log template, a system for determining a log template, a computer-readable storage medium for implementing the above method, and a computer program product. The method for determining a log template according to one aspect of this application includes: obtaining a semantic feature vector for each log text in a plurality of log texts; dividing the plurality of log texts into a plurality of log text categories based on clustering results of clustering processing of the semantic feature vectors; and determining a text subunit of each log text in a set of log texts under each log text category, and determining a log template for the set of log texts based at least on the longest common subsequence of the set of log texts determined by the text subunits.

Description

Technical Field

[0001] This application relates to the field of computers, and more specifically to a method for determining a log template, a system for determining a log template, a computer-readable storage medium for implementing the above method, and a computer program product. Background Technology

[0002] Logs are text data automatically generated and recorded in a specific format by computer systems, applications, network devices, etc., during operation. They are used to trace system behavior, state changes, and event occurrences. In daily operations and maintenance, it is necessary to process and analyze massive amounts of log data to achieve fault diagnosis, network intrusion detection, system performance optimization, and other tasks.

[0003] Log template extraction involves reconstructing the log template by extracting common text portions from multiple logs. Analyzing the distribution of log templates across massive amounts of logs allows for efficient understanding of the overall system's operational status. Log templates have significant application value in daily operations and maintenance.

[0004] Currently, log template extraction is generally based on preset template rules or relies on the natural language processing capabilities of large models. However, log template extraction based on preset template rules cannot perform semantic-level analysis of logs, resulting in the inability to extract readable log templates. Log template extraction relying on the natural language processing capabilities of large models has high processing resource requirements and overhead, making it unsuitable for scenarios involving large-scale log processing. Furthermore, traditional log template extraction methods that rely on template libraries require continuous investment of manpower and resources for data analysis and annotation, thereby refining the template library. Summary of the Invention

[0005] To address or at least alleviate one or more of the above problems, the following technical solutions are provided.

[0006] According to a first aspect of this application, a method for determining a log template is provided, the method comprising the following steps: obtaining a semantic feature vector for each log text in a plurality of log texts; dividing the plurality of log texts into a plurality of log text categories based on clustering results of clustering processing of the semantic feature vectors; and determining a text subunit of each log text in a set of log texts under each log text category, and determining a log template for the set of log texts based at least on the longest common subsequence of the set of log texts determined by the text subunits.

[0007] According to an embodiment of the method for determining a log template, before obtaining the semantic feature vector of each log text in a plurality of log texts, the method further includes: performing deduplication processing on the plurality of log texts to obtain a plurality of deduplicated log texts; matching a target feature field from the plurality of deduplicated log texts based on a preset regular expression matching rule; and replacing the target feature field with a standard text field.

[0008] According to the method for determining a log template according to one embodiment or any of the above embodiments of this application, the target feature field includes one or more of the following fields: IP address field, timestamp field, MAC address field.

[0009] According to one embodiment or any of the above embodiments of this application, the method for determining a log template includes obtaining the semantic feature vector of each log text among a plurality of log texts by: inputting the plurality of log texts into a pre-trained text embedding model to obtain the semantic feature vector of each log text among the plurality of log texts.

[0010] The method for determining a log template according to one embodiment or any of the above embodiments of this application, wherein dividing the plurality of log texts into a plurality of log text categories based on the clustering result of clustering the semantic feature vectors includes: recursively performing binary clustering on the semantic feature vectors to obtain a binary clustering result; determining whether a clustering termination condition is met based on the binary clustering result; and, in response to determining that the clustering termination condition is met, dividing the plurality of log texts into a plurality of log text categories based on the binary clustering result.

[0011] According to one embodiment or any of the above embodiments of the present application, the method for determining a log template, wherein recursively performing binary clustering processing on the semantic feature vector to obtain a binary clustering result includes: recursively performing binary clustering processing on the semantic feature vector using an unsupervised clustering algorithm to obtain a binary clustering result.

[0012] According to one embodiment or any of the above embodiments of the present application, the method for determining a log template, wherein determining whether the clustering termination condition is met based on the binary clustering result includes determining whether at least one of the following conditions is met: the number of semantic feature vectors within a cluster in the binary clustering result is less than or equal to a number threshold; the similarity between semantic feature vectors within a cluster in the binary clustering result is greater than or equal to a similarity threshold.

[0013] According to an embodiment of this application or any of the above embodiments, the method for determining a log template, wherein determining a text subunit of each log text in a set of log texts under each log text category includes: performing word segmentation on each log text in the set of log texts under each log text category to obtain one or more word units; and determining the one or more word units as the text subunit.

[0014] According to the method for determining a log template according to one embodiment or any of the above embodiments of this application, wherein the set of log texts includes a plurality of log texts, and the longest common subsequence of the set of log texts is determined by: comparing one or more text subunits of one log text with one or more text subunits of another log text in pairs to determine one or more common text subunits of the one log text and the other log text; and determining the common portion in the one or more common text subunits as the longest common subsequence of the set of log texts.

[0015] The method for determining a log template according to one embodiment or any of the above embodiments of this application, wherein determining the log template of the set of log texts based at least on the longest common subsequence of the set of log texts determined by the text subunits includes: determining the longest common subsequence of the set of log texts as the invariant portion of the log template; identifying the portion of the set of log texts other than the longest common subsequence as the variable portion of the log template; and determining the log template of the set of log texts based on the combination of the invariant portion and the variable portion.

[0016] The method for determining log templates according to one embodiment or any of the above embodiments of this application further includes: selectively merging some or all of the log templates in the plurality of log templates based on whether there is an inclusion relationship between the plurality of log templates corresponding to the sets of log texts under the plurality of log text categories.

[0017] According to a second aspect of this application, a system for determining a log template is provided, the system comprising: a memory; a processor coupled to the memory; and a computer program stored on the memory and running on the processor, the execution of the computer program causing the following operations: acquiring a semantic feature vector for each of a plurality of log texts; dividing the plurality of log texts into a plurality of log text categories based on clustering results of clustering processing of the semantic feature vectors; and determining a text subunit of each log text in a set of log texts under each log text category, determining a log template for the set of log texts based at least on the longest common subsequence of the set of log texts determined by the text subunits.

[0018] According to a third aspect of this application, a computer-readable storage medium is provided, comprising instructions that, when executed, perform the steps of the method for determining a log template according to a first aspect of this application.

[0019] According to a fourth aspect of this application, a computer program product is provided, the computer program product including instructions that, when executed by a processor, implement the steps of the method for determining a log template according to a first aspect of this application.

[0020] The log template determination scheme according to one or more embodiments of this application performs semantic-level feature analysis on log text by obtaining semantic feature vectors of log text, making the determined log template meaningful and readable, facilitating subsequent processing and analysis. By clustering the semantic feature vectors of log text, the log text is divided into multiple log text categories, and within each category, the log template for the corresponding set of log texts is determined based on at least the longest common subsequence of the set of log texts determined by the text subunits. This saves processing resources and improves processing efficiency, making it suitable for scenarios involving the processing of large amounts of logs. Furthermore, the log template determination scheme according to one or more embodiments of this application does not require manual annotation of samples and features, nor does it require separate model training. It has strong generalization ability and can be applied to determining log templates for logs from different domains (e.g., network logs, system logs, application logs), different log types (e.g., raw operation logs, alarm logs, event logs), and different applications. Attached Figure Description

[0021] The above and / or other aspects and advantages of this application will become clearer and more readily understood from the following description taken in conjunction with the accompanying drawings, in which the same or similar elements are denoted by the same reference numerals. The drawings include: Figure 1 A flowchart illustrating a method for determining a log template according to one or more embodiments of this application is shown.

[0022] Figure 2 A flowchart illustrating a method for classifying multiple log texts into multiple log text categories according to an embodiment of this application is shown.

[0023] Figure 3 A flowchart illustrating a method for determining a log template of a set of log texts based at least on the longest common subsequence of the set of log texts, according to one or more embodiments of this application, is shown.

[0024] Figure 4 A block diagram of an apparatus for determining a log template according to one or more embodiments of this application is shown.

[0025] Figure 5 A block diagram of a system for determining a log template according to one or more embodiments of this application is shown. Detailed Implementation

[0026] The present application will now be described more fully with reference to the accompanying drawings, which illustrate exemplary embodiments thereof. However, the present application may be implemented in various forms and should not be construed as being limited to the embodiments given herein. The foregoing embodiments are intended to make the disclosure herein complete and thorough, so as to more fully convey the scope of protection of the present application to those skilled in the art.

[0027] In this specification, terms such as “comprising” and “including” indicate that, in addition to having the units and steps that are directly and explicitly stated in the specification and claims, the technical solution of this application does not exclude the presence of other units and steps that are not directly or explicitly stated.

[0028] Unless otherwise specified, terms such as “first” and “second” do not indicate the order of units in terms of time, space, size, etc., but are merely used to distinguish between units.

[0029] In the following, various exemplary embodiments according to this application will be described in detail with reference to the accompanying drawings.

[0030] Figure 1 A flowchart illustrating a method for determining a log template according to one or more embodiments of this application is shown.

[0031] like Figure 1 As shown, in step S101, the semantic feature vector of each log text in the multiple log texts is obtained.

[0032] Optionally, in step S101, multiple log texts can be input into a pre-trained text embedding model to obtain the semantic feature vector of each log text. In one embodiment, the pre-trained text embedding model can be implemented as an all-MiniLM model, such as the all-MiniLM-L6-v2 model, which is based on the BERT architecture and uses MiniLM knowledge distillation technology to significantly reduce the model size while maintaining high performance, and controls the dimensionality of the semantic feature vector to facilitate subsequent clustering processing. For example, the semantic feature vector of the log text output by the all-MiniLM-L6-v2 model has a dimension of 384, which can retain sufficient semantic information and ensure the iteration speed of the subsequent clustering algorithm.

[0033] In one embodiment, before obtaining the semantic feature vector of each log text from multiple log texts, the multiple log texts can be preprocessed. Preprocessing may include deduplication and regular expression matching. Specifically, multiple log texts can be deduplicated to obtain multiple deduplicated log texts. Based on preset regular expression matching rules, target feature fields are matched from the multiple deduplicated log texts, and the target feature fields are replaced with standard text fields. In one embodiment, the target feature fields may include IP address fields, timestamp fields, MAC address fields, etc., i.e., specific IP address values, timestamp values, MAC address values, etc., and the corresponding standard text fields may include the text "IP address," "timestamp," "MAC address," etc.

[0034] In one embodiment, the hash value of each log text in multiple log texts can be calculated. Log texts with the same hash value are identified as duplicate log texts and deduplicated, so that the hash values ​​of the deduplicated log texts are different from each other. In one embodiment, the preset regular expression matching rule can be a combination of ordinary characters (e.g., text, numbers) and special metacharacters, which is used to define the matching rule to quickly and accurately locate the content that matches the rule from multiple deduplicated log texts (e.g., IP address field, timestamp field, MAC address field, etc.). For example, the log text could be 2026-01-12 15:30:45 [ERROR] 192.168.1.1 failed to read data. The target feature fields 2026-01-12 15:30:45 and 192.168.1.1 can be matched based on preset regular expression matching rules. The target feature fields 2026-01-12 15:30:45 and 192.168.1.1 can be replaced with the standard text fields "timestamp" and "IP address" to obtain the timestamp [ERROR] IP address failed to read data.

[0035] In step S103, multiple log texts are divided into multiple log text categories based on the clustering results of the semantic feature vector clustering process.

[0036] Optionally, in step S103, the semantic feature vectors can be recursively subjected to binary clustering to obtain binary clustering results. Based on the binary clustering results, it is determined whether a clustering termination condition is met. In response to the determination that the clustering termination condition is met, multiple log texts are divided into multiple log text categories based on the binary clustering results. In one embodiment, an unsupervised clustering algorithm can be used to recursively perform binary clustering on the semantic feature vectors to obtain binary clustering results. By using an unsupervised clustering algorithm to recursively perform binary clustering on the semantic feature vectors, there is no need for manual labeling of text categories or manual pre-setting of the number of categories. This is applicable to scenarios where the log text output changes frequently and can also handle large amounts of log text. In one embodiment, the clustering termination condition may include at least one of the following conditions: the number of semantic feature vectors within a cluster in the binary clustering result is less than or equal to a number threshold; the similarity between semantic feature vectors within a cluster in the binary clustering result is greater than or equal to a similarity threshold. When it is determined based on the binary clustering results that at least one of the above conditions is met, it can be determined that the clustering termination condition is met. For example, the quantity threshold can be set to 2, and the similarity threshold can be set to 0.96. For example, the similarity between semantic feature vectors can be calculated based on cosine similarity, for example, for two semantic feature vectors... cosine similarity It can be calculated using the following formula (1): Formula (1) In one embodiment, the K-means algorithm can be used to cluster semantic feature vectors, where K is set to 2. In the first step, the K-means algorithm can be used to cluster the semantic feature vectors of multiple log texts. Specifically, two semantic feature vectors are randomly selected as initial centroids. The Euclidean distance from each semantic feature vector to the two centroids is calculated. Based on the Euclidean distance, the semantic feature vectors are assigned to two classes. The centroids of the two classes are redefined based on the mean of the semantic feature vectors within each class. This process is iteratively updated until the centroid positions are stable, resulting in two clusters: semantic feature vectors of class 0 and semantic feature vectors of class 1. In the second step, the semantic feature vectors of class 0 can be used for clustering. The process in the first step is repeated. If the clustering termination condition is met, clustering stops, and class 0 becomes the clustering result. Simultaneously, the semantic feature vectors of class 1 can be used for clustering. The process in the first step is repeated. If the clustering termination condition is not met, two more clusters can be obtained: semantic feature vectors of subclass 1-0 and semantic feature vectors of subclass 1-1. In the third step, the semantic feature vectors of sub-category 1-0 and sub-category 1-1 obtained in the second step can be used as the objects for clustering. The processing in the second step is repeated until the clustering termination condition is met, and the clustering result is obtained. The K-means algorithm has strong interpretability and fast convergence speed. Considering that the value of K in the clustering algorithm cannot be preset, and the uncertainty brought about by random centroid initialization, the idea of ​​binary search is used to set K to 2 for optimization. There is no need to preset the number of categories and the feature distribution. It automatically converges to a reasonable number of categories through recursive splitting.

[0037] In one embodiment, corresponding identification information can be set for each log text among multiple log texts, thereby establishing an association between the semantic feature vector of the obtained log text and the corresponding log text identification information. After clustering the semantic feature vector to obtain the clustering result, the log text corresponding to the semantic feature vector of the log text under each cluster category can be determined according to the above association, thereby dividing multiple log texts into multiple log text categories.

[0038] In step S105, the text sub-units of each log text in the set of log texts under each log text category are determined, and the log template of the set of log texts is determined based at least on the longest common subsequence of the set of log texts determined by the text sub-units.

[0039] Optionally, in step S105, each log text in the set of log texts under each log text category can be segmented to obtain one or more word units, and the obtained one or more word units are determined as text subunits. For example, the preprocessed log text can be a timestamp [ERROR] IP address data read failure; the word units obtained after segmentation can include "timestamp", "ERROR", "IP address", and "data read failure". For example, the jieba segmentation tool can be used to segment each log text in the set of log texts under each log text category to obtain one or more word units.

[0040] In one embodiment, the Longest Common Subsequence (LCS) algorithm based on word units can be used to determine the longest common subsequence of the log text set based on text subunits of each log text in the set. In another embodiment, a dynamic programming approach can be used to determine the longest common subsequence of the log text set based on text subunits of each log text in the set.

[0041] In one embodiment, the set of log texts may include multiple log texts. One or more text sub-units of one log text can be compared pairwise with one or more text sub-units of another log text to determine one or more common text sub-units between the log texts. The common portion of these common text sub-units is then determined as the longest common subsequence of the set of log texts. For example, the set of log texts may include log text 1, log text 2, and log text 3. Pairwise comparisons can be made between log text 1 and log text 2, log text 1 and log text 3, and log text 2 and log text 3 to determine that the common text sub-units A and B between log text 1 and log text 2, A and C between log text 1 and log text 3, and A and D between log text 2 and log text 3. This determines that the common portion A of the common text sub-units is the longest common subsequence of the set of log texts.

[0042] In one embodiment, the set of log texts may include a first log text, a second log text, and a third log text. The longest common subsequence of the set of log texts can be determined by: comparing one or more first text sub-units of the first log text with one or more second text sub-units of the second log text; generating the longest common subsequence of the first log text and the second log text based on the same text sub-units in the one or more first text sub-units and the one or more second text sub-units; comparing one or more common text sub-units of the longest common subsequence of the first log text and the second log text with one or more third text sub-units of the third log text; and determining the longest common subsequence of the set of log texts based on the same text sub-units in the one or more common text sub-units and the one or more third text sub-units. For example, the text sub-units of the first log text may include a timestamp, ERROR, IP address, MAC address, login failure, and incorrect username / password; the text sub-units of the second log text may include a timestamp, ERROR, IP address, MAC address, login failure, and incorrect verification code; and the text sub-units of the third log text may include a timestamp, ERROR, IP address, MAC address, login failure, and expired password. The text sub-units of the first log text and the text sub-units of the second log text can be compared, and identical elements in the text sub-units of the first and second log texts can be included. The text sub-unit is determined as the longest common subsequence of the first and second log texts, i.e., the longest common subsequence of the first and second log texts is {timestamp, ERROR, IP address, MAC address, login failed}. Then, the text sub-units in the common subsequence are compared with the text sub-units of the third log text. The text sub-units that are identical in the common subsequence and the text sub-units of the third log text are determined as the longest common subsequence of the log text set, i.e., the longest common subsequence of the log text set is {timestamp, ERROR, IP address, MAC address, login failed}. It should be noted that the number of log texts in the log text set is not limited to the three in the above embodiments. Without departing from the spirit and scope of this application, the number of log texts in the log text set can be more or less than three, and the longest common subsequence of the log text set can also be determined by the pairwise comparison method in the above embodiments.

[0043] Optionally, in step S105, the longest common subsequence of the log text set can be determined as the invariant part of the log template, the part of the log text set excluding the longest common subsequence can be identified as the variable part of the log template, and the log template of the log text set can be determined based on the combination of the invariant part and the variable part. As an example, the variable part of the log template can be identified as [X], and the log template can be as follows: Log template: EVENT_ID[[X]], Link[[X]][[IPv4 address]] failed to read data, link closed, return value[[X]], fd[[X]] The above log template can be determined from the following plurality of logs according to the method for determining log templates in one or more embodiments of this application: Log 1: EVENT_ID

[6005] , Link[2][11.111.11.11:12345] failed to read data, link closed, return value

[62] , fd

[102] Log 2: EVENT_ID

[6004] , Link [100222][99.9.99.999:45678] failed to read data, link closed, return value

[62] , fd

[350] Log 3: EVENT_ID

[6004] , Link [100226][66.6.66.666:56789] failed to read data, link closed, return value

[62] , fd

[185] Log 4: EVENT_ID

[6005] , Link

[143] [77.7.77.777:12345] failed to read data, closed the link, return value

[62] , fd

[68] .

[0044] In one embodiment, after determining the multiple log templates corresponding to the collection of log texts under multiple log text categories, some or all of the log templates can be selectively merged based on whether there is an inclusion relationship between them. For example, if an inclusion relationship is found between two log templates, the two log templates can be merged; otherwise, they are not merged. In another embodiment, after determining the log templates for the collection of log texts, the proportion of the variable portion in the log template can be determined. When this proportion is greater than a preset threshold (e.g., 50%), the log template can be discarded.

[0045] In one or more embodiments of this application, by utilizing a word-based LCS algorithm to determine the longest common subsequence of the log text set based on the text subunits of each log text in the set, thereby determining the log template of the log text set, the semantic fragmentation problem of traditional character-level LCS can be avoided. This ensures that the log template is a complete structure that conforms to business semantics, rather than meaningless character concatenation, facilitating subsequent processing and analysis of the log template. For example, the determined log template can be further processed and analyzed using a large model. Operations personnel can summarize the log template into regular expressions to achieve efficient log analysis. By analyzing the distribution of log templates in each batch of logs, further system anomaly detection can be performed. By extracting log templates and transforming them into solutions using the unit-based LCS algorithm, the algorithm automatically solves and converges to the final template through dynamic programming. This eliminates the need for manually writing regular expressions, labeling variable positions, or defining template rules, making it applicable to different fields and log types. By utilizing the unit-based LCS algorithm to solve for the longest common subsequence of a set of log texts pairwise, it can save processing resources and improve processing efficiency. For example, it can determine the log template for 3000 alarm logs within 10 seconds using the CPU.

[0046] The method for determining log templates according to one or more embodiments of this application performs semantic-level feature analysis on log text by obtaining semantic feature vectors of log text, making the determined log templates meaningful and readable, facilitating subsequent processing and analysis. By clustering the semantic feature vectors of log text, the log text is divided into multiple log text categories, and within each category, the log template for the corresponding set of log texts is determined based on at least the longest common subsequence of the set of log texts determined by the text subunits. This saves processing resources and improves processing efficiency, making it suitable for scenarios involving the processing of large volumes of logs. Furthermore, the method for determining log templates according to one or more embodiments of this application does not require manual annotation of samples and features, nor does it require separate model training. It has strong generalization ability and can be applied to determining log templates for logs from different domains (e.g., network logs, system logs, application logs), different log types (e.g., raw operation logs, alarm logs, event logs), and different applications.

[0047] Figure 2 A flowchart illustrating a method for classifying multiple log texts into multiple log text categories according to an embodiment of this application is shown.

[0048] like Figure 2As shown, in step S201, the semantic feature vectors of multiple log texts are used as input to the K-means algorithm. The K-means algorithm is then used to perform binary clustering on the semantic feature vectors to obtain the binary clustering result. Optionally, the binary clustering result may include the semantic feature vector of category 0 and the semantic feature vector of category 1.

[0049] In step S203, it is determined whether the clustering termination condition is met based on the binary clustering result. Optionally, the determination can be based on the semantic feature vectors of category 0 and category 1. If the semantic feature vector of category 0 meets the clustering termination condition, the process proceeds to step S205. If the semantic feature vector of category 1 does not meet the clustering termination condition, the process returns to step S201, using the semantic feature vector of category 1 as input to the K-means algorithm. The K-means algorithm is then used to perform binary clustering on the semantic feature vector again to obtain the binary clustering result. Optionally, the clustering termination condition may include at least one of the following conditions: the number of semantic feature vectors within a cluster in the binary clustering result is less than or equal to a number threshold; the similarity between semantic feature vectors within a cluster in the binary clustering result is greater than or equal to a similarity threshold. If at least one of the above conditions is met based on the binary clustering result, the clustering termination condition is deemed satisfied.

[0050] In one embodiment, when the similarity between the semantic feature vectors of category 0 is greater than or equal to the similarity threshold, and when the number of semantic feature vectors of category 1 is less than or equal to the number threshold, it can be determined that the semantic feature vectors of category 0 and the semantic feature vectors of category 1 both satisfy the clustering termination condition, and then step S205 can be entered.

[0051] In one embodiment, when the number of semantic feature vectors of category 1 exceeds a threshold, it can be determined that the semantic feature vectors of category 1 do not meet the clustering termination condition. In this case, the semantic feature vectors of category 1 can be used as input to the K-means algorithm and returned to step S201. The K-means algorithm is then used to perform binary clustering on the semantic feature vectors of category 1 again to obtain a binary clustering result. This result can include the semantic feature vectors of category 1-0 and category 1-1. Next, in step S203, it can be determined whether the clustering termination condition is met based on the semantic feature vectors of category 1-0 and category 1-1. If both semantic feature vectors of category 1-0 and category 1-1 meet the clustering termination condition, the process proceeds to step S205; otherwise, it returns to step S201, using the semantic feature vectors corresponding to the categories that do not meet the clustering termination condition as input to the K-means algorithm. The K-means algorithm is then used to perform binary clustering on these semantic feature vectors again to obtain a binary clustering result.

[0052] It should be noted that steps S201 and S203 can be executed recursively until all binary clustering results satisfy the clustering termination condition.

[0053] In step S205, multiple log texts are divided into multiple log text categories based on the binary clustering results.

[0054] Figure 3 A flowchart illustrating a method for determining a log template of a set of log texts based at least on the longest common subsequence of the set of log texts, according to one or more embodiments of this application, is shown.

[0055] like Figure 3 As shown, in step S301, one or more text sub-units of one log text in a set of log texts are compared pairwise with one or more text sub-units of another log text to determine one or more common text sub-units of the one log text and the other log text.

[0056] In step S303, the common part in one or more common text subunits is determined as the longest common subsequence of the set of log texts.

[0057] In step S305, the longest common subsequence of the log text set is determined as the non-variable part of the log template, and the part of the log text set other than the longest common subsequence is identified as the variable part of the log template.

[0058] In step S307, a log template for the set of log texts is determined based on the combination of the invariant and variable parts.

[0059] Figure 4 A block diagram of an apparatus for determining a log template according to one or more embodiments of this application is shown.

[0060] like Figure 4 As shown, the apparatus 400 for determining the log template includes a data processing module 410, a clustering module 420, and a template generation module 430.

[0061] The data processing module 410 is configured to obtain the semantic feature vector of each log text from multiple log texts.

[0062] Optionally, before obtaining the semantic feature vector of each log text from multiple log texts, the data processing module 410 can preprocess the multiple log texts. Preprocessing may include deduplication and regular expression matching. Specifically, the data processing module 410 can deduplicate the multiple log texts to obtain multiple deduplicated log texts, match target feature fields from the multiple deduplicated log texts based on preset regular expression matching rules, and replace the target feature fields with standard text fields. In one embodiment, the target feature fields may include IP address fields, timestamp fields, MAC address fields, etc., i.e., specific IP address values, timestamp values, MAC address values, etc., and the corresponding standard text fields may include the text "IP address," "timestamp," "MAC address," etc.

[0063] The clustering module 420 is configured to divide multiple log texts into multiple log text categories based on the clustering results of clustering processing of semantic feature vectors.

[0064] Optionally, the clustering module 420 can recursively perform binary clustering on the semantic feature vector to obtain binary clustering results, determine whether the clustering termination condition is met based on the binary clustering results, and in response to the determination that the clustering termination condition is met, divide multiple log texts into multiple log text categories based on the binary clustering results.

[0065] The template generation module 430 is configured to determine the text subunit of each log text in the set of log texts under each log text category, and to determine the log template of the set of log texts based at least on the longest common subsequence of the set of log texts determined by the text subunits.

[0066] Optionally, the template generation module 430 can perform word segmentation on each log text in the set of log texts under each log text category to obtain one or more word units, and determine the one or more word units as text sub-units. Optionally, the template generation module 430 can use the word unit-based LCS algorithm to determine the longest common subsequence of the set of log texts based on the text sub-units of each log text in the set of log texts. The longest common subsequence of the set of log texts can be determined as the invariant part of the log template, and the part of the set of log texts other than the longest common subsequence can be identified as the variable part of the log template. The log template of the set of log texts can be determined based on the combination of the invariant part and the variable part.

[0067] Figure 5 A block diagram of a system for determining a log template according to one or more embodiments of this application is shown.

[0068] like Figure 5 As shown, the system 500 for determining a log template includes a memory 510, a processor 520, and a computer program 530 stored in the memory 510 and executable on the processor 520. The processor 520 executes the computer program 530 to implement the method for determining a log template according to one aspect of this application.

[0069] This application can also be implemented as a computer-readable storage medium including instructions that, when executed, perform a method for determining a log template according to one aspect of this application. Alternatively, this application can also be implemented as a computer-readable storage medium including instructions that, when executed, perform a method for determining potential users according to one aspect of this application.

[0070] Where applicable, the various embodiments provided in this application may be implemented using hardware, software, or a combination of hardware and software. Furthermore, where applicable, without departing from the scope of this application, the various hardware and / or software components described herein may be combined into composite components comprising software, hardware, and / or both. Where applicable, without departing from the scope of this application, the various hardware and / or software components described herein may be divided into sub-components comprising software, hardware, or both. Additionally, where applicable, it is contemplated that software components may be implemented as hardware components, and vice versa.

[0071] The software (such as program code and / or data) according to this application can be stored on one or more computer storage media. It is also contemplated that the software identified herein can be implemented using one or more networked and / or otherwise general-purpose or special-purpose computers and / or systems. Where applicable, the order of the various steps described herein can be changed, combined into compound steps, and / or divided into sub-steps to provide the features described herein.

[0072] The embodiments and examples presented herein are provided to best illustrate embodiments of this application and its particular applications, thereby enabling those skilled in the art to implement and use this application. However, those skilled in the art will understand that the above description and examples are provided for ease of illustration and example only. The descriptions presented are not intended to cover all aspects of this application or to limit this application to the precise forms disclosed.

Claims

1. A method for determining a log template, characterized in that, The method includes the following steps: Obtain the semantic feature vector of each log text from multiple log texts; Based on the clustering results of the clustering processing of the semantic feature vectors, the multiple log texts are divided into multiple log text categories; and Determine the text subunit of each log text in the set of log texts under each log text category, and determine the log template of the set of log texts based at least on the longest common subsequence of the set of log texts determined by the text subunits.

2. The method according to claim 1, wherein before obtaining the semantic feature vector of each log text among the plurality of log texts, the method further comprises: The multiple log texts are deduplicated to obtain multiple deduplicated log texts; Based on preset regular expression matching rules, target feature fields are obtained by matching from the multiple deduplicated log texts; as well as Replace the target feature field with a standard text field.

3. The method according to claim 2, wherein the target feature field includes one or more of the following fields: IP address field, timestamp field, MAC address field.

4. The method according to claim 1, wherein obtaining the semantic feature vector of each log text among the plurality of log texts includes: The multiple log texts are input into a pre-trained text embedding model to obtain the semantic feature vector of each log text.

5. The method according to claim 1, wherein dividing the plurality of log texts into plurality of log text categories based on the clustering result of clustering the semantic feature vectors includes: The semantic feature vector is recursively subjected to binary clustering to obtain the binary clustering result; Based on the binary clustering results, determine whether the clustering termination condition is met; as well as In response to the determination that the clustering termination condition is met, the multiple log texts are divided into multiple log text categories based on the binary clustering results.

6. The method according to claim 5, wherein recursively performing binary clustering on the semantic feature vector to obtain the binary clustering result includes: The semantic feature vector is recursively subjected to binary clustering using an unsupervised clustering algorithm to obtain the binary clustering result.

7. The method according to claim 5, wherein determining whether the clustering termination condition is met based on the binary clustering result includes determining whether at least one of the following conditions is met: The number of semantic feature vectors within a cluster in the binary clustering result is less than or equal to the number threshold. The similarity between semantic feature vectors within the clusters in the binary clustering results is greater than or equal to the similarity threshold.

8. The method of claim 1, wherein determining the text subunit of each log text in the set of log texts under each log text category comprises: Each log text in the set of log texts under each log text category is segmented to obtain one or more word units; The one or more word units are identified as the text subunits.

9. The method of claim 1, wherein the set of log texts comprises a plurality of log texts, and the longest common subsequence of the set of log texts is determined by the following method: By comparing one or more text sub-units of one log text with one or more text sub-units of another log text, one or more common text sub-units of the one log text and the other log text are determined. The common portion of the one or more common text subunits is determined as the longest common subsequence of the set of log texts.

10. The method of claim 1, wherein determining the log template of the set of log texts based at least on the longest common subsequence of the set of log texts determined by the text subunits comprises: The longest common subsequence of the set of log texts is determined as the invariant part of the log template; The portion of the log text set excluding the longest common subsequence is identified as the variable portion of the log template; as well as A log template for the set of log texts is determined based on a combination of the invariant and variable parts.

11. The method of claim 10, wherein the method further comprises: Based on whether there is an inclusion relationship between the multiple log templates corresponding to the collection of log texts under the multiple log text categories, some or all of the log templates in the multiple log templates are selectively merged.

12. A system for determining a log template, characterized in that, The system includes: Memory; A processor coupled to the memory; and A computer program stored in the memory and running on the processor, the execution of which causes the following operations: Obtain the semantic feature vector of each log text from multiple log texts; Based on the clustering results of the clustering processing of the semantic feature vectors, the multiple log texts are divided into multiple log text categories; and Determine the text subunit of each log text in the set of log texts under each log text category, and determine the log template of the set of log texts based at least on the longest common subsequence of the set of log texts determined by the text subunits.

13. The system of claim 12, wherein the execution of the computer program causes the following operations to be performed before obtaining the semantic feature vector of each of the plurality of log texts: The multiple log texts are deduplicated to obtain multiple deduplicated log texts; Based on preset regular expression matching rules, target feature fields are obtained by matching from the multiple deduplicated log texts; and Replace the target feature field with a standard text field.

14. The system according to claim 13, wherein the target feature field includes one or more of the following fields: IP address field, timestamp field, MAC address field.

15. The system of claim 12, wherein the execution of the computer program results in obtaining a semantic feature vector for each of a plurality of log texts, comprising: The multiple log texts are input into a pre-trained text embedding model to obtain the semantic feature vector of each log text.

16. The system of claim 12, wherein the execution of the computer program causes the plurality of log texts to be divided into a plurality of log text categories based on the clustering result of clustering processing of the semantic feature vectors, comprising: The semantic feature vector is recursively subjected to binary clustering to obtain the binary clustering result; Based on the binary clustering results, determine whether the clustering termination condition is met; as well as In response to the determination that the clustering termination condition is met, the multiple log texts are divided into multiple log text categories based on the binary clustering results.

17. The system of claim 16, wherein the execution of the computer program causes recursively performing binary clustering on the semantic feature vector to obtain a binary clustering result, comprising: The semantic feature vector is recursively subjected to binary clustering using an unsupervised clustering algorithm to obtain the binary clustering result.

18. The system of claim 16, wherein the execution of the computer program causes the determination based on the binary clustering result that the clustering termination condition is satisfied, including determining that at least one of the following conditions is satisfied: The number of semantic feature vectors within a cluster in the binary clustering result is less than or equal to the number threshold. The similarity between semantic feature vectors within the clusters in the binary clustering results is greater than or equal to the similarity threshold.

19. The system of claim 12, wherein the execution of the computer program results in determining a text subunit of each log text in a set of log texts under each log text category, comprising: Each log text in the set of log texts under each log text category is segmented to obtain one or more word units; The one or more word units are identified as the text subunits.

20. The system of claim 12, wherein the set of log texts comprises a plurality of log texts, and the execution of the computer program results in the longest common subsequence of the set of log texts being determined in the following manner: By comparing one or more text sub-units of one log text with one or more text sub-units of another log text, one or more common text sub-units of the one log text and the other log text are determined. The common portion of the one or more common text subunits is determined as the longest common subsequence of the set of log texts.

21. The system of claim 12, wherein the execution of the computer program results in determining a log template for the set of log texts based at least on the longest common subsequence of the set of log texts determined by the text subunits, comprising: The longest common subsequence of the set of log texts is determined as the invariant part of the log template; The portion of the log text set excluding the longest common subsequence is identified as the variable portion of the log template; as well as A log template for the set of log texts is determined based on a combination of the invariant and variable parts.

22. The system of claim 12, wherein the execution of the computer program further results in: Based on whether there is an inclusion relationship between the multiple log templates corresponding to the collection of log texts under the multiple log text categories, some or all of the log templates in the multiple log templates are selectively merged.

23. A computer-readable storage medium, characterized in that, The computer storage medium includes instructions that, when executed, perform the method for determining a log template according to any one of claims 1-11.

24. A computer program product, characterized in that, The computer program product includes instructions that, when executed by a processor, implement the method for determining a log template according to any one of claims 1-11.

Images