A log template extraction method, device and electronic equipment

Abstract

The application provides a log template extraction method and device and electronic equipment. The method comprises: obtaining a first log template; determining an inclusive relationship between the first log template and a second log template obtained in advance, the first log template being obtained based on a first log cluster, the second log template being obtained based on a second log cluster, and log time in the first log cluster being earlier than log time in the second log cluster; and extracting a target log template from the first log template and the second log template based on the inclusive relationship. The application can reduce the required computing resources.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method, apparatus and electronic device for extracting log templates. Background Technology

[0002] Network logs store a wealth of system status and operational information. Traditional analysis methods rely on manually writing rules to filter and check for potential anomalies. As system scale increases and distributed systems become more prevalent, a large amount of data is generated daily. Due to the high degree of repetition in daily data, log template analysis is typically used. However, in live network environments, data with the same filename is usually output daily, making log template extraction and analysis computationally expensive. Summary of the Invention

[0003] This application provides a log template extraction method, apparatus, and electronic device to address the problem of requiring significant computing resources.

[0004] In a first aspect, embodiments of this application provide a log template extraction method, including:

[0005] Get the first log template;

[0006] Determine the inclusion relationship between the first log template and the pre-acquired second log template. The first log template is obtained based on the first log cluster, and the second log template is obtained based on the second log cluster. The log times in the first log cluster are earlier than the log times in the second log cluster.

[0007] Based on the inclusion relationship, the target log template is extracted from the first log template and the second log template.

[0008] Secondly, embodiments of this application also provide a log template extraction device, comprising:

[0009] The first acquisition module is used to acquire the first log template;

[0010] The determination module is used to determine the inclusion relationship between the first log template and the pre-acquired second log template. The first log template is obtained based on the first log cluster, and the second log template is obtained based on the second log cluster. The log time in the first log cluster is earlier than the log time in the second log cluster.

[0011] An extraction module is used to extract a target log template from the first log template and the second log template based on the inclusion relationship.

[0012] Thirdly, embodiments of this application also provide an electronic device, including a processor and a memory, wherein the memory stores a program or instructions that can run on the processor, and when the program or instructions are executed by the processor, implement the steps of the log template extraction method as described in the first aspect.

[0013] Fourthly, embodiments of this application also provide a readable storage medium storing a program or instructions that, when executed by a processor, implement the steps of the log template extraction method as described in the first aspect.

[0014] In this embodiment of the application, by determining the inclusion relationship between the first log template and the pre-acquired second log template, and based on the inclusion relationship, the target log template is extracted from the first log template and the second log template. By extracting different templates generated by logs at different times, the generation of a large number of duplicate log templates can be reduced, that is, the computing resources required for log matching log templates can be reduced, and the waste of computing and storage resources can be reduced. Attached Figure Description

[0015] To more clearly illustrate the technical solutions of this application, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a flowchart illustrating a log template extraction method provided in an embodiment of this application;

[0017] Figure 2 This is a schematic diagram of another log template extraction method provided in an embodiment of this application;

[0018] Figure 3 This is a schematic diagram of a log template merging process provided in an embodiment of this application;

[0019] Figure 4 This is a schematic diagram of a log template similarity calculation process provided in an embodiment of this application;

[0020] Figure 5 This is a schematic diagram of the structure of a log template extraction device provided in an embodiment of this application;

[0021] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0022] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0023] The terms "first," "second," etc., used in the embodiments of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or devices. Additionally, the use of "and / or" in this application indicates at least one of the connected objects, such as A and / or B and / or C, representing seven possibilities: including A alone, B alone, C alone, and the presence of both A and B, both B and C, both A and C, and the presence of A, B, and C.

[0024] Please see Figure 1 , Figure 1 This is a flowchart illustrating a log template extraction method provided in an embodiment of this application, as shown below. Figure 1 As shown, it includes the following steps:

[0025] Step 101: Obtain the first log template.

[0026] Step 102: Determine the containment relationship between the first log template and the pre-acquired second log template. The first log template is obtained based on the first log cluster, and the second log template is obtained based on the second log cluster. The log time in the first log cluster is earlier than the log time in the second log cluster.

[0027] When log processing is performed on a daily basis, the logs of each day are highly repetitive compared to historical logs. The logs in the first log cluster mentioned above can be understood as the logs generated on the current day, and the logs in the second log cluster mentioned above can be understood as historical logs, such as log clusters generated on any one day or multiple days.

[0028] When analyzing multiple logs from a single day, log templates can be pre-extracted. These templates retain the constant characters from the logs and use placeholders to represent the variable characters. Placeholders can match any character. This allows multiple logs to be condensed into a single template, simplifying log processing while preserving key information (constant characters). Using log templates to represent multiple logs within a log cluster improves the efficiency of log analysis.

[0029] It should be noted that log clusters can be divided according to log identifiers, and logs of the same category can be grouped into the same log cluster. Logs of the same category share the same log identifier; in this embodiment, punctuation information can be used as the log identifier.

[0030] Step 103: Based on the inclusion relationship, extract the target log template from the first log template and the second log template.

[0031] The aforementioned containment relationships can include the inclusion relationship, intersection relationship, and equivalence relationship between two log templates. An inclusion relationship means that one log template contains another log template. For example, log template A can match any log file, and log template B can also match it. However, some log files that log template B can match cannot be matched by log template A. An intersection relationship means that the two log templates do not contain each other. For example, log template A can match any log file, and log template B cannot match any log file that log template B can match. An equivalence relationship means that the two log templates are completely equivalent. For example, log template A can match any log file, and log template B can also match any log file that log template B can match.

[0032] For two log templates that have an inclusion relationship, the log template with a larger matching range can be used as the target log template to increase the number of matching logs for the target log template and reduce the number of required log templates, thereby reducing the required computing resources.

[0033] For two log templates that have an overlapping relationship, since the logs that the two log templates can match do not overlap, both of the above log templates can be used as the above target log templates. That is, the above target log templates include the above two log templates, so as to ensure that all logs match the log templates.

[0034] For two log templates that are equivalent, either one can be chosen as the target log template to reduce redundancy.

[0035] In this embodiment of the application, by determining the inclusion relationship between the first log template and the pre-acquired second log template, and based on the inclusion relationship, the target log template is extracted from the first log template and the second log template. By extracting different templates generated by logs at different times, the generation of a large number of duplicate log templates can be reduced, that is, the computing resources required for log matching log templates can be reduced, and the waste of computing and storage resources can be reduced.

[0036] Optionally, obtaining the first log template in step 101 may specifically include the following steps:

[0037] Match the logs in the first log cluster with the second log template;

[0038] If the logs in the first log cluster do not completely match the second log template, the first log template of the first log cluster is extracted.

[0039] It should be noted that the situation where the logs in the first log cluster do not completely match the second log template includes at least some logs in the first log cluster not matching the second log template.

[0040] The extraction of the first log template can be implemented based on the frequent pattern tree algorithm. However, if there are logs that do not match the second log template, when extracting the log template corresponding to the log cluster, it is necessary to use all the logs in the log cluster to extract the log template in order to achieve the extraction of the log template.

[0041] Optionally, if all logs in the first log cluster completely match the second log template, the second log template is used to process the logs in the first log cluster.

[0042] In this way, if the existing log template can be used to match the logs in the newly generated first log cluster, there is no need to regenerate the log template, further reducing the waste of computing resources.

[0043] Optionally, extracting the first log template of the first log cluster when the logs in the first log cluster do not completely match the second log template may specifically include:

[0044] If the logs in the first log cluster do not completely match the second log template, a first frequent pattern tree is built using the parameter information of each character in the first log cluster.

[0045] Obtain candidate nodes in the first frequent pattern tree, wherein the candidate nodes include multiple leaf nodes and the positions of the characters corresponding to the multiple leaf nodes are consistent;

[0046] If the position of the candidate node is not a preset position, or the character corresponding to the candidate node is not a preset character, the multiple leaf nodes are pruned to obtain a second frequent pattern tree.

[0047] The first log template is obtained based on the second frequent pattern tree.

[0048] The parameter information for each character can include the position of each character and the frequency corresponding to that position. In the process of establishing the first frequent pattern tree, each log in the first log cluster is inserted one by one, and each character is inserted in descending order of frequency during the insertion of each log. As a result, the same character located at the same position in different logs may correspond to different nodes in the first frequent pattern tree, resulting in more branches in the first frequent pattern tree. That is, in the first frequent pattern tree, for characters with at least two positions, there are at least two nodes, and each node corresponds to one position.

[0049] It is understood that during the pruning process of the first frequent pattern tree, the characters corresponding to the leaf nodes in the first frequent pattern tree are the characters that appear least frequently in the log. By judging whether the position information of the characters corresponding to the multiple leaf nodes of the candidate nodes are consistent, if they are consistent, it can be determined that the character at that position is always changing and can be determined as a log parameter. Thus, the corresponding multiple leaf nodes are pruned from the first frequent pattern tree to improve the extraction efficiency of the log template.

[0050] When there are significant differences among multiple logs in the first log cluster, the branches of the generated first frequent pattern tree may not intersect at all. If pruning is performed, an empty tree will appear, meaning that the generated log templates are all composed of placeholders. Such log templates are meaningless for log processing.

[0051] In this embodiment, the aforementioned preset position and preset character are pre-set node information that cannot be pruned. Specifically, if the position of a node is the aforementioned preset position and the character corresponding to the node is the aforementioned preset character, the node cannot be pruned, thereby preventing the frequent pattern tree from becoming an empty tree.

[0052] It should be noted that the aforementioned preset positions and preset characters can be in one-to-one correspondence. When a node's position is the aforementioned preset position and the character corresponding to that node is the aforementioned preset character, that node is determined to be a node that cannot be pruned. When at least one of the aforementioned preset positions and preset characters includes multiple elements, for example, when the preset positions include position A and position B, and the preset characters include character 1, character 2, and character 3, the nodes corresponding to character 1 at position A, character 2 at position A, and character 3 at position B can be set as nodes that cannot be pruned. Alternatively, the nodes corresponding to character 1, character 2, and character 3 at position A, and the nodes corresponding to character 1, character 2, and character 3 at position B can also be set as nodes that cannot be pruned.

[0053] Optionally, before determining the containment relationship between the first log template and the pre-acquired second log template in step 102, the method may further include the following steps:

[0054] Obtain the log identifier of the first log cluster and the log identifier of the second log cluster;

[0055] If the log identifier of the first log cluster and the log identifier of the second log cluster are the same, obtain the similarity between the first log template and the second log template;

[0056] Determining the inclusion relationship between the first log template and the pre-acquired second log template includes:

[0057] Based on the similarity, the inclusion relationship between the first log template and the pre-acquired second log template is obtained.

[0058] It is understood that logs consist of a fixed template part and a variable parameter part. No matter how the log parameters change, the relative position of the punctuation marks remains unchanged. Each type of log template has the same punctuation mark pattern. This application uses the punctuation marks in the log as the log identifier to distinguish different types of logs.

[0059] The aforementioned log identifier can distinguish different categories of logs. When dividing log clusters, logs of the same category can be grouped into the same log cluster. Therefore, the log identifier of a log cluster can also be used to distinguish different log clusters.

[0060] In this embodiment, for the first log cluster and the second log cluster, since the log times in the first log cluster are earlier than those in the second log cluster, the logs in the first and second log clusters are isolated from each other during clustering. This means that log clusters generated at different times can correspond to the same log identifier. If the log identifiers of the first and second log clusters are different, it can be determined that the first and second log templates cannot be merged. Therefore, no processing is performed on the first and second log templates; instead, both templates can be used simultaneously to attempt matching between different logs.

[0061] The similarity between the first log template and the second log template can be calculated based on the premise that the log identifiers of the first log cluster and the second log cluster are identical. This ensures that the first and second log templates contain the same number of characters. Specifically, the similarity can be calculated based on the total number of characters in the log templates. For example, the similarity can be understood as the proportion of identical characters in the same position in both log templates to the total number of characters, or simply the number of identical characters in the same position. The more identical characters in the same position, the greater the similarity. Correspondingly, when obtaining the inclusion relationship between two log templates based on similarity, the calculation can be performed separately for different numbers of characters in the log templates. For example, for log templates with a larger number of characters, if the similarity is greater than a preset percentage, the two log templates can be considered similar; for log templates with a smaller number of characters, if the similarity is greater than a preset number of characters, the two log templates can be considered similar.

[0062] Optionally, step 103, which involves extracting the target log template from the first log template and the second log template based on the inclusion relationship, may specifically include the following steps:

[0063] If the first log template contains the second log template, the first log template is determined as the target log template;

[0064] If the first log template and the second log template are mutually exclusive, the first log template and the second log template are determined as the target log template.

[0065] In the case where the first log template includes the second log template, that is, the constant characters of the first log template can be matched with the constant characters at the corresponding positions in the second log template, and the first log template including the second log template means that the first log template and the second log template are the same.

[0066] In this way, if the first log template includes the second log template, all logs that can be matched by the second log template can also be matched by the first log template. Therefore, the first log template can be used to replace the second log template. That is, the first log template is determined as the target log template, the second log template is eliminated, the number of log templates is reduced, and the processing effect of the log templates on the logs is not affected.

[0067] In the case where the first log template and the second log template are mutually exclusive, the logs that can be matched by the first log template and the logs that can be matched by the second log template do not overlap. Therefore, both the first log template and the second log template can be added to the target log template.

[0068] Optionally, determining the containment relationship between the first log template and the second log template in step 102 may specifically include the following steps:

[0069] If the character corresponding to the preset position of at least one of the first log template and the second log template is a preset character, it is determined that the first log template and the second log template have a mutually exclusive containment relationship.

[0070] The aforementioned preset positions and preset characters can be obtained from a preset expert experience database. For example, in the expert experience database, characters that cannot be merged can be preset according to experience values. The characters that cannot be merged can be characters containing key information, or characters that, after merging, would cause the log template to be too high-level (i.e., it can match most logs, but the matching is meaningless for log analysis and processing). Furthermore, the aforementioned preset positions and preset characters can correspond to different log clusters to improve the extraction effect of log templates.

[0071] In this embodiment, when the character corresponding to the preset position of at least one of the first log template and the second log template is a preset character, it is determined that the first log template and the second log template have a mutually exclusive inclusion relationship. That is, the preset character at the preset position is a character that cannot be merged in the log template. This avoids the situation where some key characters are replaced by placeholders in the log template due to their low frequency of occurrence after merging, which would result in the target log template being too high-level, making the extraction of the log template meaningless. It also ensures that the target log template can more accurately extract logs containing key characters.

[0072] For ease of understanding, a specific example is as follows:

[0073] like Figure 2 As shown, this application embodiment provides a method for extracting log templates in an online environment, taking the analysis of logs on a daily basis as an example, including the following steps:

[0074] Step 1: Divide the day's logs into log clusters based on log signatures. The specific steps are as follows:

[0075] Iterate through all logs in the original log set for the day, extract the "log signature" for each log, and classify each log into the corresponding log cluster based on the "log signature".

[0076] The massive network device logs consist of a fixed template section and a variable parameter section. Regardless of how the log parameters change, the relative positions of punctuation marks remain constant. Each type of log template has the same punctuation mark pattern. This application uses the punctuation marks in the logs as log signatures to identify logs with different punctuation mark patterns. Specifically, the punctuation marks in each log entry are first extracted and combined into a log signature using a certain method. The logs are then classified into different log clusters based on the log signatures. For example, the log signature of the log "GET / v2 / 54fadb412c / servers / detail HTTP / 1.1"status:200len:1893time:0.2477829" is "- / - / - / - / -- / -"-:--:--:- (where "-" is used to replace the character (token) in the log), thus classifying logs with the same log signature into the same log cluster.

[0077] Step 2: Obtain the log template set from the previous day, and perform data annotation and deduplication on the logs of different log clusters.

[0078] The previous day's log template set includes one or more log templates extracted from the previous day's logs. Log template extraction is based on a frequent pattern tree. For example, if the previous day's log set includes three log entries: "GET / v2.1 / servers / detail HTTP / 1.1"status:200len:1893time:0.2477", "GET / v2.1 / servers / 54fadb412c4e40 HTTP / 1.1"status:404len:241time:0.2389", and "GET / v2.1 / servers / detail HTTP / 1.2"status:200len:242time:0.2429", the extracted log template can be "GET / v2.1 / servers / *HTTP / 1.1"status:*len:*time:*" (where * represents a placeholder and can match any character; GET, v2.1, etc., are constant characters in the log template).

[0079] The data annotation and deduplication algorithm is implemented as follows:

[0080] Note: The log and log template are split into tokens and compared one by one according to their positions (the * in the template is matched with the parameters in the log). If any constant in the log does not match any non-* token in the log template, the log is considered not to match the log template.

[0081] Deduplication: If a log cluster completely matches the template corresponding to that log cluster, then all duplicates are removed without template extraction; if no template is found for that log cluster, or if any log in that log cluster does not match the template, then all logs are treated as a new dataset and template extraction is performed.

[0082] It should be noted that if a template for a certain log cluster only matches a portion of the logs, the frequent itemset tree of that log cluster needs to be regenerated. Therefore, the training result will affect the current template. Thus, all data of that log cluster (not just data that does not match the current template) must participate in the template extraction process.

[0083] For better understanding, see the following example:

[0084] The log set from the previous day can be divided into log cluster A and log cluster B. Log cluster A includes the following logs:

[0085] GET / v2.1 / servers / 184ddf HTTP / 1.1"status:200len:2364time:0.15;

[0086] GET / v2.1 / servers / 0eak5 HTTP / 1.1"status:201len:2454time:0.16;

[0087] GET / v2.1 / servers / 0156e5 HTTP / 1.1"status:302len:2865time:0.14;

[0088] Log template E1 is extracted from log cluster A: GET / v2.1 / servers / *HTTP / 1.1"status:*len:*time:*;

[0089] Log cluster B includes the following logs:

[0090] Accepted password for hkfsk from 10port 4568ssh2;

[0091] Accepted password for sdfxx from 11port 4548ssh2;

[0092] Log template E2 is extracted from log cluster B: Accepted password for *from*port*ssh2;

[0093] The log set for the day can be divided into log cluster A, log cluster B, and log cluster C;

[0094] Log cluster A includes the following logs:

[0095] Log 1: GET / v2.2 / servers / 0e HTTP / 1.1"status:200 len:4562 time:0.2 (Does not match log template E1, see Table 1 for details);

[0096] Log 2: GET / v2.1 / servers / 0ebaef HTTP / 1.1"status:200 len:4862 time:0.2 (Matches log template E1. See the matching process in Log 1 for details, which will not be repeated here).

[0097] Table 1

[0098]

[0099] Log cluster B includes the following logs:

[0100] Log 3: Accepted password for ceed from 10port 4276ssh2 (matches log template E2, see Table 2 for details);

[0101] Table 2

[0102]

[0103]

[0104] Log cluster C includes the following logs:

[0105] Log 4: Removed session 206578;

[0106] Log 5: Removed session 286578;

[0107] It is understandable that logs 4 and 5 do not match the above log template.

[0108] By annotating the logs for that day, the following conclusions can be drawn:

[0109] (1) For log cluster A, since some data on the current day could not match the template of the previous day, it is necessary to perform template extraction operation on the entire log cluster.

[0110] (2) For log cluster B, since all data on the current day matches the template of the previous day, there is no need to perform template extraction, and the template remains unchanged;

[0111] (3) For log cluster C, since there is no template matching, template extraction is performed as new data.

[0112] Step 3: Perform template extraction on the deduplicated data. Specifically, first, construct a frequent itemset tree based on token frequency statistics and generate a log template. During the construction of the frequent itemset tree, to avoid pruning into an empty tree, an expert experience base can be introduced to prevent some nodes from being pruned.

[0113] The expert experience base can be manually configured or automatically generated by a program. The expert experience base includes log signatures (corresponding to log clusters), token positions, and characters that cannot be truncated, as shown in Table 3:

[0114] Table 3

[0115] - / - / - / -- / -"-:--:--: Position 0 (pos_0) GET; DELETE; POST - / - / - / -- / -"-:--:--: Position 7 (pos_7) 404；503

[0116] It is understandable that for a log template with the signature "- / - / - / -- / -"-:--:--:", the characters appearing at position 0: GET, DELETE, and POST must not be truncated and must be retained in the log template; the characters appearing at position 7: 404 and 503 must not be truncated and must be retained in the log template. The position of a character can be represented by pos_n, where n is an integer greater than or equal to 0.

[0117] During the branch pruning process, if the words in Table 3 above appear in the corresponding position, the node cannot be pruned.

[0118] To better understand, consider the following log entry as an example:

[0119] "GET / v2.1 / servers / auth HTTP / 1.1" status:200len:237time:0.2378;

[0120] "GET / v2.1 / servers / auth HTTP / 1.1" status:202len:239time:0.2380;

[0121] "GET / v2.1 / servers / auth HTTP / 1.1" status:404len:241time:0.2389;

[0122] "GET / v2.1 / servers / auth HTTP / 1.1" status:404len:242time:0.2390;

[0123] "GET / v2.1 / servers / auth HTTP / 1.1" status:503len:241time:0.2389;

[0124] "GET / v2.1 / servers / auth HTTP / 1.1" status:503len:242time:0.2390;

[0125] If pos_7 contains four branches: 404, 503, 202, and 200, then branches 404 and 503 are retained, while branches 200 and 202 are pruned, resulting in the following three log templates:

[0126] "GET / v2.1 / servers / auth HTTP / 1.1" status:*len:*time:*;

[0127] "GET / v2.1 / servers / auth HTTP / 1.1" status:404len:*time:*;

[0128] "GET / v2.1 / servers / auth HTTP / 1.1" status:503len:*time:*.

[0129] Step 4: Merge the log template generated on the current day with the log template generated on the previous day, such as... Figure 3 As shown, the merging process is as follows:

[0130] Find the log template signature. If no template with the same signature exists, then the two do not belong to the same log cluster (not in the same frequent itemset tree) and should be classified as different templates. Therefore, they can be added directly.

[0131] If log signatures are identical, template similarity needs to be calculated. If similar templates are found (template similarity reaches a threshold), they are added directly. The method for calculating template similarity is as follows: Figure 4As shown, firstly, the newly generated log template and the existing log template are compared after removing the log signature, resulting in their respective character lists. Then, the characters at each position are compared (if one of the different characters is *, it is counted as the same). The total number of characters is then checked: if the total number of characters is less than 10, and the number of differing characters is less than 3, the two templates are considered similar; if the number of differing characters is greater than or equal to 3, they are considered dissimilar. If the total number of characters is greater than or equal to 10, and the percentage difference is less than 10%, they are considered similar; if the percentage difference is greater than or equal to 10%, they are considered dissimilar. If they are not similar templates, both are added to the log template set.

[0132] Examples of similarity calculations are shown in Table 4:

[0133] Table 4

[0134]

[0135] If a similar template is found, the hierarchical relationship needs to be determined first. The hierarchical relationship refers to the containment relationship between templates. The containment relationship between two templates A and B is divided into four types, as shown in Table 5:

[0136] Table 5

[0137] A is located at B A>B A completely encompasses B A is located below B A<B A is completely contained within B. A intersects with B A! = B A and B are mutually intolerant A is equivalent to B A = B A and B are completely equivalent.

[0138] The containment relationship between templates refers to the inclusion relationship between the data ranges that two templates can match. A specific example is as follows:

[0139] Example 1:

[0140] Template A: "GET / v2.1 / servers / *HTTP / 1.1" status:*len:*time:*;

[0141] Template B "GET / v2.1 / * / *HTTP / 1.1" status:* len:* time:*;

[0142] Template A can match the following logs:

[0143] "GET / v2.1 / servers / abc HTTP / 1.1" status:202len:2378time:0.3719;

[0144] "GET / v2.1 / servers / efg HTTP / 1.1" status:302len:2379time:0.3780;

[0145] However, template A cannot match the log: "GET / v2.1 / flavors / efg HTTP / 1.1" status:302 len:2379 time:0.3780;

[0146] Template B can match the three logs mentioned above.

[0147] Therefore, template B can match a larger range of logs than template A, which can be denoted as A being subordinate to B (A <B)。

[0148] Example 2:

[0149] Template A: "GET / v2.1 / servers / *HTTP / 1.1" status:*len:*time:*;

[0150] Template B "GET / v2.1 / servers / *HTTP / 1.2" status:*len:*time:*;

[0151] Template A can match the following logs:

[0152] "GET / v2.1 / servers / abc HTTP / 1.1" status:202len:2378time:0.3719;

[0153] "GET / v2.1 / servers / efg HTTP / 1.1" status:302len:2379time:0.3780;

[0154] Template A cannot match the log: "GET / v2.1 / servers / efg HTTP / 1.2" status:302 len:2379 time:0.3780;

[0155] Template B is the opposite; it can match logs that template A cannot match, but it cannot match logs that template A can match.

[0156] Therefore, neither template B nor template A can match some of the data that the other can match, which can be written as A intersecting B (A != B).

[0157] The process for determining the containment relationship between templates is as follows:

[0158] The template is split into a set of tokens, and two tokens at the same position are compared with each other;

[0159] Record the positions of the asterisks in the two templates respectively, forming a set;

[0160] Compare the sizes of the sets containing the asterisks in the two templates to determine the containment relationship;

[0161] The inclusion relationship is determined as follows:

[0162] If the constant part (not *) differs, it is directly determined to be an intersection;

[0163] If the constant parts are the same, the determination is based on the containment relationship between the sets;

[0164] For example:

[0165] Template A: "GET / v2.1 / servers / *HTTP / 1.1" status:*len:*time:*;

[0166] Template B: "GET / v2.1 / * / *HTTP / 1.1" status:* len:* time:*;

[0167] Template C: "GET / v2.1 / * / abc HTTP / 1.2" status:* len:* time:*;

[0168] Template D: "GET / v2.1 / servers / *HTTP / 1.2" status:*len:*time:*;

[0169] The corresponding *position sets are as follows:

[0170] setA(pos_3, pos_7, pos_9, pos_11);

[0171] setB(pos_2, pos_3, pos_7, pos_9, pos_11);

[0172] setC(pos_2, pos_7, pos_9, pos_11);

[0173] setD(pos_3, pos_7, pos_9, pos_11);

[0174] Template B and template A have the same constant parts, and setB contains setA. The containment relationship is determined as follows: A <B;

[0175] Since the constant parts of template B and template A are different, the inclusion relationship is determined as: A! = C;

[0176] The HTTP version number (1.2) of template D is different from the HTTP version number (1.1) of template A. Therefore, the inclusion relationship is determined as: A! = D;

[0177] After determining the containment relationship, the template merging steps are as described above. Figure 3 Corresponding implementation examples.

[0178] For new templates and existing similar templates with the same module and priority, the following are examples of adding methods based on their hierarchical relationship:

[0179] a) The new template is located above a similar template (new template > similar template):

[0180] Because the new template is more inclusive, it can completely replace existing similar templates without the need for repeated additions.

[0181] b) The new template is located below similar templates (new template < similar templates):

[0182] It should be noted that since the new logs were pre-marked and deduplicated according to the existing template in step 1 above, scenario b will not occur in the actual scenario.

[0183] Furthermore, since existing similar templates are more inclusive, the new template is just a special case and there is no need to add it again.

[0184] c) The new template intersects with a similar template (new template != similar template):

[0185] The steps for merging overlapping cases are as follows:

[0186] The template is split into a set of tokens, and two tokens at the same position are compared with each other;

[0187] If the tokens for two positions are different (whether both are constants or one is *), if the corresponding words and positions are not in the expert experience base, they are abstracted as *; if they are in the expert experience base, they are retained.

[0188] The final template retained is the template after the difference token is abstracted as *;

[0189] When a new template intersects with a similar template, there are two types: the first is that the new template's * and the similar template's constant token conflict, leading to the intersection; the second is that the new template and the similar template's two constant tokens conflict, leading to the intersection. Both types need to be replaced with * to form a new, higher-level template to replace the similar template.

[0190] d) The new template is equivalent to a similar template (new template = similar template):

[0191] It should be noted that since the new logs were pre-marked and deduplicated using the existing template in step 1 above, scenario d will not occur in real-world scenarios. Furthermore, in this case, since the new template is essentially the same as similar templates, there is no need to add a new template; the original log template can be maintained.

[0192] In this embodiment, in response to the characteristics of data in online environments and the problems arising from the isolated template extraction of daily log data using existing technologies, a log template extraction process for online environments is proposed. This process can continuously and iteratively process the data, while introducing an expert experience base to optimize branch pruning effects and refining and deduplicating the daily generated templates to achieve unified management and rapid convergence.

[0193] Please refer to Figure 5 , Figure 5 This is a structural diagram of a log template extraction device provided in an embodiment of this application, as shown below. Figure 5 As shown, the log template extraction of 500 includes:

[0194] The first acquisition module 501 is used to acquire the first log template;

[0195] The determining module 502 is used to determine the inclusion relationship between the first log template and the pre-acquired second log template, wherein the first log template is obtained based on the first log cluster, the second log template is obtained based on the second log cluster, and the log time in the first log cluster is earlier than the log time in the second log cluster;

[0196] Extraction module 503 is used to extract the target log template from the first log template and the second log template based on the inclusion relationship.

[0197] Optionally, the first acquisition module 501 may specifically include:

[0198] The matching unit is used to match the logs in the first log cluster with the second log template;

[0199] The extraction unit is used to extract the first log template of the first log cluster when the logs in the first log cluster do not completely match the second log template.

[0200] Optionally, the extraction unit may specifically include:

[0201] A subunit is established to build a first frequent pattern tree using the parameter information of each character in the first log cluster when the logs in the first log cluster do not completely match the second log template.

[0202] The first acquisition subunit is used to acquire candidate nodes in the first frequent pattern tree, wherein the candidate nodes include multiple leaf nodes and the positions of the characters corresponding to the multiple leaf nodes are consistent.

[0203] The pruning subunit is used to prune the multiple leaf nodes when the position of the candidate node is not a preset position or the character corresponding to the candidate node is not a preset character, so as to obtain a second frequent pattern tree.

[0204] The second acquisition subunit is used to acquire the first log template based on the second frequent pattern tree.

[0205] Optionally, the log template extraction 500 may further include:

[0206] The second acquisition module is used to acquire the log identifier of the first log cluster and the log identifier of the second log cluster;

[0207] The third acquisition module is used to acquire the similarity between the first log template and the second log template when the log identifier of the first log cluster and the log identifier of the second log cluster are the same.

[0208] The determining module 502 may specifically include:

[0209] The acquisition unit is used to acquire the inclusion relationship between the first log template and the pre-acquired second log template based on the similarity.

[0210] Optionally, the extraction module 503 may specifically include:

[0211] The first determining unit is configured to determine the first log template as the target log template when the first log template contains the second log template.

[0212] The second determining unit is used to determine the first log template and the second log template as the target log template when the first log template and the second log template are mutually exclusive.

[0213] Optionally, the determining module 502 may specifically include:

[0214] The third determining unit is used to determine that the first log template and the second log template are mutually exclusive and inclusive when the character corresponding to the preset position of at least one of the first log template and the second log template is a preset character.

[0215] It should be noted that the log template extraction device 500 provided in this application embodiment can achieve Figure 1 The various processes in the method embodiments achieve the same technical effect, and will not be described again here to avoid repetition.

[0216] This application also provides an electronic device. Because the principle by which the electronic device solves the problem is similar to that in the embodiments of this application... Figure 1 The log template extraction method shown is similar; therefore, the implementation of this electronic device can be found in the method implementation, and repeated details will not be elaborated upon. For example... Figure 6As shown, the electronic device in this application embodiment includes a memory 620, a transceiver 610, and a processor 600;

[0217] The memory 620 is used to store computer programs; the transceiver 610 is used to send and receive data under the control of the processor 600; the processor 600 is used to read the computer program in the memory 620 and perform the following operations:

[0218] Get the first log template;

[0219] Determine the inclusion relationship between the first log template and the pre-acquired second log template. The first log template is obtained based on the first log cluster, and the second log template is obtained based on the second log cluster. The log times in the first log cluster are earlier than the log times in the second log cluster.

[0220] Based on the inclusion relationship, the target log template is extracted from the first log template and the second log template.

[0221] Among them, Figure 6 In this context, the bus architecture may include any number of interconnected buses and bridges, specifically linking various circuits together, represented by one or more processors (processor 600) and memory (memory 620). The bus architecture may also link together various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. A bus interface provides an interface. Transceiver 610 may be multiple elements, including transmitters and transceivers, providing a unit for communicating with various other devices over a transmission medium. Processor 600 is responsible for managing the bus architecture and general processing, and memory 620 may store data used by processor 600 during operation.

[0222] The processor 600 can be a central processing unit (CPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a complex programmable logic device (CPLD). The processor can also adopt a multi-core architecture.

[0223] Optionally, obtaining the first log template includes:

[0224] Match the logs in the first log cluster with the second log template;

[0225] If the logs in the first log cluster do not completely match the second log template, the first log template of the first log cluster is extracted.

[0226] Optionally, extracting the first log template of the first log cluster when the logs in the first log cluster do not completely match the second log template includes:

[0227] If the logs in the first log cluster do not completely match the second log template, a first frequent pattern tree is built using the parameter information of each character in the first log cluster.

[0228] Obtain candidate nodes from the first frequent pattern tree, wherein the candidate nodes include multiple leaf nodes and the positions of the characters corresponding to the multiple leaf nodes are consistent;

[0229] If the position of the candidate node is not a preset position, or the character corresponding to the candidate node is not a preset character, the multiple leaf nodes are pruned to obtain a second frequent pattern tree.

[0230] The first log template is obtained based on the second frequent pattern tree.

[0231] Optionally, the processor 600 is also configured to read the computer program in the memory 620 and perform the following operations:

[0232] Obtain the log identifier of the first log cluster and the log identifier of the second log cluster;

[0233] If the log identifier of the first log cluster and the log identifier of the second log cluster are the same, obtain the similarity between the first log template and the second log template;

[0234] Determining the inclusion relationship between the first log template and the pre-acquired second log template includes:

[0235] Based on the similarity, the inclusion relationship between the first log template and the pre-acquired second log template is obtained.

[0236] Optionally, extracting the target log template from the first log template and the second log template based on the inclusion relationship includes:

[0237] If the first log template contains the second log template, the first log template is determined as the target log template;

[0238] If the first log template and the second log template are mutually exclusive, the first log template and the second log template are determined as the target log template.

[0239] Optionally, determining the containment relationship between the first log template and the second log template includes:

[0240] If the character corresponding to the preset position of at least one of the first log template and the second log template is a preset character, it is determined that the first log template and the second log template have a mutually exclusive containment relationship.

[0241] The electronic device provided in this application embodiment can perform the above-described functions. Figure 1 The method embodiments shown are similar in principle and technical effect, and will not be described again here.

[0242] This application embodiment also provides a readable storage medium storing a program or instructions that, when executed by a processor, implement the above-described functionality. Figure 1 The various processes in the method embodiments described herein can achieve the same technical effect, and will not be repeated here to avoid repetition.

[0243] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0244] This application embodiment also provides a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the above. Figure 1 The various processes in the method embodiments described herein can achieve the same technical effect, and will not be repeated here to avoid repetition.

[0245] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0246] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0247] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0248] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A method for extracting log templates, characterized in that, include: Get the first log template; Determine the inclusion relationship between the first log template and the pre-acquired second log template. The first log template is obtained based on the first log cluster, and the second log template is obtained based on the second log cluster. The log times in the first log cluster are earlier than the log times in the second log cluster. Based on the inclusion relationship, the target log template is extracted from the first log template and the second log template; wherein, the inclusion relationship includes the inclusion relationship, intersection relationship and equivalence relationship between the first log template and the second log template; The process of obtaining the first log template includes: Match the logs in the first log cluster with the second log template; If the logs in the first log cluster do not completely match the second log template, extract the first log template from the first log cluster; When the logs in the first log cluster do not completely match the second log template, extracting the first log template of the first log cluster includes: If the logs in the first log cluster do not completely match the second log template, a first frequent pattern tree is established using the parameter information of each character in the first log cluster. The parameter information of each character includes the position of each character and the frequency corresponding to the position. The process of establishing the first frequent pattern tree is to insert each log in the first log cluster one by one, and to insert each character in descending order of frequency during the insertion of each log. Obtain candidate nodes in the first frequent pattern tree, wherein the candidate nodes include multiple leaf nodes and the positions of the characters corresponding to the multiple leaf nodes are consistent; If the position of the candidate node is not a preset position, or the character corresponding to the candidate node is not a preset character, the multiple leaf nodes are pruned to obtain a second frequent pattern tree. The first log template is obtained based on the second frequent pattern tree using the frequent pattern tree algorithm.

2. The method as described in claim 1, characterized in that, Before determining the containment relationship between the first log template and the pre-acquired second log template, the method further includes: Obtain the log identifier of the first log cluster and the log identifier of the second log cluster; If the log identifier of the first log cluster and the log identifier of the second log cluster are the same, obtain the similarity between the first log template and the second log template; Determining the inclusion relationship between the first log template and the pre-acquired second log template includes: Based on the similarity, the inclusion relationship between the first log template and the pre-acquired second log template is obtained.

3. The method as described in claim 1, characterized in that, The step of extracting the target log template from the first log template and the second log template based on the inclusion relationship includes: If the first log template contains the second log template, the first log template is determined as the target log template; If the first log template and the second log template are mutually exclusive, the first log template and the second log template are determined as the target log template.

4. The method as described in claim 3, characterized in that, Determining the containment relationship between the first log template and the second log template includes: If the character corresponding to the preset position of at least one of the first log template and the second log template is a preset character, it is determined that the first log template and the second log template have a mutually exclusive containment relationship.

5. A log template extraction device, characterized in that, include: The first acquisition module is used to acquire the first log template; The determination module is used to determine the inclusion relationship between the first log template and the pre-acquired second log template. The first log template is obtained based on the first log cluster, and the second log template is obtained based on the second log cluster. The log time in the first log cluster is earlier than the log time in the second log cluster. An extraction module is used to extract a target log template from the first log template and the second log template based on the inclusion relationship; wherein, the inclusion relationship includes the inclusion relationship, intersection relationship and equivalence relationship between the first log template and the second log template; The first acquisition module includes: The matching unit is used to match the logs in the first log cluster with the second log template; The extraction unit is used to extract the first log template of the first log cluster when the logs in the first log cluster do not completely match the second log template. The extraction unit includes: A subunit is established to build a first frequent pattern tree using the parameter information of each character in the first log cluster when the logs in the first log cluster do not completely match the second log template. The parameter information of each character includes the position of each character and the frequency corresponding to the position. The process of building the first frequent pattern tree is to insert each log in the first log cluster one by one, and to insert each character in descending order of frequency during the insertion of each log. The first acquisition subunit is used to acquire candidate nodes in the first frequent pattern tree, wherein the candidate nodes include multiple leaf nodes and the positions of the characters corresponding to the multiple leaf nodes are consistent. The pruning subunit is used to prune the multiple leaf nodes when the position of the candidate node is not a preset position or the character corresponding to the candidate node is not a preset character, so as to obtain a second frequent pattern tree. The second acquisition subunit is used to acquire the first log template based on the second frequent pattern tree using the frequent pattern tree algorithm.

6. An electronic device, characterized in that, It includes a processor and a memory, the memory storing a program or instructions that can run on the processor, the program or instructions being executed by the processor to implement the steps of the log template extraction method as described in any one of claims 1 to 4.

7. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the log template extraction method as described in any one of claims 1 to 4.

Images