A method, device, electronic device and storage medium for determining a log template

Abstract

The embodiment of the present invention discloses a method, device, electronic device and storage medium for determining a log template. The method for determining a log template includes: when receiving a log to be processed, obtaining a target matching rule from a matching rule library in sequence according to the priority order of each matching rule in the matching rule library; using the target matching rule to perform an operation of matching the log to be processed with each log template in a log template library; if a log template matching the log to be processed cannot be obtained, returning to perform an operation of obtaining a target matching rule from a matching rule library in sequence according to the priority order of each matching rule in the matching rule library; when it is determined that a log template matching the log to be processed cannot be obtained using all matching rules, adding the log to be processed as a new log template to the log template library. The technical solution of the embodiment of the present invention can reduce the computational complexity of log template extraction and improve the versatility of log template extraction.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of computer technology, and in particular, to a method, device, electronic device, and storage medium for determining a log template. Background Art

[0002] Logs record a large amount of valuable information during system operation. The post-maintenance of the system is also mainly based on viewing logs. Logs play an important role in the system development and operation and maintenance process.

[0003] Among them, the extraction of log templates is a very important part of log analysis. By extracting the log template, logs can be clustered so that logs of the same type belong to the same template. Generally speaking, normal logs occupy the vast majority, and there are few abnormal logs. After clustering through templates, operation and maintenance personnel can focus on templates with fewer logs and quickly find the information they want. In addition to being able to cluster logs, log template extraction is also a pre-step based on data-driven log analysis, which structures semi-structured logs to facilitate subsequent log analysis. For example, after extracting the log template, variables can be further extracted. Since variables may indicate certain operating indicators of the system, system anomaly detection can be performed by analyzing the variables.

[0004] However, traditional log template extraction methods often require an offline training process. Usually, the operation logs of the system are collected within a period of time during the initial operation, and then a model is trained using these logs. Finally, the trained model is used for template extraction of online logs. However, this method has several problems: (1) Models need to be trained for different systems, which makes model training more cumbersome and computationally complex. (2) The logs collected in the initial stage may not cover the logs corresponding to all events in the system. For example, the initial operation stage of the system may not be able to run to all possible logical flows. (3) The system usually needs to be iteratively updated, and the update is very likely to cause changes in the logs. The logs collected in the initial stage do not contain these new logs, and the model cannot analyze and predict these new logs, which makes the model less versatile and less usable. Summary of the invention

[0005] The embodiments of the present invention provide a method, device, electronic device and storage medium for determining a log template, which reduce the computational complexity of log template extraction and improve the versatility of log template extraction without performing model training.

[0006] In a first aspect, an embodiment of the present invention provides a method for determining a log template, including:

[0007] When receiving the log to be processed, the target matching rules are obtained from the matching rule library in sequence according to the priority order of each matching rule in the matching rule library;

[0008] Using the target matching rule, the operation of matching the log to be processed with each log template in the log template library is performed;

[0009] If the log template matching the log to be processed cannot be obtained, the operation of obtaining the target matching rules from the matching rule base in sequence according to the priority order of each matching rule in the matching rule base is returned;

[0010] When it is determined that a log template matching the log to be processed cannot be obtained by using all matching rules, the log to be processed is added to the log template library as a new log template.

[0011] In a second aspect, an embodiment of the present invention further provides a device for determining a log template, including:

[0012] The target matching rule acquisition module is used to obtain the target matching rules from the matching rule library in sequence according to the priority order of each matching rule in the matching rule library when receiving the log to be processed;

[0013] A matching processing module, used to use a target matching rule to perform an operation of matching the log to be processed with each log template in the log template library;

[0014] A loop execution module, for returning to execute the operation of obtaining target matching rules from the matching rule base in sequence according to the priority order of each matching rule in the matching rule base if a log template matching the log to be processed is not obtained;

[0015] The log template library updating module is used to add the log to be processed as a new log template to the log template library when it is determined that no log template matching the log to be processed can be obtained by using all matching rules.

[0016] In a third aspect, an embodiment of the present invention further provides an electronic device, the electronic device comprising:

[0017] one or more processors;

[0018] A storage device for storing one or more programs;

[0019] When one or more programs are executed by one or more processors, the one or more processors implement the log template determination method provided by any embodiment of the present invention.

[0020] In a fourth aspect, an embodiment of the present invention further provides a computer storage medium on which a computer program is stored, and when the program is executed by a processor, the method for determining a log template provided by any embodiment of the present invention is implemented.

[0021] The technical solution of this embodiment is to obtain the target matching rules from the matching rule library in sequence according to the priority order of each matching rule in the matching rule library when receiving the log to be processed, and then use the target matching rules to perform the operation of matching the log to be processed with each log template in the log template library. If the log template matching the log to be processed cannot be obtained, then return to the operation of obtaining the target matching rules from the matching rule library in sequence according to the priority order of each matching rule in the matching rule library. When it is determined that the log template matching the log to be processed cannot be obtained using all matching rules, the log to be processed is added to the log template library as a new log template. This solution does not need to train a model for identifying log templates. Different matching rules can be used to directly determine the log template of the log to be processed. The log template that matches the new log generated after the system is updated can also be determined. In addition, the log template that matches the log to be processed is determined by different target matching rules, which can avoid the high computational complexity caused by a single matching rule or the poor matching effect of a single matching rule. It solves the problems of cumbersome training, high computational complexity and poor versatility of the log template extraction model in the prior art. It can reduce the computational complexity of log template extraction and improve the versatility of log template extraction without model training. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 is a flow chart of a method for determining a log template provided in Embodiment 1 of the present invention;

[0023] Figure 2 is a schematic diagram of a template prefix tree provided in Embodiment 2 of the present invention;

[0024] Figure 3 is a schematic diagram of a method for determining a log template provided in Embodiment 2 of the present invention;

[0025] Figure 4 is a schematic diagram of a device for determining a log template provided by Embodiment 3 of the present invention;

[0026] Figure 5 A schematic diagram of the structure of an electronic device provided in Embodiment 4 of the present invention. DETAILED DESCRIPTION

[0027] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention.

[0028] It should also be noted that, for ease of description, only the part relevant to the present invention but not all content is shown in the accompanying drawings. It should be mentioned before discussing exemplary embodiments in more detail that some exemplary embodiments are described as processing or methods depicted as flow charts. Although the flow chart describes each operation (or step) as sequential processing, many operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of each operation can be rearranged. When its operation is completed, the processing can be terminated, but it can also have additional steps not included in the accompanying drawings. The processing can correspond to methods, functions, procedures, subroutines, subprograms, etc.

[0029] Embodiment 1

[0030] Figure 1 is a flow chart of a method for determining a log template provided by the first embodiment of the present invention. This embodiment is applicable to the case of efficient extraction of log templates. The method can be executed by a log template determination device, which can be implemented by software and / or hardware and can generally be integrated in an electronic device. Figure 1 As shown, the method includes the following operations:

[0031] S110 . When a log to be processed is received, target matching rules are sequentially obtained from the matching rule library according to the priority order of each matching rule in the matching rule library.

[0032] Among them, the log to be processed can be a log obtained from any terminal. The matching rule library can be a database storing multiple matching rules. The target matching rule can be a matching rule with the highest current priority that can be selected in the matching rule library. It should be noted that after matching rule A in the matching rule library is selected as the target matching rule, when the target matching rule needs to be selected from the matching rule library again, matching rule A will no longer be selected as the target matching rule.

[0033] In an embodiment of the present invention, when a log to be processed is received, the priority order of each matching rule in the matching rule base can be determined first, and then the matching rule with the highest current priority that can be selected in the matching rule base can be used as the target matching rule according to the priority order of each matching rule.

[0034] In an optional embodiment of the present invention, the matching rule library may include: a template prefix tree matching rule, a simplest cycle matching rule, and a longest common subsequence matching rule.

[0035] Among them, the template prefix tree matching rule can be used to match the log to be processed and the log template according to the template prefix tree. The template prefix tree can be a dictionary tree composed of characters in the log template, which is used to store the character data in the log template in order. The simplest cycle matching rule can be used to match the log to be processed and the log template according to the data pointer. The longest common subsequence matching rule can be used to match the log to be processed and the log template according to the longest common character of the log to be processed and the log template. The priority of the template prefix tree matching rule is higher than the simplest cycle matching rule, and the priority of the simplest cycle matching rule is higher than the longest common subsequence matching rule. The embodiment of the present invention does not limit the specific matching rule type in the matching rule library, that is, the matching rule library may include matching rules other than the template prefix tree matching rule, the simplest cycle matching rule and the longest common subsequence matching rule.

[0036] S120: Using the target matching rule, an operation of matching the log to be processed with each log template in the log template library is performed.

[0037] The log template library may be a database storing multiple log templates. The log template may be used to represent the inherent format of the same type of log. For example, a log template may represent a type of system event.

[0038] In an embodiment of the present invention, after obtaining the target matching rule, the characters of the log to be processed and the characters of each log template in the log template library can be matched according to the target matching rule to determine whether there is a log template in the log template library that matches the log to be processed.

[0039] S130: If a log template matching the log to be processed is not obtained, the process returns to execute an operation of obtaining target matching rules from the matching rule library in sequence according to the priority order of each matching rule in the matching rule library.

[0040] In an embodiment of the present invention, if a log template that matches the log to be processed cannot be obtained according to the current target matching rule, another target matching rule that needs to be matched with the log to be processed can be determined from the matching rule library in turn according to the priority order of each matching rule in the matching rule library, and then the operation of matching the log to be processed with each log template in the log template library is performed according to the target matching rule determined again.

[0041] S140: When it is determined that a log template matching the log to be processed cannot be obtained by using all matching rules, the log to be processed is added to the log template library as a new log template.

[0042] In an embodiment of the present invention, when all matching rules are used to perform matching operations between the log to be processed and each log template in the order of priority of each matching rule, and no log template matching the log to be processed is obtained, the log to be processed can be used as a new log template and the new log template can be stored in the log template library.

[0043] In an optional embodiment of the present invention, after adding the log to be processed as a new log template to the log template library, the method may further include: updating the template prefix tree according to the characters of the log to be processed.

[0044] In an embodiment of the present invention, after adding the log to be processed as a new log template to the log template library, the characters of the log to be processed can be further converted into a path consisting of a root node and leaf nodes, and then the path corresponding to the log to be processed is added to the template prefix tree.

[0045] The technical solution of this embodiment is to obtain the target matching rules from the matching rule library in sequence according to the priority order of each matching rule in the matching rule library when receiving the log to be processed, and then use the target matching rules to perform the operation of matching the log to be processed with each log template in the log template library. If the log template matching the log to be processed cannot be obtained, then return to the operation of obtaining the target matching rules from the matching rule library in sequence according to the priority order of each matching rule in the matching rule library. When it is determined that the log template matching the log to be processed cannot be obtained using all matching rules, the log to be processed is added to the log template library as a new log template. This solution does not need to train a model for identifying log templates. Different matching rules can be used to directly determine the log template of the log to be processed. The log template that matches the new log generated after the system is updated can also be determined. In addition, the log template that matches the log to be processed is determined by different target matching rules, which can avoid the problem of high computational complexity caused by a single matching rule. It solves the problems of cumbersome training, high computational complexity and poor versatility of the log template extraction model in the prior art. It can reduce the computational complexity of log template extraction and improve the versatility of log template extraction without model training.

[0046] Embodiment 2

[0047] This embodiment is specific based on the above embodiment. In this embodiment, a specific optional implementation scheme is provided for using target matching rules to perform operations of matching the log to be processed with each log template in the log template library, and updating the log template and the template prefix tree.

[0048] The complexity of the longest common subsequence is O(n 2), if there are m log templates, the computational complexity will reach O(mn 2 ). As log templates continue to accumulate, the log template library will tend to be stable. The log templates in the log template library remain basically unchanged, no new log templates will be added, and the old log templates will not be updated. The longest common subsequence between the log to be processed and the old log template will be the same as the old log template, that is, the old log template constitutes a subsequence of the new log. Therefore, when using the longest common subsequence matching rule to determine the log template that matches the log to be processed, the pre-rule with lower computational complexity can be used to determine the matching of the log template and the log to be processed. When the ratio of the string length of the log template to the total string length of the log to be processed exceeds the ratio threshold, it is determined that the log template matches the log to be processed.

[0049] In an optional embodiment of the present invention, a target matching rule is used to perform an operation of matching the log to be processed with each log template in the log template library, which may include: when the currently acquired target matching rule is a template prefix tree matching rule, obtaining a template prefix tree that matches the log template library; sequentially matching each path included in the template prefix tree with the log to be processed, and when it is determined that the log to be processed completely matches the target path, calculating the ratio of the string length of the target path to the total string length in the log to be processed; if the ratio exceeds a preset ratio threshold, determining that the log to be processed matches the log template corresponding to the target path; otherwise, determining that the log template matching the log to be processed has not been obtained.

[0050] The ratio threshold may be a preset value used to determine whether the log to be processed matches the log template. A path in the template prefix tree corresponds to a log template in the log template library, and each path node in each path corresponds to a different character included in the log template. The target path may be a path in the template prefix tree that can completely match the log to be processed. The computational complexity of the template prefix tree matching rule is O(n).

[0051] In an embodiment of the present invention, if the target matching rule currently obtained is a template prefix tree matching rule, a template prefix tree matching the log template library is further determined. After obtaining the template prefix tree matching the log template library, the characters in the log to be processed can be further parsed, and the characters included in each path of the template prefix tree can be matched with the characters of the log to be processed in sequence. When the log to be processed completely matches the target path, the length of the string included in the target path and the total length of the string of the log to be processed can be calculated, and the proportion of the string length of the target path to the total length of the string in the log to be processed can be further calculated. If the proportion of the string length of the target path to the total length of the string in the log to be processed exceeds a preset ratio threshold, it can be determined that the log to be processed matches the log template corresponding to the target path, that is, the log to be processed belongs to the log template corresponding to the target path. If the proportion of the string length of the target path to the total length of the string in the log to be processed is lower than a preset ratio threshold, it can be determined that the log template corresponding to the log to be processed cannot be obtained according to the template prefix tree matching rule.

[0052] It should be noted that when the log template that matches the log to be processed is determined according to the template prefix tree matching rules, once the target path is determined, the log to be processed will not be matched with other paths in the template prefix tree. For example, assuming that the log to be processed is XAYBC, the template prefix tree includes two paths: root node -XY and root node -ABC. If the first character in the log to be processed matches X, the log to be processed will continue to match the path of the root node -XY and will no longer match other paths. A in the log to be processed cannot match Y in the path, and A will be skipped to match Y in the log to be processed with Y in the path. Since Y is the root node of the path root node -XY, the target path that fully matches the log to be processed is the root node -XY.

[0053] Figure 2 is a schematic diagram of a template prefix tree provided by Embodiment 2 of the present invention. In a specific example, Figure 2As shown, the template prefix tree includes 4 paths, namely root node-ACG, root node-ADH, root node-AE and root node-BF. Assuming that the log to be processed is ACXG, according to the template prefix tree matching rules, the paths included in the template prefix tree are matched with the word sequence of the log to be processed in sequence. Since the first character in the log to be processed is A, the path with the character A adjacent to the root node can be matched with the next character. When the root node-ACG in the template prefix tree is matched with the word sequence of the log to be processed, A and C in the log to be processed can match the first two characters in the path, while X in the log to be processed cannot match the next character G in the path. X in the log to be processed can be skipped, and G in the log to be processed can be matched with the next character G in the path. Since G is the leaf node of the path (the last node of the path), it can be determined that the log to be processed completely matches the path.

[0054] In an optional embodiment of the present invention, a target matching rule is used to perform an operation of matching the log to be processed with each log template in a log template library, which may include: when the currently acquired target matching rule is the simplest circular matching rule, obtaining the currently processed log template from the log template library in turn; creating a first pointer and a second pointer to point to the first character of the log to be processed and the currently processed log template respectively; iteratively executing the operation of simultaneously pointing the first pointer and the second pointer to the next character when the characters pointed to by the first pointer and the second pointer are the same, and only pointing the first pointer to the next character when the characters pointed to by the first pointer and the second pointer are different; when the first pointer moves to the last character of the log to be processed, detecting whether the second pointer is located at the last character of the currently processed log template; if so, determining that the log to be processed matches the currently processed log template when the ratio of the character string length of the currently processed log template to the total character string length in the log to be processed exceeds a preset ratio threshold; otherwise, returning to executing the operation of sequentially obtaining the currently processed log template from the log template library, and when it is determined that all log templates do not match the log to be processed, determining that the log template matching the log to be processed has not been obtained.

[0055] The first pointer may be a data pointer pointing to a character in the log to be processed. The second pointer may be a data pointer pointing to a character in the currently processed log template. The currently processed log template may be a log template that currently needs to be matched with the log to be processed.

[0056] In an embodiment of the present invention, if the log template that matches the log to be processed cannot be determined according to the template prefix tree matching rule, the simplest circular matching rule can be used as the current target matching rule, and the current processing log template can be obtained from the log template library in turn. After obtaining the current processing log template, a first pointer and a second pointer can be created. In the initial state, the first pointer points to the first character of the log to be processed, and the second pointer points to the first character of the current processing log template. When the character pointed to by the first pointer is the same as the character pointed to by the second pointer during iterative execution, the first pointer points to the next character of the log to be processed, and the second pointer points to the next character of the current processing log template. When the character pointed to by the first pointer is different from the character pointed to by the second pointer, only the first pointer points to the next character, and the character pointed to by the second pointer remains unchanged. If the first pointer points to the last character of the log to be processed, it is further detected whether the character pointed to by the second pointer is located at the last character of the current processing log template. If the first pointer and the second pointer point to the last character of the log to be processed and the currently processed log template respectively, then the ratio of the string length of the currently processed log template to the total string length in the log to be processed is further calculated. When the ratio of the string length of the currently processed log template to the total string length in the log to be processed exceeds the preset ratio threshold, it can be determined that the log to be processed matches the currently processed log template. If the first pointer points to the last character of the log to be processed and the second pointer does not point to the last character of the currently processed log template, or the first pointer and the second pointer point to the last character of the log to be processed and the currently processed log template respectively but the calculated ratio is lower than the ratio threshold, then the operation of obtaining the currently processed log template from the log template library in sequence is returned. If all log templates do not match the log to be processed, it can be determined that the log template matching the log to be processed cannot be obtained according to the simplest circular matching rule. Among them, the complexity of comparing a single log template with the log to be processed is O(n), and the complexity of comparing m log templates is O(mn).

[0057] Optionally, when determining the log template that matches the log to be processed based on the simplest circular matching rule, all log templates can be counted according to the string length. When the ratio of the string length of the log template to the total string length of the log to be processed is lower than the ratio threshold, the log template whose string length cannot meet the ratio threshold can be directly skipped to reduce the calculation complexity.

[0058] Exemplarily, assume that the current processing log template is Name Update is added. The log to be processed is NameUpdate 10 is added 8763, and the ratio threshold is 1 / 2. In the initial state, the first pointer points to Name, and the second pointer points to Name. Since the first pointer and the second pointer point to the same character, the first pointer is pointed to Update and the second pointer is pointed to Update. Since the first pointer and the second pointer point to the same character, the first pointer is pointed to 10 and the second pointer is pointed to is. At this time, the first pointer and the second pointer point to different characters, and only the first pointer is pointed to the next character is. Since the second pointer has not moved and points to the character is, the first pointer and the second pointer point to the same character. The first pointer points to the next character added, and the second pointer points to added. Since the two pointers point to the same character, the two pointers are moved at the same time. The first pointer points to 8763, and the second pointer points to the last character, so that the second pointer cannot move. At this time, the first pointer has moved to the last character of the log to be processed, and the second pointer points to the last character of the currently processed log template. The string length of the currently processed log template is 4, and the total string length in the log to be processed is 6. The string length of the currently processed log template accounts for more than 1 / 2 of the total string length in the log to be processed. At this time, it can be determined that the log to be processed matches the currently processed log template.

[0059] In an optional embodiment of the present invention, a target matching rule is used to perform an operation of matching the log to be processed with each log template in the log template library, which may include: when the currently acquired target matching rule is the longest common subsequence matching rule, respectively calculating the common subsequences of the log to be processed and each log template; determining the longest common subsequence in each common subsequence, the target log template to which the longest common subsequence belongs, and the string length of the target log template; calculating a first ratio of the string length of the longest common subsequence to the total string length in the log to be processed, and a second ratio of the string length of the longest common subsequence to the string length of the target log template; if both the first ratio and the second ratio exceed a preset ratio threshold, determining that the log to be processed matches the target log template; otherwise, determining that a log template matching the log to be processed has not been obtained.

[0060] The common subsequence may be a character sequence that exists in both the log to be processed and the log template. The longest common subsequence may be a sequence with the longest character length in the common subsequence. The target log template may be a log template with the longest common subsequence. The first ratio may be the ratio of the character string length of the longest common subsequence to the total character string length in the log to be processed. The second ratio may be the ratio of the character string length of the longest common subsequence to the character string length of the target log template.

[0061] In an embodiment of the present invention, when the log template matching the to-be-processed log cannot be determined according to the template prefix tree matching rule and the simplest loop matching rule, the longest common subsequence matching rule can be used as the target matching rule. When determining that the currently acquired target matching rule is the longest common subsequence matching rule, it is necessary to calculate the common subsequences of the to-be-processed log and each log template in the matching rule library respectively, and then compare the common subsequences to obtain the longest common subsequence. After obtaining the longest common subsequence, the log template to which the longest common subsequence belongs can be further determined, and the log template to which the longest common subsequence belongs is used as the target log template, and then the string length of the target log template is calculated. After obtaining the string length of the target log template, the first ratio of the string length of the longest common subsequence to the total string length in the to-be-processed log and the second ratio of the string length of the longest common subsequence to the string length of the target log template are further calculated. If both the first ratio and the second ratio exceed the preset ratio threshold, it can be determined that the to-be-processed log matches the target log template. If any of the first ratio and the second ratio is lower than the preset ratio threshold, it can be determined that the log template matching the to-be-processed log cannot be determined by the longest common subsequence matching rule.

[0062] Exemplarily, when the to-be-processed log s1 and the to-be-processed log s2 actually belong to two different log templates, and the ratio of the string length of the common subsequence of s1 and s2 to all the characters of s2 exceeds the ratio threshold, but the ratio of the character length of the common subsequence to all the characters of s1 is lower than the ratio threshold, it is assumed that a matching log template is first determined for s1, and then a matching log template is determined for s2. When processing s1, s1 is added to the log template library as a new log template. When determining a matching template for s2, s2 will be matched to the log template s1 according to the template prefix tree matching rule or the simplest matching rule, resulting in template recognition error. Under the condition that both the first ratio and the second ratio exceed the ratio threshold, it is determined that the to-be-processed log matches the target log template, which can prevent the to-be-processed log from matching the wrong log template.

[0063] In an optional embodiment of the present invention, after determining that the log to be processed matches the target log template, it can also include: obtaining at least one longest common subsequence corresponding to the target log template, and selecting a target longest common subsequence from the at least one longest common subsequence; using the target longest common subsequence as an update log template, and using the update log template to replace the target log template; updating the template prefix tree according to the characters of the target longest common subsequence.

[0064] The target longest common subsequence may be a sequence in the longest common subsequence.

[0065] In an embodiment of the present invention, after determining the target log template that matches the log to be processed according to the longest common subsequence matching rule, at least one longest common subsequence in the target log template corresponding to the log to be processed can be determined, and then the target longest common subsequence can be selected from the at least one longest common subsequence. After obtaining the target longest common subsequence, the target longest common subsequence can be used as an update log template, and then the update log template is stored in a log template library, and the target log template is deleted from the log template library. The characters included in the target longest common subsequence are parsed to construct a new path of the template prefix tree according to the characters of the target longest common subsequence, and the path of the target log template in the template prefix tree is further deleted.

[0066] Figure 3 This is a schematic diagram of a method for determining a log template provided by the second embodiment of the present invention. As shown in the figure, the log to be processed can be preprocessed first so that the log to be processed completes word segmentation and character lowercase conversion. Then, the log template that matches the log to be processed is determined by the template prefix tree matching rule. Specifically: sequentially match each path included in the template prefix tree with the log to be processed, and determine whether there is a target path that completely matches the log to be processed. If there is a target path and the ratio of the character string length of the target path to the total character string length in the log to be processed exceeds 1 / 2 (ratio threshold), then it is determined that the log to be processed matches the log template corresponding to the target path, otherwise it is determined that the log template that matches the log to be processed cannot be obtained.

[0067] When the log template that matches the log to be processed cannot be obtained through the template prefix tree matching rule, the log template that matches the log to be processed is determined through the simplest circular matching rule. Specifically: the log to be processed is matched with the currently processed log template. If the first pointer points to the last character of the log to be processed, the second pointer points to the last character of the currently processed log template, and the string length of the currently processed log template accounts for more than 1 / 2 of the total string length in the log to be processed, it is determined that the log to be processed matches the currently processed log template. Otherwise, the currently processed log template is obtained from the log template library in turn. When all log templates cannot match the log to be processed, it is determined that the log template that matches the log to be processed cannot be obtained.

[0068] In the case that the log template that matches the log to be processed cannot be obtained by the simplest circular matching rule, the log template that matches the log to be processed is determined by the longest common subsequence matching rule. Specifically, if the first ratio of the string length of the longest common subsequence to the total string length of the log to be processed and the second ratio of the string length of the longest common subsequence to the string length of the target log template both exceed the ratio threshold, it is determined that the log to be processed matches the target log template, otherwise it is determined that the log template that matches the log to be processed cannot be obtained. When the log template that matches the log to be processed cannot be obtained by the longest common subsequence matching rule, the log to be processed is added to the log template library as a new log template, and the template prefix tree is updated according to the log to be processed.

[0069] The technical solution of this embodiment is to obtain the target matching rules from the matching rule library in sequence according to the priority order of each matching rule in the matching rule library when receiving the log to be processed, and then use the target matching rules to perform the operation of matching the log to be processed with each log template in the log template library. If the log template matching the log to be processed cannot be obtained, then return to the operation of obtaining the target matching rules from the matching rule library in sequence according to the priority order of each matching rule in the matching rule library. When it is determined that the log template matching the log to be processed cannot be obtained using all matching rules, the log to be processed is added to the log template library as a new log template. This solution does not need to train a model for identifying log templates. Different matching rules can be used to directly determine the log template of the log to be processed. The log template that matches the new log generated after the system is updated can also be determined. In addition, the log template that matches the log to be processed is determined by different target matching rules, which can avoid the problem of high computational complexity caused by a single matching rule. It solves the problems of cumbersome training, high computational complexity and poor versatility of the log template extraction model in the prior art. It can reduce the computational complexity of log template extraction and improve the versatility of log template extraction without model training.

[0070] It should be noted that any arrangement and combination of the technical features in the above embodiments also falls within the protection scope of the present invention.

[0071] Embodiment 3

[0072] Figure 4 is a schematic diagram of a device for determining a log template provided by Embodiment 3 of the present invention, such as Figure 4 As shown, the device includes: a target matching rule acquisition module 210, a matching processing module 220, a loop execution module 230 and a log template library update module 240, wherein:

[0073] The target matching rule acquisition module 210 is used to obtain the target matching rules from the matching rule library in sequence according to the priority order of each matching rule in the matching rule library when receiving the log to be processed;

[0074] The matching processing module 220 is used to use the target matching rule to perform the operation of matching the log to be processed with each log template in the log template library;

[0075] The loop execution module 230 is used for returning to the operation of acquiring target matching rules from the matching rule base in order of priority of each matching rule in the matching rule base if a log template matching the log to be processed is not obtained;

[0076] The log template library updating module 240 is used to add the log to be processed as a new log template to the log template library when it is determined that no log template matching the log to be processed can be obtained by using all matching rules.

[0077] The technical solution of this embodiment is to obtain the target matching rules from the matching rule library in sequence according to the priority order of each matching rule in the matching rule library when receiving the log to be processed, and then use the target matching rules to perform the operation of matching the log to be processed with each log template in the log template library. If the log template matching the log to be processed cannot be obtained, then return to the operation of obtaining the target matching rules from the matching rule library in sequence according to the priority order of each matching rule in the matching rule library. When it is determined that the log template matching the log to be processed cannot be obtained using all matching rules, the log to be processed is added to the log template library as a new log template. This solution does not need to train a model for identifying log templates. Different matching rules can be used to directly determine the log template of the log to be processed. The log template that matches the new log generated after the system is updated can also be determined. In addition, the log template that matches the log to be processed is determined by different target matching rules, which can avoid the problem of high computational complexity caused by a single matching rule. It solves the problems of cumbersome training, high computational complexity and poor versatility of the log template extraction model in the prior art. It can reduce the computational complexity of log template extraction and improve the versatility of log template extraction without model training.

[0078] Optionally, the matching rule library includes: a template prefix tree matching rule, a simplest cycle matching rule and a longest common subsequence matching rule; wherein the template prefix tree matching rule has a higher priority than the simplest cycle matching rule, and the simplest cycle matching rule has a higher priority than the longest common subsequence matching rule.

[0079] Optionally, the matching processing module 220 is specifically used to: when the currently acquired target matching rule is a template prefix tree matching rule, obtain a template prefix tree that matches the log template library; wherein a path of the template prefix tree corresponds to a log template in the log template library, and each path node in each path corresponds to a different character included in the log template; sequentially match each path included in the template prefix tree with the log to be processed, and when it is determined that the log to be processed completely matches the target path, calculate the ratio of the character string length of the target path to the total character string length in the log to be processed; if the ratio exceeds a preset ratio threshold, it is determined that the log to be processed matches the log template corresponding to the target path; otherwise, it is determined that the log template matching the log to be processed has not been obtained.

[0080] Optionally, the matching processing module 220 is specifically used to: when the currently acquired target matching rule is the simplest circular matching rule, obtain the currently processed log template from the log template library in turn; create a first pointer and a second pointer to point to the first character of the log to be processed and the currently processed log template respectively; iteratively execute the operation of pointing the first pointer and the second pointer to the next character at the same time when the characters pointed to by the first pointer and the second pointer are the same, and only pointing the first pointer to the next character when the characters pointed to by the first pointer and the second pointer are different; when the first pointer moves to the last character of the log to be processed, detect whether the second pointer is located at the last character of the currently processed log template; if so, determine that the log to be processed matches the currently processed log template when the ratio of the string length of the currently processed log template to the total string length in the log to be processed exceeds a preset ratio threshold; otherwise, return to execute the operation of obtaining the currently processed log template from the log template library in turn, and when it is determined that all log templates do not match the log to be processed, determine that the log template matching the log to be processed has not been obtained.

[0081] Optionally, the matching processing module 220 is specifically used to: when the currently acquired target matching rule is the longest common subsequence matching rule, respectively calculate the common subsequences of the log to be processed and each log template; determine the longest common subsequence in each of the common subsequences, the target log template to which the longest common subsequence belongs, and the string length of the target log template; calculate a first ratio of the string length of the longest common subsequence to the total string length in the log to be processed, and a second ratio of the string length of the longest common subsequence to the string length of the target log template; if both the first ratio and the second ratio exceed a preset ratio threshold, determine that the log to be processed matches the target log template; otherwise, determine that a log template matching the log to be processed has not been obtained.

[0082] Optionally, the log template library update module 240 is specifically used to: obtain at least one longest common subsequence corresponding to the target log template, and select a target longest common subsequence from the at least one longest common subsequence; use the target longest common subsequence as an update log template, and use the update log template to replace the target log template.

[0083] Optionally, the log template determination device further includes a template prefix tree updating module, which is used to update the template prefix tree according to the characters of the target longest common subsequence.

[0084] Optionally, the template prefix tree updating module is specifically used to: update the template prefix tree according to the log characters to be processed.

[0085] The above-mentioned log template determination device can execute the log template determination method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method. For technical details not described in detail in this embodiment, please refer to the log template determination method provided by any embodiment of the present invention.

[0086] Since the above-mentioned device for determining a log template is a device that can execute the method for determining a log template in the embodiment of the present invention, based on the method for determining a log template in the embodiment of the present invention, a person skilled in the art can understand the specific implementation of the device for determining a log template in the present embodiment and its various variations, so how the device for determining a log template implements the method for determining a log template in the embodiment of the present invention will not be described in detail here. As long as a person skilled in the art implements the device used by the method for determining a log template in the embodiment of the present invention, it belongs to the scope of protection of this application.

[0087] Embodiment 4

[0088] Figure 5 A schematic diagram of the structure of an electronic device provided in Embodiment 4 of the present invention. Figure 5 A block diagram of an electronic device 312 suitable for implementing embodiments of the present invention is shown. Figure 5 The electronic device 312 shown is only an example and should not limit the functions and scope of use of the embodiments of the present invention. The electronic device 312 may be, for example, a computer device or a server device.

[0089] like Figure 5 As shown, the electronic device 312 is in the form of a general purpose computing device. The components of the electronic device 312 may include, but are not limited to: one or more processors 316, a storage device 328, and a bus 318 connecting different system components (including the storage device 328 and the processor 316).

[0090] Bus 318 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. For example, these architectures include but are not limited to Industry Standard Architecture (ISA) bus, MicroChannel Architecture (MCA) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.

[0091] The electronic device 312 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device 312, including volatile and non-volatile media, removable and non-removable media.

[0092] The storage device 328 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 330 and / or cache memory 332. The electronic device 312 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 334 may be used to read and write non-removable, non-volatile magnetic media ( Figure 5 not shown, usually called a "hard drive"). Although Figure 5 Not shown in the figure, a disk drive for reading and writing a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing a removable non-volatile optical disk (e.g., a read-only optical disk (Compact Disc-ReadOnly Memory, CD-ROM), a digital video disk (Digital Video Disc-Read Only Memory, DVD-ROM) or other optical media) may be provided. In these cases, each drive may be connected to the bus 318 via one or more data medium interfaces. The storage device 328 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present invention.

[0093] A program 336 having a set (at least one) of program modules 326 may be stored, for example, in a storage device 328, such program modules 326 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules 326 generally perform the functions and / or methods of the embodiments described herein.

[0094] The electronic device 312 may also communicate with one or more external devices 314 (e.g., keyboards, pointing devices, cameras, displays 324, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 312, and / or communicate with any device that enables the electronic device 312 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication may be performed through an input / output (I / O) interface 322. In addition, the electronic device 312 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN) and / or a public network, such as the Internet) through a network adapter 320. As shown, the network adapter 320 communicates with other modules of the electronic device 312 through a bus 318. It should be understood that although not shown in the figure, other hardware and / or software modules can be used in conjunction with the electronic device 312, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, disk arrays (Redundant Arrays of Independent Disks, RAID) systems, tape drives, and data backup storage systems.

[0095] The processor 316 executes various functional applications and data processing by running the program stored in the storage device 328, for example, the method for determining the log template provided in the above embodiment of the present invention is implemented: when receiving the log to be processed, the target matching rules are obtained from the matching rule library in sequence according to the priority order of each matching rule in the matching rule library; the target matching rules are used to perform the operation of matching the log to be processed with each log template in the log template library; if the log template matching the log to be processed cannot be obtained, the operation of obtaining the target matching rules from the matching rule library in sequence according to the priority order of each matching rule in the matching rule library is returned; when it is determined that the log template matching the log to be processed cannot be obtained using all matching rules, the log to be processed is added to the log template library as a new log template.

[0096] The technical solution of this embodiment is to obtain the target matching rules from the matching rule library in sequence according to the priority order of each matching rule in the matching rule library when receiving the log to be processed, and then use the target matching rules to perform the operation of matching the log to be processed with each log template in the log template library. If the log template matching the log to be processed cannot be obtained, then return to the operation of obtaining the target matching rules from the matching rule library in sequence according to the priority order of each matching rule in the matching rule library. When it is determined that the log template matching the log to be processed cannot be obtained using all matching rules, the log to be processed is added to the log template library as a new log template. This solution does not need to train a model for identifying log templates. Different matching rules can be used to directly determine the log template of the log to be processed. The log template that matches the new log generated after the system is updated can also be determined. In addition, the log template that matches the log to be processed is determined by different target matching rules, which can avoid the problem of high computational complexity caused by a single matching rule. It solves the problems of cumbersome training, high computational complexity and poor versatility of the log template extraction model in the prior art. It can reduce the computational complexity of log template extraction and improve the versatility of log template extraction without model training.

[0097] Embodiment 5

[0098] Embodiment 5 of the present invention further provides a computer storage medium storing a computer program, wherein the computer program is used to execute the method for determining a log template described in any of the above embodiments of the present invention when executed by a computer processor: when receiving a log to be processed, obtaining target matching rules from the matching rule library in sequence according to the priority order of each matching rule in the matching rule library; using the target matching rules, performing an operation of matching the log to be processed with each log template in the log template library; if a log template matching the log to be processed cannot be obtained, returning to perform an operation of obtaining target matching rules from the matching rule library in sequence according to the priority order of each matching rule in the matching rule library; when it is determined that a log template matching the log to be processed cannot be obtained using all matching rules, adding the log to be processed as a new log template to the log template library.

[0099] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, a device or a device or used in combination with it.

[0100] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0101] The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wire, optical cable, radio frequency (RF), etc., or any suitable combination of the foregoing.

[0102] Computer program code for performing the operations of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0103] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A method for determining a log template, characterized in that: include: When receiving the log to be processed, the target matching rules are obtained from the matching rule library in sequence according to the priority order of each matching rule in the matching rule library; The matching rule library includes: a template prefix tree matching rule, a simplest cycle matching rule and a longest common subsequence matching rule; wherein the template prefix tree matching rule has a higher priority than the simplest cycle matching rule, and the simplest cycle matching rule has a higher priority than the longest common subsequence matching rule; Using a target matching rule, an operation of matching the log to be processed with each log template in a log template library is performed; If the log template matching the log to be processed cannot be obtained, the operation of obtaining the target matching rules from the matching rule base in sequence according to the priority order of each matching rule in the matching rule base is returned; When it is determined that a log template matching the log to be processed cannot be obtained by using all matching rules, the log to be processed is added to the log template library as a new log template; The operation of using the target matching rule to match the log to be processed with each log template in the log template library includes: When the currently acquired target matching rule is the longest common subsequence matching rule, respectively calculate the common subsequences of the log to be processed and each log template; determine the longest common subsequence in each of the common subsequences, the target log template to which the longest common subsequence belongs, and the string length of the target log template; calculate a first ratio of the string length of the longest common subsequence to the total string length in the log to be processed, and a second ratio of the string length of the longest common subsequence to the string length of the target log template; if both the first ratio and the second ratio exceed a preset ratio threshold, determine that the log to be processed matches the target log template; otherwise, determine that a log template matching the log to be processed has not been obtained; Acquire at least one longest common subsequence corresponding to the target log template, and select a target longest common subsequence from the at least one longest common subsequence; use the target longest common subsequence as an update log template, and use the update log template to replace the target log template; update the template prefix tree according to the characters of the target longest common subsequence.

2. The method according to claim 1, characterized in that The target matching rule is adopted to perform the operation of matching the log to be processed with each log template in the log template library, and further includes: When the currently acquired target matching rule is a template prefix tree matching rule, acquiring a template prefix tree matching the log template library; Wherein, a path of the template prefix tree corresponds to a log template in the log template library, and each path node in each path corresponds to a different character included in the log template; Sequentially matching each path included in the template prefix tree with the log to be processed, and when it is determined that the log to be processed completely matches the target path, calculating the ratio of the character string length of the target path to the total character string length in the log to be processed; If the ratio exceeds a preset ratio threshold, it is determined that the log to be processed matches the log template corresponding to the target path; otherwise, it is determined that the log template matching the log to be processed is not obtained.

3. The method according to claim 1, characterized in that The target matching rule is adopted to perform the operation of matching the log to be processed with each log template in the log template library, and further includes: When the currently acquired target matching rule is the simplest circular matching rule, the currently processed log template is acquired from the log template library in sequence; Create a first pointer and a second pointer to point to the first character of the log to be processed and the current processing log template respectively; Iteratively executing an operation in which when the characters pointed to by the first pointer and the second pointer are the same, the first pointer and the second pointer are simultaneously pointed to the next character, and when the characters pointed to by the first pointer and the second pointer are different, only the first pointer is pointed to the next character; When the first pointer moves to the last character of the log to be processed, detecting whether the second pointer is located at the last character of the currently processed log template; If so, then when the ratio of the string length of the currently processed log template to the total string length in the log to be processed exceeds a preset ratio threshold, it is determined that the log to be processed matches the currently processed log template; otherwise, return to execute the operation of obtaining the currently processed log template from the log template library in sequence, and when it is determined that all log templates do not match the log to be processed, it is determined that the log template matching the log to be processed has not been obtained.

4. The method according to claim 1, characterized in that: After adding the log to be processed as a new log template to the log template library, it also includes: The template prefix tree is updated according to the log characters to be processed.

5. A device for determining a log template, characterized in that: include: The target matching rule acquisition module is used to obtain the target matching rules from the matching rule library in sequence according to the priority order of each matching rule in the matching rule library when receiving the log to be processed; The matching rule library includes: a template prefix tree matching rule, a simplest cycle matching rule and a longest common subsequence matching rule; wherein the template prefix tree matching rule has a higher priority than the simplest cycle matching rule, and the simplest cycle matching rule has a higher priority than the longest common subsequence matching rule; A matching processing module, used for matching the log to be processed with each log template in the log template library by using a target matching rule; A loop execution module, for returning to execute the operation of obtaining target matching rules from the matching rule base in sequence according to the priority order of each matching rule in the matching rule base if a log template matching the log to be processed is not obtained; The log template library update module is used to add the log to be processed as a new log template to the log template library when it is determined that the log template matching the log to be processed cannot be obtained by using all matching rules; The matching processing module is specifically used to calculate the common subsequences of the log to be processed and each log template respectively when the currently acquired target matching rule is the longest common subsequence matching rule; determine the longest common subsequence in each of the common subsequences, the target log template to which the longest common subsequence belongs, and the string length of the target log template; calculate a first ratio of the string length of the longest common subsequence to the total string length in the log to be processed, and a second ratio of the string length of the longest common subsequence to the string length of the target log template; if both the first ratio and the second ratio exceed a preset ratio threshold, determine that the log to be processed matches the target log template; otherwise, determine that a log template matching the log to be processed has not been obtained; a log template library update module, configured to obtain at least one longest common subsequence corresponding to the target log template, and select a target longest common subsequence from the at least one longest common subsequence; use the target longest common subsequence as an update log template, and use the update log template to replace the target log template; The template prefix tree updating module is used to update the template prefix tree according to the characters of the target longest common subsequence.

6. An electronic device, characterized in that: The electronic device comprises: one or more processors; A storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method for determining a log template as described in any one of claims 1 to 4.

7. A computer storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method for determining a log template as described in any one of claims 1 to 4 is implemented.

Images