Log pattern recognition method and device, computer readable storage medium, terminal

Abstract

A log pattern recognition method and device, a computer readable storage medium and a terminal, the method comprising: performing word segmentation on a log to be recognized to obtain one or more words; performing layer-by-layer search on a preset search tree using the number of words and each word; when an internal node can be reached to a leaf node layer, and the maximum value of the first similarity between each search word and the stored log pattern in the reached leaf node is less than a first threshold, or when the internal node cannot be searched, the second similarity between each search word and a preset number of stored log patterns selected is calculated respectively; comparing the maximum value of the second similarity with a second threshold to determine whether to use the longest common subsequence between the stored log pattern corresponding to the maximum value of the second similarity and each search word as the pattern of the log to be recognized. The present application helps to improve the accuracy of log pattern recognition.

Description

Technical Field

[0001] The present invention relates to the field of log recognition technology, and in particular to a log pattern recognition method and apparatus, a computer-readable storage medium, and a terminal. Background Technology

[0002] With the continuous development of information technology, the complexity of information systems is gradually increasing, and the log data they generate is exceptionally massive. Manually reviewing logs for information system maintenance and problem localization has become impossible. Therefore, an automated method is needed to extract information from logs. Since logs are semi-structured text data that follows certain templates, and these templates are fixed expressions of a specific node or problem, they contain important information. Therefore, extracting one or more templates from massive logs is essential; the process of extracting templates is essentially the process of identifying log patterns.

[0003] In existing technologies, log pattern recognition is often based on the assumption that different log lengths necessarily indicate different patterns. For example, the classic log pattern recognition algorithm, Drain, forcibly assumes that logs with the same pattern must have the same length, and uses a trie / prefix tree to search and build internal nodes to achieve log pattern recognition. However, in reality, two highly similar logs do not necessarily have the same length. Therefore, relying on the assumption that "different log lengths necessarily indicate different patterns" for log pattern recognition can lead to the failure to identify some logs with different lengths but the same actual pattern, resulting in insufficient accuracy of the log pattern recognition results. Summary of the Invention

[0004] The technical problem solved by the embodiments of the present invention is that the accuracy of log pattern recognition results is insufficient because the prior art relies on the assumption that "different log lengths necessarily lead to different patterns" for Japanese-style pattern recognition.

[0005] To achieve the above objectives, embodiments of the present invention provide a log pattern recognition method, comprising the following steps: performing word segmentation on the log to be recognized to obtain one or more words; using the number of words and the search terms for each word, performing a layer-by-layer search on a preset search tree, wherein the search tree includes a search term number node layer, one or more internal node layers, and a leaf node layer storing one or more existing log patterns; when the leaf node layer can be reached via existing internal nodes, and the maximum value of the first similarity between each search term and the existing log pattern in the reached leaf node is less than a first threshold, or when no internal node is found, selecting a preset number of existing log patterns from all existing log patterns, and calculating a second similarity between the selected existing log patterns and each search term respectively; comparing the maximum value of each obtained second similarity with a second threshold to determine whether to use the longest common subsequence between the existing log pattern corresponding to the maximum value of the second similarity and each search term as the pattern of the log to be recognized.

[0006] Optionally, the method further includes: when no internal node is found, using the search term, taking the previously found internal node as the parent node, adding internal nodes layer by layer in the search tree until the leaf node layer is reached; wherein, the order of the search terms corresponds one-to-one with the internal node layers of the search tree.

[0007] Optionally, the method further includes: when the leaf node layer can be reached via the existing internal nodes, and the maximum value of the first similarity between each search term and the existing log pattern in the reached leaf node is less than a first threshold, storing the search sequence formed by each search term as a newly added existing log pattern in the reached leaf node; or, when the internal node cannot be found, storing the search sequence formed by each search term as a newly added existing log pattern in the leaf node under the newly added internal node.

[0008] Optionally, before performing a layer-by-layer search on the preset search tree, the method further includes: determining the category of each word; and for each word, determining whether to use the word as the search term or the attribute of the word as the search term based on the word's category.

[0009] Optionally, the category of the word includes English words pre-stored in a preset constant set, as well as other words; determining whether to use the word as the search term or the attribute of the word as the search term based on the category of the word includes: for each word, if the word is an English word pre-stored in the preset constant set, then the word is used as the search term; if the word is one of the other words, then the attribute of the word is used as the search term.

[0010] Optionally, before selecting a preset number of existing log patterns from all existing log patterns, the method further includes: determining a first similarity between the existing log patterns in the reached leaf nodes and each search term using the following formula:

[0011]

[0012]

[0013] Where sim1 indicates the first similarity between each pattern word of the existing log pattern in the arrived leaf node and each search word, f() indicates the first similarity calculation function, N indicates the length value or number of pattern words of the existing log pattern in the arrived leaf node, and a i Used to indicate the i-th search term determined according to the word segmentation order among various search terms, b i This is used to indicate the i-th pattern word of each stored log pattern in the leaf node that has been reached.

[0014] Optionally, selecting a preset number of existing log patterns from all existing log patterns includes: calculating the distance between the vector of each existing log pattern and the vector of each search term; and determining the preset number of existing log patterns that are closest to each other.

[0015] Optionally, one or more of the following conditions must be met: the vector is selected from: hash vector, word frequency and inverse text frequency index TFIDF vector, word2vec vector; the distance is selected from: Euclidean distance, Mahalanobis distance, Manhattan distance.

[0016] Optionally, based on the following formula, the second similarity with each search term is calculated separately using the selected existing log pattern:

[0017]

[0018]

[0019] Where sim2 indicates the second similarity between each pattern word of the selected stored log pattern and each search word, C() indicates the iterative function for calculating the second similarity, max() indicates the maximum value calculation function, and x i Used to indicate the i-th search term determined according to the word segmentation order among various search terms, y j Used to indicate the j-th pattern word among the selected stored log patterns; during the iteration process, the initial values ​​of i and j are l respectively. x and l y , l x Used to indicate the length value of each search term or the number of search terms, l yUsed to indicate the length value or number of pattern words of the selected stored log pattern.

[0020] Optionally, the method further includes: when it is possible to reach the leaf node layer via existing internal nodes, and the maximum value of the first similarity between each search term and the existing log pattern in the reached leaf node is greater than or equal to the first threshold, then the existing log pattern corresponding to the maximum value of the first similarity is fused with each search term to obtain an updated log pattern; the updated log pattern is used to replace the existing log pattern corresponding to the maximum value of the first similarity in the reached leaf node; a fused log pattern corresponding to the replaced existing log pattern is determined; and the corresponding fused log pattern is updated in the fused log pattern set; wherein the fused log pattern set is used to store the fused log pattern.

[0021] Optionally, updating the corresponding fusion log pattern in the fusion log pattern set includes: performing fusion processing on the corresponding fusion log pattern and the update log pattern; and replacing the corresponding fusion log pattern with the processed fusion log pattern.

[0022] Optionally, fusing the existing log pattern corresponding to the maximum value of the first similarity with each search term to obtain an update log pattern includes: determining whether each search term is the same as a pattern word in the same order in the existing log pattern corresponding to the maximum value of the first similarity, based on the word segmentation order in each search term; if they are the same, then the search term is used as the pattern word in the same order in the update log pattern; if they are different, then a wildcard is used as the pattern word in the same order in the update log pattern.

[0023] Optionally, the first threshold is determined at least based on the number of digits and the number of wildcards in the search terms; wherein, the larger the number of digits, the smaller the first threshold, and the larger the number of wildcards, the smaller the first threshold; wherein each search term is determined based on a word, and each search term is the same as or different from its corresponding word.

[0024] Optionally, determining the first threshold based on the number of digits and wildcards in the search term includes: determining a first intermediate coefficient based on the quotient between the number of digits in the search term and the total number of search terms, and determining a second intermediate coefficient based on the number of digits in the search term and the number of wildcards in the search term; and determining the first threshold based on the weighted calculation result of the first intermediate coefficient and the second intermediate coefficient using a preset weight ratio.

[0025] Optionally, the first threshold can be determined using the following formula:

[0026] st = min{1, stinit +0.5×log base (η+1)};

[0027]

[0028] base = max{2, digLen+1};

[0029] Where st indicates the first threshold, seqLen indicates the total number of search terms, digLen indicates the number of digits in each search term, η indicates the number of wildcards in each search term, min{} indicates the minimum value function, max{} indicates the maximum value function, st1 is the first intermediate coefficient, and st2 is the second intermediate coefficient.

[0030] Optionally, comparing the maximum value of each obtained second similarity with a second threshold to determine whether to use the longest common subsequence between the existing log pattern corresponding to the maximum value of the second similarity and each search term as the pattern of the log to be identified includes: if the second similarity between each pattern word of the existing log pattern corresponding to the maximum value of the second similarity and each search term is less than the second threshold, then determining to use the longest common subsequence between the existing log pattern corresponding to the maximum value of the second similarity and each search term as the pattern of the log to be identified.

[0031] Optionally, the process of segmenting the log to be identified to obtain one or more words includes: using the Named Entity Recognition (NER) method to segment the log to be identified to obtain one or more words.

[0032] This invention also provides a log pattern recognition device, characterized in that it includes: a word segmentation processing module, used to segment the log to be recognized into words to obtain one or more words; a layer-by-layer search module, used to perform a layer-by-layer search on a preset search tree using the number of words and the search terms for each word, wherein the search tree includes a search term number node layer, one or more internal node layers, and a leaf node layer storing one or more existing log patterns; a similarity calculation module, used to select a preset number of existing log patterns from all existing log patterns and calculate a second similarity between the selected existing log patterns and each search term when the leaf node layer can be reached via existing internal nodes, and the maximum value of the first similarity between each search term and the existing log pattern in the reached leaf node is less than a first threshold, or when no internal node is found, and calculate a second similarity between the selected existing log patterns and each search term; and a log pattern determination module, used to compare the maximum value of each obtained second similarity with a second threshold to determine whether to use the longest common subsequence between the existing log pattern corresponding to the maximum value of the second similarity and each search term as the pattern of the log to be recognized.

[0033] This invention also provides a computer-readable storage medium storing a computer program thereon, wherein the computer program, when run by a processor, executes the steps of the above-described log pattern recognition method.

[0034] This invention also provides a terminal, including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the processor executes the steps of the above-described log pattern recognition method when running the computer program.

[0035] Compared with the prior art, the technical solution of the embodiments of the present invention has the following beneficial effects:

[0036] In this embodiment of the invention, the log to be identified is segmented to obtain one or more words; using the number of words and the search terms for each word, a preset search tree is searched layer by layer, wherein the search tree includes a search term node layer, one or more internal node layers, and a leaf node layer storing one or more existing log patterns; when the leaf node layer can be reached through existing internal nodes, and the maximum value of the first similarity between each search term and the existing log pattern in the reached leaf node is less than a first threshold, or when no internal node is found, a preset number of existing log patterns are selected from all existing log patterns, and a second similarity between the selected existing log patterns and each search term is calculated; the maximum value of each obtained second similarity is compared with a second threshold to determine whether to use the longest common subsequence between the existing log pattern corresponding to the maximum value of the second similarity and each search term as the pattern of the log to be identified. Compared to existing technologies that rely on the assumption that different log lengths necessarily correspond to different patterns for log pattern recognition, the accuracy of the above-mentioned scheme may be insufficient, especially in identifying logs with different lengths but the same actual pattern. The embodiments of this invention divide log pattern recognition into two stages. In the first stage, based on the assumptions of existing technologies, a pre-defined search tree is searched layer by layer to determine whether to use the search sequence formed by each search term as the initial recognition result. In the second stage, based on the initial recognition result, the second similarity and a second threshold of each search term with a pre-defined number of existing log patterns are compared, and the final recognition result is determined by finding the maximum common subsequence. Because the second stage uses existing log patterns stored in leaf nodes of nodes with different numbers of search terms in the search tree (where the number of pattern words differs from the number of search terms) during the calculation of the second similarity, it helps to merge log patterns with different lengths but high similarity into one pattern, improving recognition accuracy while maintaining efficiency. Furthermore, compared to traditional log pattern recognition, which typically uses clustering to load all logs to be identified into memory at once, resulting in significant storage and computational overhead, this invention can process the logs to be identified in a streaming manner, processing one log at a time, thus reducing overhead and improving efficiency.

[0037] Furthermore, when no internal node is found, the search term is used, with the previously found internal node as the parent node, and internal nodes are added layer by layer in the search tree until the leaf node layer is reached; wherein, the order of the search terms corresponds one-to-one with the internal node layers of the search tree. Using the above scheme, during the layer-by-layer search of the preset search tree, a search term count node can be added at the search term count node layer to store the number of words not found, internal nodes can be added at the internal node layer to store the search terms not found, and leaf nodes can be added at the leaf node layer to store the search sequence of search terms not found. Therefore, when practically applied to the identification of log sets containing a large number of logs to be identified, the identification results can be determined efficiently and accurately, and the search tree can also be updated and improved.

[0038] Furthermore, when the leaf node layer can be reached via existing internal nodes, and the maximum value of the first similarity between each search term and the existing log pattern in the reached leaf node is less than a first threshold, the search sequence formed by each search term is stored as a newly added existing log pattern in the reached leaf node; or, when no internal node is found, the search sequence formed by each search term is stored as a newly added existing log pattern in the leaf node under the newly added internal node. In this embodiment of the invention, during the process of performing a layer-by-layer search on the preset search tree, the existing log patterns stored in the reached or newly added leaf nodes can also be updated to continuously update the historical records stored in the search tree.

[0039] Furthermore, selecting a preset number of existing log patterns from all existing log patterns includes: calculating the distance between the vector of each existing log pattern and the vector of each search term; and determining the preset number of existing log patterns that are closest to each other. Compared to directly traversing text characters to calculate the similarity between each log pattern and then determining the preset number of existing log patterns, which is computationally expensive and inefficient, this embodiment of the invention uses a method of first vectorizing each log pattern and then calculating the distance between the vectors to determine the preset number of existing log patterns that are closest to each other. This helps to improve computational efficiency and thus improve log pattern recognition efficiency.

[0040] Furthermore, the first threshold is determined at least based on the number of digits and wildcards in the search terms; wherein, the larger the number of digits, the smaller the first threshold, and the larger the number of wildcards, the smaller the first threshold; wherein each search term is determined based on a word, and each search term is the same as or different from the corresponding word. Compared to setting a fixed value as the similarity threshold in existing log pattern recognition, when the log length is fixed and it contains many known variables (numbers, IP addresses, etc.), the proportion of constants can be considered reduced, so a smaller threshold is more reasonable for calculating similarity. Conversely, when there are fewer known variables, a larger threshold is more reasonable, but the existing technology does not consider this situation, and using a fixed threshold may lead to insufficient accuracy. The embodiments of the present invention adopt an adaptive threshold method, dynamically adjusting the threshold based on the information of known variables (i.e., the number of search terms that are inconsistent with the word), which helps to improve the accuracy of the recognition results.

[0041] Furthermore, the process of segmenting the log to be identified to obtain one or more words includes: using the Named Entity Recognition (NER) method to segment the log to obtain one or more words. Due to the arbitrary nature of log text writing, simple delimiters cannot effectively segment the log. Therefore, this embodiment of the invention employs a named entity recognition-based approach, which can accurately segment the log to be identified to determine each word (token), and also accurately determine the attributes (tags) of each word, thus improving the accuracy of subsequent searches and related calculations. Attached Figure Description

[0042] Figure 1 This is a flowchart of the first log pattern recognition method in the embodiments of the present invention;

[0043] Figure 2 This is a partial flowchart of the second log pattern recognition method in this embodiment of the invention;

[0044] Figure 3 This is a schematic diagram of the specific structure of a search tree in an embodiment of the present invention;

[0045] Figure 4 This is a partial flowchart of the third log pattern recognition method in this embodiment of the invention;

[0046] Figure 5 This is a schematic diagram of the structure of a log pattern recognition device according to an embodiment of the present invention. Detailed Implementation

[0047] As mentioned earlier, with the continuous development of information technology, log data is extremely large, making it essential to identify log patterns from massive amounts of logs.

[0048] Existing technologies often rely on the assumption that different log lengths necessarily indicate different patterns for log pattern recognition. For example, the classic log pattern recognition algorithm currently in use, the Drain algorithm, forces the assumption that the same pattern must have the same log length. It uses a trie / prefix tree to search and build internal nodes to achieve log pattern recognition.

[0049] The inventors of this invention discovered through research that, in reality, two highly similar logs do not necessarily have the same length. Therefore, relying on the assumption that "different log lengths necessarily mean different patterns" for Japanese-style pattern recognition will result in some logs with different lengths but the same actual pattern not being recognized, thus leading to insufficient accuracy in log pattern recognition results.

[0050] In this embodiment of the invention, the log to be identified is segmented to obtain one or more words; using the number of words and the search terms for each word, a preset search tree is searched layer by layer, wherein the search tree includes a search term node layer, one or more internal node layers, and a leaf node layer storing one or more existing log patterns; when the leaf node layer can be reached through existing internal nodes, and the maximum value of the first similarity between each search term and the existing log pattern in the reached leaf node is less than a first threshold, or when no internal node is found, a preset number of existing log patterns are selected from all existing log patterns, and a second similarity between the selected existing log patterns and each search term is calculated; the maximum value of each obtained second similarity is compared with a second threshold to determine whether to use the longest common subsequence between the existing log pattern corresponding to the maximum value of the second similarity and each search term as the pattern of the log to be identified, compared with existing... The existing technology relies on the assumption that different log lengths necessarily correspond to different patterns for log pattern recognition, which may result in insufficient accuracy, especially in failing to identify logs of different lengths but with the same actual pattern. This invention divides log pattern recognition into two stages. In the first stage, based on the assumptions of the prior art, a pre-defined search tree is searched layer by layer to determine whether to use the search sequence formed by each search term as the initial recognition result. In the second stage, based on the initial recognition result, the second similarity of each search term with a pre-defined number of existing log patterns is compared with a second threshold, and the final recognition result is determined by finding the maximum common subsequence. Since the second stage uses existing log patterns stored in leaf nodes of nodes with different numbers of search terms in the search tree (where the number of pattern words differs from the number of search terms) during the calculation of the second similarity, it helps to merge log patterns of different lengths but high similarity into one pattern, improving recognition accuracy while maintaining efficiency. Furthermore, compared to traditional log pattern recognition that typically uses clustering to load all logs to be recognized into memory at once, resulting in significant storage and computational overhead, this invention can process logs in a streaming manner, processing one log at a time, reducing overhead and improving efficiency.

[0051] To make the above-mentioned objectives, features and beneficial effects of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0052] Reference Figure 1 , Figure 1 This is a flowchart of the first log pattern recognition method in an embodiment of the present invention. The first log pattern recognition method may include steps S11 to S14:

[0053] Step S11: Perform word segmentation on the log to be recognized to obtain one or more words;

[0054] Step S12: Using the number of words and the search terms for each word, perform a layer-by-layer search on the preset search tree, wherein the search tree includes a search term number node layer, one or more internal node layers, and a leaf node layer that stores one or more existing log patterns.

[0055] Step S13: When the leaf node layer can be reached through the existing internal nodes, and the maximum value of the first similarity between each search term and the existing log pattern in the reached leaf node is less than the first threshold, or when the internal node cannot be found, a preset number of existing log patterns are selected from all existing log patterns, and the second similarity between the selected existing log patterns and each search term is calculated using the selected existing log patterns.

[0056] Step S14: Compare the maximum value of each obtained second similarity with the second threshold to determine whether to use the longest common subsequence between the existing log pattern corresponding to the maximum value of the second similarity and each search term as the pattern of the log to be identified.

[0057] In the specific implementation of step S11, the log to be identified can be event records generated by network devices, systems, and service programs during operation. Each log entry can record a description of the relevant operation, such as the URL, date, time, user, and action. Network operating systems often have various log files, such as application logs, security logs, system logs, scheduler service logs, FTP logs, WWW logs, DNS server logs, Domain Name System (DNS) logs, etc. These logs vary depending on the services enabled by the system. For example, when a user performs inter-process communication (IPC) probing on the system, the system will quickly record the URL, IP address, time, and username used by the probe in the security log. After probing with File Transfer Protocol (FTP), the system will record the IP address, time, and username used for probing in the FTP log.

[0058] Understandably, each log entry to be identified contains one or more words (also known as tokens). For example, the log entry "Receive from node 001" has four tokens: Receive, from, node, and 001. Each token has its own attributes (in the log domain, the attribute categories of each token can be denoted as tags), including but not limited to the following attribute categories: <time>Indicates time, <num>Representing numbers, <date>Indicates the date. <url>Indicates the URL. <ip>Indicates IP address, <file>Indicates the file name. <path>This indicates the path of a file in the calculation. <email>Indicates an email address. <en>Representing English words, <cn>The symbols represent Chinese words, and <*> represent wildcards. Wildcards are typically used to represent variables in log patterns.

[0059] Furthermore, as a non-limiting embodiment, the Named Entity Recognition (NER) method can be used to segment the log to be recognized to obtain one or more words.

[0060] Named Entity Recognition (NER, also known as entity recognition, entity segmentation, and entity extraction) is a subtask of information extraction. It aims to locate and classify named entities in text into predefined categories, such as people, organizations, locations, time expressions, quantities, currency values, percentages, etc. From a model perspective, it can be categorized into rule-based methods, unsupervised learning methods, and supervised learning methods. From an input perspective, it can be categorized into character-level methods, word-level methods, and methods combining both. In specific implementations, other methods capable of accurately segmenting the log data to be recognized can also be used; this embodiment of the invention does not limit these methods.

[0061] In this embodiment of the invention, due to the arbitrary nature of log text writing, simple delimiters cannot effectively segment the log. Therefore, by adopting a named entity recognition-based approach, the log to be identified can be accurately segmented to determine each word (token), and the attributes (tags) of each word can also be accurately determined, which helps to improve the accuracy of subsequent searches and related calculations.

[0062] In a specific implementation of step S12, the search tree is a tree-shaped data structure that may contain historical records of additions or updates made during the identification of one or more log entries to be identified. In some non-limiting embodiments, the search tree may be a trie, a prefix tree, or a word lookup tree, specifically including a layer of search term count nodes, one or more internal node layers, and a leaf node layer storing one or more existing log patterns.

[0063] In this search tree, each leaf node has the same depth. The depth of the search tree can be reasonably set based on the length (number of words) of the log to be identified. In some non-limiting embodiments, the search tree depth can be selected from a suitable value between 5 and 20.

[0064] The search term count node layer can be located below the root node of the search tree, serving as the first layer of the search tree. It stores the number of words in the log to be identified, allowing for the rapid determination of the parent node for subsequent searches based on the number of words in the log to be identified, thus improving the efficiency of log pattern recognition. One or more internal node layers can be located below the search term count node layer, storing the search terms for each word. The leaf node layer can be located below the last internal node layer, storing the search sequence formed by each search term during the process of searching layer by layer until the leaf node layer (the order in which the log to be identified is segmented is the order of each word, which is also the order of the search terms in the search sequence).

[0065] Furthermore, before performing a layer-by-layer search on the preset search tree, the method further includes: determining the category of each word; for each word, determining whether to use the word as the search term or the attribute of the word as the search term based on the word's category.

[0066] Furthermore, as a non-limiting embodiment, the category of the word includes English words pre-stored in a preset constant set, as well as other words; determining whether to use the word as the search term or the attribute of the word as the search term based on the category of the word includes: for each word, if the word is an English word pre-stored in the preset constant set, then the word is used as the search term; if the word is one of the other words, then the attribute of the word is used as the search term.

[0067] In practice, the pre-stored English words in the preset set of constants can be acquired through long-term accumulation. In the field of log recognition, constants have only one attribute: English words. <en>Therefore, when determining whether the current token is a constant or a variable, first check if the current token's attribute is... <en>If so, it is likely a constant; otherwise, it is often not a constant. The other words mentioned may include time. <time>,number <num>,date <date>URL <url>IP address <ip>And so on, but not limited to these.

[0068] In the specific implementation of step S13, during the layer-by-layer search of the search tree, if the leaf node layer can be reached through the existing internal nodes, it means that during the layer-by-layer search of the search tree, the search term quantity node layer has a search term quantity node with the same quantity value as the word, and each internal node layer has an internal node with the same value as each search term. In this case, the first similarity between each search term (or the search sequence formed by each search term) and the existing log pattern in the reached leaf node is calculated. If the maximum value of the first similarity is less than the first threshold, it means that the similarity between the search sequence formed by each search term and the existing log pattern in the reached leaf node is low. The search sequence can be regarded as a brand new log pattern and can be used as the initial identification result.

[0069] Alternatively, during the process of searching the search tree layer by layer, if no internal node is found (including the case where no search term node with the same number of words as the word is found at the search term number node layer), the search sequence can also be regarded as a brand new log pattern and can be used as the initial result.

[0070] The order of the search terms in the sequence formed by the search terms is the same as the order in which the log to be identified is segmented.

[0071] Furthermore, before selecting a preset number of existing log patterns from all existing log patterns, the method further includes:

[0072] The first similarity between the existing log patterns in the reached leaf nodes and each search term is determined using the following formula:

[0073]

[0074]

[0075] Where sim1 indicates the first similarity between each pattern word of the existing log pattern in the arrived leaf node and each search word, f() indicates the first similarity calculation function, N indicates the length value or number of pattern words of the existing log pattern in the arrived leaf node, and a i Used to indicate the i-th search term determined according to the word segmentation order among various search terms, b i This is used to indicate the i-th pattern word of each stored log pattern in the leaf node that has been reached.

[0076] It is understandable that the greater the first similarity, the more likely the existing log patterns in the leaf nodes are similar to or the same as the search sequences formed by the various search terms.

[0077] Furthermore, the first threshold is determined at least based on the number of digits and the number of wildcards in the search term; wherein, the larger the number of digits, the smaller the first threshold, and the larger the number of wildcards, the smaller the first threshold; wherein, each search term is determined based on a word, and each search term is the same as or different from its corresponding word.

[0078] Furthermore, determining the first threshold based on the number of digits and wildcards in the search term includes: determining a first intermediate coefficient based on the quotient between the number of digits in the search term and the total number of search terms, and determining a second intermediate coefficient based on the number of digits in the search term and the number of wildcards in the search term; and determining the first threshold based on the weighted calculation result of the first intermediate coefficient and the second intermediate coefficient using a preset weight ratio.

[0079] Furthermore, the first threshold can be determined using the following formula:

[0080] st=min{1,0.5×st1+0.5×st2};

[0081]

[0082] st2 = log max{2,digLen+1} (η+1);

[0083] Where st indicates the first threshold, seqLen indicates the total number of search terms, digLen indicates the number of digits in each search term, η indicates the number of wildcards in each search term, min{} indicates the minimum value function, max{} indicates the maximum value function, st1 is the first intermediate coefficient, and st2 is the second intermediate coefficient.

[0084] In specific implementation, the preset weight ratio is not only 1:1, that is, the weight values ​​of the first intermediate coefficient and the second intermediate coefficient are 0.5 and 0.5 respectively, but other weight ratios can also be reasonably set according to the needs of different application scenarios. This embodiment of the invention does not limit this.

[0085] In some non-limiting embodiments, the first threshold may be determined to have a suitable value between [0.4, 0.85], for example, it may be 0.5 or 0.6.

[0086] In this embodiment of the invention, compared to setting a fixed value as the similarity threshold in existing log pattern recognition, when the log length is fixed and it contains many known variables (numbers, IP addresses, etc.), the proportion of constants can be considered to be reduced. Therefore, a smaller threshold is more reasonable when calculating similarity. Conversely, when there are fewer known variables, a larger threshold is more reasonable. However, the prior art does not consider this situation, and using a fixed threshold may lead to insufficient accuracy. This embodiment of the invention adopts an adaptive threshold method, which dynamically adjusts the threshold based on the information of known variables (i.e., the number of search terms that are inconsistent with the word), which helps to improve the accuracy of the recognition results.

[0087] Furthermore, selecting a preset number of existing log patterns from all existing log patterns includes: calculating the distance between the vector of each existing log pattern and the vector of each search term; and determining the preset number of existing log patterns that are closest to each other.

[0088] Furthermore, it satisfies one or more of the following: the vector is selected from: hash vector, word frequency and inverse text frequency index TFIDF vector, word2vec vector; the distance is selected from: Euclidean distance, Mahalanobis distance, Manhattan distance, but is not limited to these.

[0089] For example, in practical implementation, the characteristic that the structure of logs is more important than semantics in log pattern recognition can be taken advantage of. First, the vectors of each existing log pattern and each search term (the search sequence formed by each search term) are hashed and vectorized. Then, the Euclidean distance between the hash vectors of each existing log pattern and the hash vectors of each search term is calculated. This can reduce the computation and storage costs without affecting the accuracy of the recognition results.

[0090] In this embodiment of the invention, compared to directly using the method of traversing text characters to calculate the similarity between each log pattern and determine the preset number of existing log patterns, which is computationally expensive and inefficient, this embodiment of the invention uses a method of first vectorizing each log pattern and then calculating the distance between vectors to determine the preset number of existing log patterns that are closest to each other. This helps to improve computational efficiency and thus improve log pattern recognition efficiency.

[0091] In the specific implementation of step S14, comparing the maximum value of each obtained second similarity with the second threshold to determine whether to use the longest common subsequence between the existing log pattern corresponding to the maximum value of the second similarity and each search term as the pattern of the log to be identified includes: if the second similarity between each pattern word of the existing log pattern corresponding to the maximum value of the second similarity and each search term is less than the second threshold, then it is determined that the longest common subsequence between the existing log pattern corresponding to the maximum value of the second similarity and each search term is used as the pattern of the log to be identified.

[0092] Specifically, after determining the existing log pattern corresponding to the maximum value of the second similarity, the longest common subsequence of the existing log pattern and each search term can be calculated, and this can be used as the pattern of the log to be identified.

[0093] In practice, existing conventional methods can be used to calculate the longest common subsequence between the stored log pattern corresponding to the maximum value of the second similarity and each search term.

[0094] In some non-limiting embodiments, the second threshold may be determined to have a suitable value between [0.35, 0.75], for example, it may be 0.4 or 0.5.

[0095] Furthermore, based on the following formula, the second similarity with each search term is calculated using the selected existing log pattern:

[0096]

[0097]

[0098] Where sim2 indicates the second similarity between each pattern word of the selected stored log pattern and each search word, C() indicates the iterative function for calculating the second similarity, max() indicates the maximum value calculation function, and x i Used to indicate the i-th search term determined according to the word segmentation order among various search terms, y j Used to indicate the j-th pattern word in each of the selected stored log patterns;

[0099] During the iteration process, the initial values ​​of i and j are l respectively. x and l y , l x Used to indicate the length value of each search term or the number of search terms, l y Used to indicate the length value or number of pattern words of the selected stored log pattern.

[0100] Furthermore, after determining the pattern of the log to be identified, the method further includes: storing the pattern of the log to be identified in a fused log pattern set.

[0101] In specific implementations, one or more storage areas may be set up to store the log patterns obtained at each processing stage. For example, it may include a set of fused log patterns that stores the final identification result of each log to be identified, a set that stores the initial identification results (sequences formed by each search term) determined at the search tree stage, and a set that stores other related intermediate results or records. This embodiment of the invention does not limit this.

[0102] In specific implementation, during the calculation of the second similarity, a preset number of stored log patterns can be selected from all stored log patterns, and the second similarity between the selected stored log patterns and each search term can be calculated respectively; alternatively, the final recognition results stored in the fused log pattern set can be directly used to select a preset number of log patterns to calculate the second similarity between the selected log patterns and each search term, thereby reducing the amount of computation and improving the computational efficiency.

[0103] Compared to existing technologies that rely on the assumption that different log lengths necessarily correspond to different patterns for log pattern recognition (for example, given two log entries, Receive from node 001 and Receive from node 003 804, which should theoretically correspond to the same pattern Receive from node<*>), if existing technologies assume that the initial identification result determined by searching the search tree is used directly as the final identification result for the log entries to be identified, then the final identification result will be: Receive from node 001 and Receive from node 003. The error 804 indicates insufficient accuracy in the identification results, particularly in failing to identify logs with different lengths but identical patterns. This invention divides log pattern recognition into two stages. In the first stage, based on existing assumptions, a pre-defined search tree is searched layer by layer to determine whether to use the search sequence formed by each search term as the initial identification result. In the second stage, based on the initial identification result, the second similarity and a second threshold of each search term with a pre-defined number of existing log patterns are compared, and the final identification result is determined by finding the maximum common subsequence. Since the second stage uses existing log patterns stored in leaf nodes of nodes with different numbers of search terms in the search tree (with different numbers of pattern words than search terms) during the calculation of the second similarity, it helps to merge log patterns with different lengths but high similarity into one pattern, improving identification accuracy while maintaining efficiency. Furthermore, compared to traditional log pattern recognition that typically uses clustering to load all logs to be identified into memory at once, resulting in high storage and computational overhead, this invention can process logs in a streaming manner, processing one log at a time, reducing overhead and improving efficiency.

[0104] Reference Figure 2 , Figure 2 This is a partial flowchart of the second log pattern recognition method in an embodiment of the present invention. The second log pattern recognition method may include... Figure 1 Steps S11 to S14 shown may further include step S21 and step S22. Step S21 or step S22 may be executed after step S12, and there may be no specific order of execution between steps S21 and S22. Step S21 or step S22 may be executed before or after step S14. The following describes the... Figure 1 The different contents in the text will be explained.

[0105] In step S21, when no internal node is found, the search term is used, and the previously found internal node is used as the parent node. Internal nodes are added layer by layer in the search tree until the leaf node layer is reached. The order of the search terms corresponds one-to-one with the internal node layers of the search tree.

[0106] In step S22, when the leaf node layer can be reached through the existing internal nodes, and the maximum value of the first similarity between each search term and the existing log pattern in the reached leaf node is less than the first threshold, the search sequence formed by each search term is stored as a new existing log pattern in the reached leaf node; or, when the internal node cannot be found, the search sequence formed by each search term is stored as a new existing log pattern in the new leaf node under the new internal node.

[0107] It should be noted that in specific implementation, when the number of search terms or internal nodes cannot be found, in addition to the above-mentioned method of adding nodes in real time (adding nodes layer by layer while searching), the number of words or search terms that could not be found and their corresponding search records can be saved first, and then nodes can be added periodically or irregularly.

[0108] Reference Figure 3 , Figure 3 This is a schematic diagram illustrating the specific structure of a search tree in an embodiment of the present invention. The search tree includes a root node, a layer of search term count nodes, a layer of internal nodes, and a layer of leaf nodes. Each leaf node stores one or more existing log patterns, such as... Figure 3 The examples shown are existing log patterns 1 to 6.

[0109] Among them, the existing log patterns 1 to 3 have the same length value, and the existing log patterns 4 to 6 have the same length value, but their length values ​​are different.

[0110] As a non-limiting embodiment, the detailed process of performing a layer-by-layer search on the search tree using the number of words and the search terms for each word can be described as follows:

[0111] (1) Using the number of words (corresponding to the length value of the log to be identified), search each node of the search term number node layer of the search tree. If there is no search term number node with the same number of words / length value of the log to be identified, add a new search term number node in the search term number node layer and assign it the length value. Use the newly added node as the parent node of the next search. If there is a search term number node with the same length value, use the search term number node directly as the parent node of the next search.

[0112] (2) Using the search term of the word, the internal node layer of the search tree is searched layer by layer. In each search, if there is no internal node with the same search term, an internal node is added in the searched internal node layer and the value of the search term is assigned as the parent node of the next search. If there is an internal node with the same search term, the internal node is directly used as the parent node of the next search until the leaf node is reached and the search stops.

[0113] (3) When the leaf node layer can be reached through the existing internal nodes, determine whether the maximum value of the first similarity between the search sequence formed by each search term and the existing log pattern in the reached leaf node is less than or equal to the first threshold. If yes, store the search sequence formed by each search term as a new existing log pattern in the reached leaf node. If no, merge the existing log pattern corresponding to the maximum value of the first similarity with each search term to obtain an updated log pattern. Then, the updated log pattern can be used to replace the existing log pattern corresponding to the maximum value of the first similarity in the reached leaf node. Alternatively, when the internal node cannot be found, store the search sequence formed by each search term as a new existing log pattern in the new leaf node under the new internal node.

[0114] In a specific implementation, after replacing the existing log pattern corresponding to the maximum value of the first similarity in the arrived leaf node with the updated log pattern, the process may further include: determining the fused log pattern corresponding to the replaced existing log pattern; updating the corresponding fused log pattern in the fused log pattern set; wherein, the fused log pattern set is used to store the log pattern after fusion processing.

[0115] In this embodiment of the invention, during the process of performing a layer-by-layer search on the preset search tree, a search term count node can be added to the search term count node layer of the search tree to store the number of words that could not be found; an internal node can be added to the internal node layer to store the search terms that could not be found; and a leaf node can be added to the leaf node layer to store the search sequence formed by the search terms that could not be found. Furthermore, the existing log patterns stored in the reached or newly added leaf nodes can be updated. Thus, when applied to the identification of a log set containing a large number of logs to be identified, the identification result can be determined efficiently and accurately, and the search tree can also be updated and improved.

[0116] For more detailed information regarding steps S21 to S22 in the specific implementation, please refer to the preceding text and... Figure 1 The steps described in the document will be executed, and will not be repeated here.

[0117] Reference Figure 4 , Figure 4 This is a partial flowchart of the third log pattern recognition method in this embodiment of the invention. The third log pattern recognition method may include... Figure 1 The steps S11 to S14 shown may also include Figure 2 Steps S21 to S22 shown may further include steps S41 to S44, wherein step S41 may be executed after step S12, and there may be no specific execution order between steps S41 and S13, steps S41 and S21, and steps S41 and S22. The following describes the relationship between... Figure 1 , Figure 2 The different contents in the text will be explained.

[0118] In step S41, when the leaf node layer can be reached through the existing internal nodes, and the maximum value of the first similarity between each search term and the existing log pattern in the reached leaf node is greater than or equal to the first threshold, the existing log pattern corresponding to the maximum value of the first similarity is fused with each search term to obtain the updated log pattern.

[0119] Furthermore, the process of fusing the existing log pattern corresponding to the maximum value of the first similarity with each search term to obtain the updated log pattern includes: determining whether each search term is the same as a pattern word in the same order in the existing log pattern corresponding to the maximum value of the first similarity, based on the word segmentation order in each search term; if they are the same, then the search term is used as the pattern word in the same order in the updated log pattern; if they are different, then a wildcard is used as the pattern word in the same order in the updated log pattern.

[0120] In step S42, the existing log pattern corresponding to the maximum value of the first similarity in the arrived leaf node is replaced with the updated log pattern.

[0121] In step S43, the fused log pattern corresponding to the existing log pattern being replaced is determined.

[0122] In step S44, the corresponding fusion log pattern is updated in the fusion log pattern set.

[0123] The fused log pattern set is the same as the fused log pattern set described in step S14. This fused log pattern set can be used to store the fused log patterns (i.e., the final identification result of the log to be identified determined in step S14). It is understood that there is a many-to-one mapping relationship between the stored log patterns in each leaf node of the search tree and the fused log patterns (the final identification results of each log to be identified) stored in the fused log pattern set.

[0124] Furthermore, updating the corresponding fusion log pattern in the fusion log pattern set includes: performing fusion processing on the corresponding fusion log pattern and the update log pattern; and replacing the corresponding fusion log pattern with the processed fusion log pattern.

[0125] In specific implementation, the process of fusing the corresponding fusion log pattern and update log pattern is described in step S41, which involves fusing the existing log pattern corresponding to the maximum value of the first similarity with each search term to obtain the relevant description of the update log pattern. This process will not be repeated here.

[0126] In specific implementation, please refer to the preceding text for more similar content regarding steps S41 to S44. Figures 1 to 3 The relevant descriptions in the document will be executed, and will not be repeated here.

[0127] Reference Figure 5 , Figure 5 This is a schematic diagram of a log pattern recognition device according to an embodiment of the present invention. The log pattern recognition device may include:

[0128] The word segmentation module 51 is used to segment the log to be recognized into one or more words.

[0129] The layer-by-layer search module 52 is used to perform a layer-by-layer search on a preset search tree using the number of words and the search terms for each word. The search tree includes a search term number node layer, one or more internal node layers, and a leaf node layer that stores one or more existing log patterns.

[0130] The similarity calculation module 53 is used to select a preset number of existing log patterns from all existing log patterns and calculate the second similarity between each search term and the existing log patterns in the reached leaf node when the leaf node layer can be reached through existing internal nodes and the maximum value of the first similarity between each search term and the existing log patterns in the reached leaf node is less than a first threshold, or when internal nodes cannot be found, and to select the existing log patterns from all existing log patterns and calculate the second similarity between each search term and the existing log patterns respectively.

[0131] The log pattern determination module 54 is used to compare the maximum value of each obtained second similarity with the second threshold to determine whether to use the longest common subsequence between the existing log pattern corresponding to the maximum value of the second similarity and each search term as the pattern of the log to be identified.

[0132] For details regarding the principle, implementation, and beneficial effects of this log pattern recognition device, please refer to the preceding text. Figures 1 to 4 The description of the log pattern recognition method shown is not repeated here.

[0133] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, performs the steps of the above-described log pattern recognition method. The computer-readable storage medium may include non-volatile or non-transitory memory, and may also include optical discs, hard disk drives, solid-state drives, etc.

[0134] Specifically, in this embodiment of the invention, the processor can be a central processing unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0135] It should also be understood that the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0136] This invention also provides a terminal, including a memory and a processor. The memory stores a computer program that can run on the processor. When the processor runs the computer program, it performs the steps of the log pattern recognition method described above. The terminal may include, but is not limited to, mobile phones, computers, tablets, and other terminal devices, and may also be servers, cloud platforms, etc.

[0137] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article indicates that the preceding and following related objects have an "or" relationship.

[0138] In the embodiments of this application, "multiple" refers to two or more.

[0139] The descriptions of "first," "second," etc., appearing in the embodiments of this application are for illustrative purposes and to distinguish the objects being described. They have no order and do not indicate any special limitation on the number of devices in the embodiments of this application, nor do they constitute any limitation on the embodiments of this application.

[0140] It should be noted that the sequence number of each step in this embodiment does not represent a limitation on the execution order of each step.

[0141] While the present invention has been disclosed above, it is not limited thereto. Any person skilled in the art can make various modifications and alterations without departing from the spirit and scope of the invention; therefore, the scope of protection of the present invention should be determined by the scope defined in the claims.< / ip> < / url> < / date> < / num> < / time> < / en> < / en> < / cn> < / en> < / email> < / path> < / file> < / ip> < / url> < / date> < / num> < / time>

Claims

1. A log pattern recognition method, characterized in that, include: The log to be identified is segmented to obtain one or more words; Using the number of words and the search terms for each word, a preset search tree is searched layer by layer. The search tree includes a search term number node layer, one or more internal node layers, and a leaf node layer that stores one or more existing log patterns. When the leaf node layer can be reached through the existing internal nodes, and the maximum value of the first similarity between each search term and the existing log pattern in the reached leaf node is less than the first threshold, or when no internal node is found, the distance between the vector of each existing log pattern and the vector of each search term is calculated respectively, the nearest preset number of existing log patterns are determined, the preset number of existing log patterns are selected from all existing log patterns, and the second similarity between the selected existing log patterns and each search term is calculated respectively. The maximum value of each obtained second similarity is compared with a second threshold to determine whether to use the longest common subsequence between the existing log pattern corresponding to the maximum value of the second similarity and each search term as the pattern of the log to be identified. If the second similarity between each pattern word of the existing log pattern corresponding to the maximum value of the second similarity and each search term is less than the second threshold, then it is determined that the longest common subsequence between the existing log pattern corresponding to the maximum value of the second similarity and each search term is used as the pattern of the log to be identified.

2. The method according to claim 1, characterized in that, The method further includes: When no internal node is found, the search term is used, and the previously found internal node is used as the parent node. Internal nodes are added layer by layer in the search tree until the leaf node layer is reached. The order of the search terms for the words corresponds one-to-one with the internal node layers of the search tree.

3. The method according to claim 1, characterized in that, The method further includes: When it is possible to reach the leaf node layer through the existing internal nodes, and the maximum value of the first similarity between each search term and the existing log pattern in the reached leaf node is less than the first threshold, the search sequence formed by each search term is stored as a new existing log pattern in the reached leaf node. or, When no internal node is found, the search sequence formed by each search term is stored as a new existing log pattern in the new leaf node under the new internal node.

4. The method according to claim 1, characterized in that, Before performing a layer-by-layer search on the preset search tree, the method further includes: Determine the category of each word; For each word, the search term is determined based on either the word's category or the word's attribute.

5. The method according to claim 4, characterized in that, The word category includes English words pre-stored in a preset constant set, as well as other words; Determining whether to use a word as a search term based on its category or its attribute includes: For each word, if the word is an English word pre-stored in a preset constant set, then the word is used as the search term; If the word is one of the other words, then the attribute of that word is used as the search term.

6. The method according to claim 1, characterized in that, Before selecting a preset number of existing log patterns from all existing log patterns, the method further includes: The first similarity between the existing log patterns in the reached leaf nodes and each search term is determined using the following formula: ; ; in, The first similarity between each pattern word and each search word in the existing log pattern in the reached leaf node is used to indicate the similarity between them. Used to indicate the first similarity calculation function. Used to indicate the length value or number of pattern words in the stored log pattern in the arrived leaf node. Used to indicate the first word in each search term, determined according to the word segmentation order. 1 search term, The first log pattern used to indicate the reached leaf node is one of the existing log patterns. A pattern word.

7. The method according to claim 1, characterized in that, Meet one or more of the following: The vectors are selected from: hash vectors, TFIDF vectors (term frequency and inverse text frequency index), and Word2vec vectors. The distances are selected from: Euclidean distance, Mahalanobis distance, and Manhattan distance.

8. The method according to claim 1, characterized in that, Based on the following formula, the second similarity with each search term is calculated using the selected existing log pattern: ; ; in, The second similarity used to indicate the relationship between each pattern word of the selected stored log pattern and each search word. Used to indicate the iterative function for calculating the second similarity. Used to indicate the function for calculating the maximum value. Used to indicate the first word in each search term, determined according to the word segmentation order. 1 search term, Used to indicate the first of the selected existing log modes One pattern word; During the iteration process The initial values ​​are respectively and , Used to indicate the length value of each search term or the number of search terms. Used to indicate the length value or number of pattern words of the selected stored log pattern.

9. The method according to claim 1, characterized in that, The method further includes: When the leaf node layer can be reached through the existing internal nodes, and the maximum value of the first similarity between each search term and the existing log pattern in the reached leaf node is greater than or equal to the first threshold, the existing log pattern corresponding to the maximum value of the first similarity is fused with each search term to obtain the updated log pattern. Replace the existing log pattern corresponding to the maximum value of the first similarity in the arrived leaf node with the updated log pattern; Determine the fusion log pattern corresponding to the existing log pattern being replaced; Update the corresponding fusion log pattern in the fusion log pattern set; The fused log pattern set is used to store the log patterns after fusion processing.

10. The method according to claim 9, characterized in that, Updating the corresponding fusion log pattern in the fusion log pattern set includes: The corresponding fusion log mode and update log mode are fused together. The processed fusion log pattern is used to replace the corresponding fusion log pattern.

11. The method according to claim 9, characterized in that, The existing log pattern corresponding to the maximum value of the first similarity is fused with each search term to obtain the updated log pattern, which includes: Based on the word segmentation order in each search term, determine whether each search term is the same as the pattern word in the same order in the existing log pattern corresponding to the maximum value of the first similarity. If they are the same, then the search term is used as the pattern term in the same order in the update log pattern; If they are different, a wildcard is used as the pattern word in the same order in the update log pattern.

12. The method according to claim 1, characterized in that, The first threshold is determined based at least on the number of digits and the number of wildcards in the search term; Wherein, the larger the number of digits, the smaller the first threshold; and the larger the number of wildcards, the smaller the first threshold. Each search term is determined based on a word, and each search term may be the same as or different from its corresponding word.

13. The method according to claim 12, characterized in that, The first threshold is determined based on the number of digits and wildcards in the search term as follows: A first intermediate coefficient is determined based on the quotient between the number of digits in the search term and the total number of search terms, and a second intermediate coefficient is determined based on the number of digits in the search term and the number of wildcards in the search term; The first threshold is determined by using a preset weight ratio and the weighted calculation result of the first intermediate coefficient and the second intermediate coefficient.

14. The method according to claim 13, characterized in that, The first threshold is determined using the following formula: ; ; ; in, Used to indicate the first threshold Used to indicate the total number of search terms. Used to indicate the number of numbers in each search term. Used to indicate the number of wildcards in each search term. Used to indicate the function for finding the minimum value Used to indicate the function for finding the maximum value. As the first intermediate coefficient, This is the second intermediate coefficient.

15. The method according to claim 1, characterized in that, The process of segmenting the log to be identified to obtain one or more words includes: The Named Entity Recognition (NER) method is used to segment the log to be recognized into one or more words.

16. A log pattern recognition device, characterized in that, include: The word segmentation module is used to segment the log to be recognized into one or more words. The layer-by-layer search module is used to perform a layer-by-layer search on a preset search tree using the number of words and the search terms for each word. The search tree includes a search term number node layer, one or more internal node layers, and a leaf node layer that stores one or more existing log patterns. The similarity calculation module is used to calculate the distance between the vector of each existing log pattern and the vector of each search term when the leaf node layer can be reached through the existing internal nodes and the maximum value of the first similarity between each search term and the existing log pattern in the reached leaf node is less than a first threshold, or when the internal nodes cannot be found, to determine the nearest preset number of existing log patterns, and to calculate the second similarity between the selected existing log patterns and the search terms respectively. The log pattern determination module is used to compare the maximum value of each obtained second similarity with a second threshold to determine whether to use the longest common subsequence between the existing log pattern corresponding to the maximum value of the second similarity and each search term as the pattern of the log to be identified. If the second similarity between each pattern word of the existing log pattern corresponding to the maximum value of the second similarity and each search term is less than the second threshold, then it is determined that the longest common subsequence between the existing log pattern corresponding to the maximum value of the second similarity and each search term is used as the pattern of the log to be identified.

17. A computer-readable storage medium having a computer program stored thereon, characterized in that, The computer program is executed by the processor to perform the steps of the log pattern recognition method according to any one of claims 1 to 15.

18. A terminal comprising a memory and a processor, wherein the memory stores a computer program capable of running on the processor, characterized in that, When the processor runs the computer program, it performs the steps of the log pattern recognition method according to any one of claims 1 to 15.

Images