Log template extraction method and device, computer device, and storage medium

Abstract

The application relates to a log template extraction method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: determining a target vocabulary of a to-be-processed log from each vocabulary according to the occurrence frequency of the vocabulary in the to-be-processed log; the occurrence frequency of the target vocabulary is greater than the occurrence frequency of the vocabulary other than the target vocabulary among the each vocabulary; querying a plurality of log template sets determined in advance according to the target vocabulary of the to-be-processed log to obtain a target log template set corresponding to the to-be-processed log; each log template set comprises a plurality of log templates with the same target vocabulary; the similarity between each log template in the target log template set and the to-be-processed log is obtained respectively; and the target log template set is updated based on the similarity to obtain an updated target log template set. The method can improve the extraction effect of the log template.

Description

Technical Field

[0001] This application relates to the field of software algorithm technology, and in particular to a log template extraction method, apparatus, computer equipment, storage medium and computer program product. Background Technology

[0002] System logs play a crucial role in system operation and maintenance, helping administrators and maintenance personnel debug system faults and analyze anomalies, as well as identify the causes of system failures. However, as systems grow in scale, manually retrieving and analyzing logs becomes increasingly impractical, necessitating automated log analysis using machines or with machine assistance. Log template extraction is the first step in automated log analysis, and the quality of the extracted templates directly impacts the subsequent analysis results.

[0003] In existing technologies, log similarity is often calculated by grouping logs based on their segmented length and by using strict similarity or long common sequence. However, existing technologies for log template extraction suffer from poor extraction results. Summary of the Invention

[0004] Therefore, it is necessary to provide a log template extraction method, apparatus, computer equipment, computer-readable storage medium, and computer program product to address the aforementioned technical problem of poor log template extraction performance.

[0005] Firstly, this application provides a method for extracting log templates. The method includes:

[0006] Based on the frequency of occurrence of each word in the log to be processed, the target word of the log to be processed is determined from the words; the frequency of occurrence of the target word is greater than the frequency of occurrence of the words in the log other than the target word.

[0007] Based on the target vocabulary of the log to be processed, multiple pre-determined log template sets are queried to obtain a target log template set corresponding to the log to be processed; the target vocabulary of each log template in each log template set is the same.

[0008] The similarity between each log template in the target log template set and the log to be processed is obtained respectively;

[0009] Based on the similarity, the target log template set is updated to obtain the updated target log template set.

[0010] In one embodiment, before determining the target words of the log to be processed from the words based on their frequency of occurrence, the method further includes:

[0011] The log to be processed is preprocessed to obtain the preprocessed log;

[0012] The preprocessed log is segmented according to a preset delimiter to obtain multiple words contained in the log to be processed.

[0013] In one embodiment, the log to be processed includes a variable portion and a constant portion; the preprocessing of the log to be processed to obtain a preprocessed log includes:

[0014] Determine the log information of the variable portion of the log to be processed;

[0015] Delete the log information in the variable section, or replace the log information in the variable section with preset words to obtain the preprocessed log.

[0016] In one embodiment, obtaining the similarity between each log template in the target log template set and the log to be processed includes:

[0017] For any log template in the target log template set, a similarity matrix is ​​constructed between the log template and the log to be processed; each element in the similarity matrix corresponds to a vocabulary group, which includes a word from the log template and a word from the log to be processed;

[0018] The element value of each element in the similarity matrix is ​​determined based on the correlation between two words in the vocabulary group corresponding to each element in the similarity matrix.

[0019] Based on the element value of each element, the normalized path distance of the similarity matrix is ​​determined;

[0020] Based on the regularized path distance, the similarity between any log template and the log to be processed is determined.

[0021] In one embodiment, determining the element value of each element in the similarity matrix based on the relevance between two words in the vocabulary group corresponding to each element in the similarity matrix includes:

[0022] For any element in the similarity matrix, if the two words corresponding to the element are the same, then the element value is determined to be 0.

[0023] If the two words corresponding to the element are not the same, then the element value is determined to be 1.

[0024] In one embodiment, determining the similarity between any log template and the log to be processed based on the normalized path distance includes:

[0025] Obtain the first number of each word contained in any log template, and obtain the second number of each word contained in the log to be processed;

[0026] From the first number and the second number, determine the target number with the larger value;

[0027] Based on the number of targets and the regularized path distance, the similarity between any log template and the log to be processed is determined.

[0028] In one embodiment, updating the target log template set based on the similarity to obtain an updated target log template set includes:

[0029] If there is a target log template in the target log template set that has a similarity to the log to be processed that is greater than a preset threshold, then the log to be processed and the target log template are merged to obtain an updated target log template set.

[0030] If the similarity between each log template in the target log template set and the log to be processed is less than or equal to the preset threshold, then the log to be processed is used as a new log template, and the new log template is added to the target log template set to obtain an updated target log template set.

[0031] Secondly, this application also provides a log template extraction device. The device includes:

[0032] The vocabulary determination module is used to determine the target vocabulary of the log to be processed from the words based on the frequency of occurrence of each word contained in the log to be processed; the frequency of occurrence of the target vocabulary is greater than the frequency of occurrence of the words other than the target vocabulary in the words.

[0033] The template query module is used to query a pre-determined set of multiple log templates based on the target vocabulary of the log to be processed, and obtain a set of target log templates corresponding to the log to be processed; the target vocabulary of each log template in each set is the same.

[0034] The similarity calculation module is used to obtain the similarity between each log template in the target log template set and the log to be processed.

[0035] The template update module is used to update the target log template set based on the similarity to obtain the updated target log template set.

[0036] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0037] Based on the frequency of occurrence of each word in the log to be processed, the target word of the log to be processed is determined from the words; the frequency of occurrence of the target word is greater than the frequency of occurrence of the words in the log other than the target word.

[0038] Based on the target vocabulary of the log to be processed, multiple pre-determined log template sets are queried to obtain a target log template set corresponding to the log to be processed; the target vocabulary of each log template in each log template set is the same.

[0039] Obtain the similarity between each log template in the target log template set and the log to be processed; based on the similarity, update the target log template set to obtain the updated target log template set.

[0040] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0041] Based on the frequency of occurrence of each word in the log to be processed, the target word of the log to be processed is determined from the words; the frequency of occurrence of the target word is greater than the frequency of occurrence of the words in the log other than the target word.

[0042] Based on the target vocabulary of the log to be processed, multiple pre-determined log template sets are queried to obtain a target log template set corresponding to the log to be processed; the target vocabulary of each log template in each log template set is the same.

[0043] The similarity between each log template in the target log template set and the log to be processed is obtained respectively;

[0044] Based on the similarity, the target log template set is updated to obtain the updated target log template set.

[0045] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0046] Based on the frequency of occurrence of each word in the log to be processed, the target word of the log to be processed is determined from the words; the frequency of occurrence of the target word is greater than the frequency of occurrence of the words in the log other than the target word.

[0047] Based on the target vocabulary of the log to be processed, multiple pre-determined log template sets are queried to obtain a target log template set corresponding to the log to be processed; the target vocabulary of each log template in each log template set is the same.

[0048] The similarity between each log template in the target log template set and the log to be processed is obtained respectively;

[0049] Based on the similarity, the target log template set is updated to obtain the updated target log template set.

[0050] The aforementioned log template extraction method, apparatus, computer equipment, storage medium, and computer program product extract words with higher word frequencies from the logs to be processed as target words by statistically analyzing word frequencies. Based on the target words, a pre-determined set of log templates is queried, and log templates with the same target words are used as the target log template set. The similarity between the logs to be processed and the target log templates is calculated separately, and the target log template set is updated based on the calculated similarity. Utilizing high-frequency words to query log templates improves the initial matching efficiency, and updating the target log template set based on the similarity between the logs to be processed and the target log templates improves the log template extraction effect. Attached Figure Description

[0051] Figure 1 This is a diagram illustrating the application environment of a log template extraction method in one embodiment;

[0052] Figure 2 This is a flowchart illustrating a log template extraction method in one embodiment;

[0053] Figure 3 This is a flowchart illustrating the steps for calculating similarity in one embodiment;

[0054] Figure 4 This is a schematic diagram illustrating the calculation of similarity based on a similarity matrix in one embodiment;

[0055] Figure 5 This is a flowchart illustrating the online log template extraction method in another embodiment;

[0056] Figure 6 This is a structural block diagram of a log template extraction device in one embodiment;

[0057] Figure 7 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0059] The log template extraction method provided in this application embodiment can be applied to, for example, Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. Server 104 collects the logs to be processed generated by terminal 102 and determines the target words of the logs from various words. Based on the target words of the logs to be processed, server 104 queries a pre-determined set of multiple log templates to obtain a set of target log templates corresponding to the logs to be processed. Then, server 104 obtains the similarity between each log template in the target log template set and the logs to be processed. Based on the similarity, server 104 updates the target log template set to obtain an updated set of target log templates. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated on server 104 or placed on a cloud or other network server. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. Server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0060] In one embodiment, such as Figure 2 As shown, a log template extraction method is provided, which can be applied to... Figure 1 Taking server 104 as an example, the following steps are included:

[0061] Step S202: Based on the frequency of occurrence of each word in the log to be processed, determine the target word from each word; the frequency of occurrence of the target word is greater than the frequency of occurrence of other words in each word.

[0062] In this context, logs refer to system logs, which record information about hardware, software, and system problems, and also monitor events occurring within the system. Users can use them to investigate the causes of problems or search for traces left by attackers during attacks. System logs include system logs, application logs, and security logs.

[0063] Frequency refers to the number of times the same word appears in the log.

[0064] Optionally, the server identifies the target word from multiple words in the log to be processed based on the frequency of each word in the log to be processed, wherein the frequency of the target word is greater than the frequency of the remaining words in the log to be processed.

[0065] Step S204: Based on the target vocabulary of the log to be processed, query a set of pre-determined log templates to obtain a set of target log templates corresponding to the log to be processed; each set of log templates includes log templates with the same target vocabulary.

[0066] Log templates refer to logs that are typically composed of constants and variables, also known as template words and parameter words. Log templates are constructed by extracting the template words from them.

[0067] Optionally, the server obtains the target words corresponding to each log template set; matches the target words of the log to be processed with the target words corresponding to each log template set to obtain the matching results; and determines the log template set whose target words are the same as the target words of the log to be processed as the target log template set corresponding to the log to be processed.

[0068] Step S206: Obtain the similarity between each log template in the target log template set and the log to be processed.

[0069] The similarity degree refers to the text similarity, which is the degree of similarity between two texts. It can be calculated using supervised and unsupervised methods. For example, supervised methods use supervised models such as Naive Bayes classifiers to make judgments, while unsupervised methods use methods such as Euclidean distance to directly calculate the distance or similarity between texts.

[0070] Optionally, the server calculates the similarity between each log template in the target log template set and the log to be processed.

[0071] Step S208: Update the target log template set based on similarity to obtain the updated target log template set.

[0072] Among them, updating refers to adjusting the target log template set based on the logs to be processed, such as merging or adding new log templates.

[0073] Optionally, the server updates and adjusts the target log template set based on the calculated similarity to obtain an updated log template set.

[0074] In the above log template extraction method, words with higher frequency are extracted as target words by statistically analyzing the word frequency in the logs to be processed. A pre-determined set of log templates is queried based on the target words, and log templates with the same target words are used as the target log template set. The similarity between the logs to be processed and the target log templates is calculated separately, and the target log template set is updated based on the calculated similarity. Querying log templates with high-frequency words improves the initial matching efficiency, and updating the target log template set based on the similarity between the logs to be processed and the target log templates improves the log template extraction effect.

[0075] In one embodiment, before step S202 determines the target words of the log to be processed from the words based on their frequency of occurrence, the method further includes:

[0076] Step S202a: Preprocess the log to be processed to obtain the preprocessed log;

[0077] Preprocessing refers to data preprocessing, which mainly includes data cleaning, data integration, data transformation, and data reduction.

[0078] Optionally, the server performs data cleaning, data integration, data transformation, and data reduction on the words in the logs to be processed.

[0079] Step S202b: Perform word segmentation on the preprocessed log according to the preset delimiter to obtain multiple words contained in the log to be processed.

[0080] The preset delimiter can be used to segment the preprocessed logs according to special symbols, such as commas, periods, colons, and spaces.

[0081] Among them, word segmentation refers to enabling computer systems to automatically add spaces or other boundary markers between words in Chinese text.

[0082] Optionally, preset special symbols can be added between words in the processed log to obtain multiple words contained in the log to be processed after word segmentation.

[0083] In this embodiment, the log to be processed is preprocessed and then segmented to obtain multiple words included in the log to be processed, thereby achieving the effect of cutting off the coupling between the upper and lower parts and reducing the influence of word order.

[0084] In one embodiment, the log to be processed includes a variable portion and a constant portion; step S202a preprocesses the log to be processed to obtain a preprocessed log, including:

[0085] Identify the log information in the variable section of the log to be processed; delete the log information in the variable section, or replace the log information in the variable section with preset vocabulary, to obtain the preprocessed log.

[0086] Preprocessing can involve finding variable words in the logs to be processed and performing replacement operations, such as removing log timestamps and replacing information such as Internet Protocol addresses, email addresses, and hash values ​​with special symbols.

[0087] In particular, the frequency of the variable part in the log to be processed is lower than the frequency of the constant part.

[0088] Optionally, the server determines the log information in the variable section of the log to be processed, deletes some of the log information in the variable section, or replaces some of the log information in the variable section with the same special symbol, to obtain the preprocessed log. For example, the log time in the variable section can be deleted, the IP address in the variable section can be replaced with "ip", and other variables can be replaced with "*".

[0089] In this embodiment, by deleting and replacing variables that appear less frequently in the log to be processed, the variables are not used as the basis for subsequent similarity calculations, thus improving the accuracy of the calculations.

[0090] In one embodiment, such as Figure 3 As shown, step S206 obtains the similarity between each log template in the target log template set and the log to be processed, including:

[0091] Step S302: For any log template in the target log template set, construct a similarity matrix between the log template and the log to be processed; each element in the similarity matrix corresponds to a vocabulary group, which includes a word from the log template and a word from the log to be processed.

[0092] The similarity matrix is ​​a matrix used to calculate similarity. The matrix is ​​a set of complex or real numbers arranged in a long-room array.

[0093] In this context, a vocabulary group refers to a vocabulary for the log to be processed and a vocabulary for the log template corresponding to each row and column of each element in the matrix.

[0094] Optionally, for any log template in the log template set, the server constructs a similarity matrix between the log template and the log to be processed, using the words contained in the log template and the log to be processed as rows and columns, respectively. Each element in the similarity matrix corresponds to a word group, which includes a word from any log template and a word from the log to be processed.

[0095] Step S304: Determine the element value of each element in the similarity matrix based on the correlation between two words in the vocabulary group corresponding to each element in the similarity matrix.

[0096] Relevance can be defined as whether two words in a word group are the same or different.

[0097] Optionally, the server determines the element value of each element in the similarity matrix based on whether the two words in the vocabulary group corresponding to each element in the similarity matrix are the same or different.

[0098] Step S306: Determine the normalized path distance of the similarity matrix based on the element value of each element.

[0099] The regularized path distance refers to the shortest distance from the top left corner to the bottom right corner of the similarity matrix calculated using the dynamic regularization algorithm.

[0100] Optionally, the server uses a dynamic normalization algorithm to calculate the shortest distance from the top left corner to the bottom right corner of the similarity matrix based on the element value of each element in the similarity matrix.

[0101] Step S308: Based on the normalized path distance, determine the similarity between any log template and the log to be processed.

[0102] The similarity can be calculated using a preset formula.

[0103] Optionally, the server substitutes the normalized path distance into a preset similarity calculation formula to calculate the similarity between any log template and the log to be processed.

[0104] In this embodiment, a similarity matrix is ​​constructed based on the target log template and the log to be processed, and the normalized path distance of the similarity matrix is ​​calculated. The similarity is calculated based on the normalized path distance, which enables more accurate similarity calculations to be obtained for two logs of different lengths, thus improving the accuracy of the calculation.

[0105] In one embodiment, step S304 determines the element value of each element in the similarity matrix based on the relevance between two words in the vocabulary group corresponding to each element in the similarity matrix, including:

[0106] For any element in the similarity matrix, if the two words corresponding to the element are the same, the element value is determined to be 0; if the two words corresponding to the element are not the same, the element value is determined to be 1.

[0107] Here, an element refers to a matrix of real or complex numbers.

[0108] Optionally, for any element in the similarity matrix, the server assigns a value of 0 if the two words in the corresponding vocabulary group are the same, and assigns a value of 1 if the two words in the corresponding vocabulary group are different.

[0109] In this embodiment, by assigning values ​​to the elements in the corresponding similarity matrix based on the correlation between the vocabulary of any log template and the vocabulary of the log to be processed, a data basis is provided for calculating the regularized path distance.

[0110] In one embodiment, step S308, based on the normalized path distance, determines the similarity between any log template and the log to be processed, including:

[0111] Obtain the first number of each word contained in any log template, and obtain the second number of each word contained in the log to be processed; determine the target number with the larger value from the first number and the second number; determine the similarity between any log template and the log to be processed based on the target number and the normalized path distance.

[0112] Wherein, the first number and the second number are the word count length of the log to be processed and the word count length of any log template in the target log template set, respectively.

[0113] Optionally, the server obtains a first count of each word contained in any log template and a second count of each word contained in the log to be processed. From the second count, a target number with a larger value is determined. Based on the target number and the normalized path distance, the similarity between any log template and the log to be processed is calculated using a formula. Figure 4 The diagram illustrates the calculation of similarity based on a similarity matrix. The log entries representing rows in the similarity matrix contain five target words: Connect, to, service, port, and failed (a first count of 5). The log entries representing columns in the similarity matrix contain four target words: Connect, to, service, and failed (a second count of 4). In this diagram, the normalized path distance is calculated based on the element values ​​in the similarity matrix. The minimum distance from the top left corner to the bottom right corner of the similarity matrix, i.e., the path with the smallest sum of element values, is calculated as 1. The similarity is then calculated using the following formula:

[0114] (max(len1,len2)-d) / max(len1,len2)

[0115] Where d is the regular path distance, len1 and len2 are the first and second numbers respectively, and according to the above analysis, the value of max(len1,len2) is 5, and the similarity calculation result is 4 / 5.

[0116] In this embodiment, the similarity calculation is performed by using a dynamic time warping algorithm, which reduces the error in similarity calculation caused by different lengths.

[0117] In one embodiment, step S208 updates the target log template set based on similarity to obtain an updated target log template set, including:

[0118] If there is a target log template in the target log template set whose similarity to the log to be processed is greater than a preset threshold, then the log to be processed and the target log template are merged to obtain an updated target log template set; if the similarity between each log template in the target log template set and the log to be processed is less than or equal to the preset threshold, then the log to be processed is used as a new log template, and the new log template is added to the target log template set to obtain an updated target log template set.

[0119] Among them, the logs to be processed can be real-time logs in the system, and the target log template set is obtained by extracting high-frequency words from offline logs after preprocessing and word segmentation.

[0120] Optionally, if there is a target log template in the target log template set whose similarity to the log to be processed is greater than a preset threshold, the server will merge the log to be processed and the corresponding target log template to obtain an updated target log template; if the similarity between each log template in the target log template set and the log to be processed is less than the threshold, the server will use the log to be processed as a new log template and add the new log template to the target log template set to obtain an updated target log template set.

[0121] In this embodiment, by comparing the calculated similarity with a preset threshold and updating the target log template set based on the comparison result, the log template set is dynamically updated, thereby improving the accuracy of log template extraction.

[0122] In another embodiment, such as Figure 5 As shown, an online log template extraction method is provided, and the specific steps include:

[0123] Step S502: Determine the log information of the variable part of the log to be processed; delete the log information of the variable part, or replace the log information of the variable part with preset words to obtain the preprocessed log; perform word segmentation on the preprocessed log according to the preset delimiter to obtain multiple words contained in the log to be processed.

[0124] Specifically, the server determines the log information in the variable section of the log to be processed, deletes the log information in the variable section, or replaces the log information in the variable section with the same special symbols, thus obtaining the preprocessed log. Preset special symbols are then added between the words in the processed log to obtain the multiple words contained in the log to be processed after word segmentation.

[0125] Step S504: Based on the frequency of occurrence of each word in the log to be processed, determine the target word from each word; the frequency of occurrence of the target word is greater than the frequency of occurrence of other words in each word.

[0126] Specifically, the server identifies the target word from multiple words in the log to be processed based on the frequency of each word in the log to be processed. The frequency of the target word is greater than the frequency of the remaining words in the log to be processed.

[0127] Step S506: Based on the target vocabulary of the log to be processed, query a set of pre-determined log templates to obtain a set of target log templates corresponding to the log to be processed; each log template set includes log templates with the same target vocabulary.

[0128] Specifically, the server obtains the target words corresponding to each log template set; matches the target words of the log to be processed with the target words corresponding to each log template set to obtain the matching results; and determines the log template set whose target words are the same as the target words of the log to be processed as the target log template set corresponding to the log to be processed.

[0129] Step S508: For any log template in the target log template set, construct a similarity matrix between the log template and the log to be processed; each element in the similarity matrix corresponds to a vocabulary group, which includes a word from the log template and a word from the log to be processed.

[0130] Specifically, for any log template in the log template set, the server constructs a similarity matrix between the log template and the log to be processed, using the words contained in the log template and the log to be processed as rows and columns, respectively. Each element in the similarity matrix corresponds to a word group, which includes a word from any log template and a word from the log to be processed.

[0131] Step S510: For any element in the similarity matrix, if the two words corresponding to the element are the same, then the element value is determined to be 0; if the two words corresponding to the element are not the same, then the element value is determined to be 1.

[0132] Specifically, for any element in the similarity matrix, the server assigns a value of 0 if the two words in the corresponding word group are the same, and assigns a value of 1 if the two words in the corresponding word group are different.

[0133] Step S512: Obtain the first number of each word contained in any log template and the second number of each word contained in the log to be processed; determine the target number with the larger value from the first number and the second number; determine the similarity between any log template and the log to be processed based on the target number and the normalized path distance.

[0134] Specifically, the server obtains the first number of each word contained in any log template and the second number of each word contained in the log to be processed. From the second number, it determines the target number with the larger value. Based on the target number and the normalized path distance, it substitutes them into the calculation formula to calculate the similarity between any log template and the log to be processed.

[0135] Step S514: If there is a target log template in the target log template set whose similarity to the log to be processed is greater than a preset threshold, then the log to be processed and the target log template are merged to obtain an updated target log template set; if the similarity between each log template in the target log template set and the log to be processed is less than or equal to the preset threshold, then the log to be processed is used as a new log template, and the new log template is added to the target log template set to obtain an updated target log template set.

[0136] Specifically, if there is a target log template in the target log template set whose similarity to the log to be processed is greater than a preset threshold, the server will merge the log to be processed and the corresponding target log template to obtain an updated target log template; if the similarity between each log template in the target log template set and the log to be processed is less than the threshold, the server will use the log to be processed as a new log template and add the new log template to the target log template set to obtain an updated target log template set.

[0137] In this embodiment, by statistically analyzing the word frequencies in the logs to be processed, words with higher word frequencies are extracted as target words. Based on the target words, a pre-determined set of log templates is queried, and log templates with the same target words are used as the target log template set. The similarity between the logs to be processed and the target log templates is calculated, and the target log template set is updated based on the calculated similarity. Querying log templates using high-frequency words improves the initial matching efficiency, and updating the target log template set based on the similarity between the logs to be processed and the target log templates improves the log template extraction effect.

[0138] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0139] Based on the same inventive concept, this application also provides a log template extraction device for implementing the log template extraction method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations in one or more log template extraction device embodiments provided below can be found in the limitations of the log template extraction method described above, and will not be repeated here.

[0140] In one embodiment, such as Figure 6 As shown, a log template extraction device 600 is provided, including: a vocabulary determination module 602, a template query module 604, a similarity calculation module 606, and a template update module 608, wherein:

[0141] The vocabulary determination module 602 is used to determine the target vocabulary of the log to be processed from each vocabulary based on the frequency of occurrence of each vocabulary contained in the log to be processed; the frequency of occurrence of the target vocabulary is greater than the frequency of occurrence of each vocabulary other than the target vocabulary.

[0142] The template query module 604 is used to query a set of pre-determined log templates based on the target words of the log to be processed, and obtain a set of target log templates corresponding to the log to be processed; the target words of each log template in each set are the same.

[0143] The similarity calculation module 606 is used to obtain the similarity between each log template in the target log template set and the log to be processed.

[0144] The template update module 608 is used to update the target log template set based on similarity to obtain the updated target log template set.

[0145] Furthermore, in one embodiment, the vocabulary determination module 602 is also used to preprocess the log to be processed to obtain the preprocessed log; and to perform word segmentation on the preprocessed log according to a preset delimiter to obtain multiple words contained in the log to be processed.

[0146] Furthermore, in one embodiment, the log to be processed includes a variable portion and a constant portion. The vocabulary determination module 602 is also used to determine the log information of the variable portion of the log to be processed; delete the log information of the variable portion, or replace the log information of the variable portion with preset vocabulary to obtain the preprocessed log.

[0147] Furthermore, in one embodiment, the similarity calculation module 606 is also used to construct a similarity matrix between any log template in the target log template set and the log to be processed; each element in the similarity matrix corresponds to a vocabulary group, which includes a word in the log template and a word in the log to be processed; the element value of each element in the similarity matrix is ​​determined based on the correlation between the two words in the vocabulary group corresponding to each element in the similarity matrix; the normalized path distance of the similarity matrix is ​​determined based on the element value of each element; and the similarity between any log template and the log to be processed is determined based on the normalized path distance.

[0148] Furthermore, in one embodiment, the similarity calculation module 606 is also used to determine the element value of any element in the similarity matrix as 0 if the two words corresponding to the element are the same, and to determine the element value of the element as 1 if the two words corresponding to the element are not the same.

[0149] Furthermore, in one embodiment, the similarity calculation module 606 is also used to obtain a first number of each word contained in any log template and a second number of each word contained in the log to be processed; determine the target number with the larger value from the first number and the second number; and determine the similarity between any log template and the log to be processed based on the target number and the normalized path distance.

[0150] Furthermore, in one embodiment, the template update module 608 is further configured to: if there is a target log template in the target log template set whose similarity to the log to be processed is greater than a preset threshold, then merge the log to be processed and the target log template to obtain an updated target log template set; if the similarity between each log template in the target log template set and the log to be processed is less than or equal to the preset threshold, then use the log to be processed as a new log template and add the new log template to the target log template set to obtain an updated target log template set.

[0151] Each module in the aforementioned log template extraction device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0152] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 7 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs in the non-volatile storage media to run. The database stores logs to be processed, a set of log templates, a constructed similarity matrix, and data involved in similarity calculations. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When executed by the processor, the computer program implements a log template extraction method.

[0153] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0154] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0155] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0156] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0157] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0158] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0159] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0160] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for extracting log templates, characterized in that, The method includes: Based on the frequency of occurrence of each word in the log to be processed, the target word of the log to be processed is determined from the words; the frequency of occurrence of the target word is greater than the frequency of occurrence of the words in the log other than the target word. Based on the target vocabulary of the log to be processed, multiple pre-determined log template sets are queried to obtain a target log template set corresponding to the log to be processed; the target vocabulary of each log template in each log template set is the same. For any log template in the target log template set, a similarity matrix is ​​constructed between the log template and the log to be processed. The log template contains a first number of words, and the log to be processed contains a second number of words. Each element in the similarity matrix corresponds to a word group, which includes one word from the log template and one word from the log to be processed. The element value of each element in the similarity matrix is ​​determined based on the correlation between the two words in the word group corresponding to each element. The correlation includes whether the two words in the word group are the same or different. The normalized path distance of the similarity matrix is ​​determined based on the element value of each element. A target number with a larger value is determined from the first number and the second number. The ratio of the difference between the target number and the normalized path distance to the target number is determined as the similarity between the log template and the log to be processed. Based on the similarity and preset threshold, the target log template set is merged or added to obtain an updated target log template set.

2. The method according to claim 1, characterized in that, Before determining the target words of the log to be processed from the words based on their frequency of occurrence, the method further includes: The log to be processed is preprocessed to obtain the preprocessed log; The preprocessed log is segmented according to a preset delimiter to obtain multiple words contained in the log to be processed.

3. The method according to claim 2, characterized in that, The log to be processed includes a variable portion and a constant portion; the preprocessing of the log to be processed to obtain the preprocessed log includes: Determine the log information of the variable portion of the log to be processed; Delete the log information in the variable section, or replace the log information in the variable section with preset words to obtain the preprocessed log.

4. The method according to claim 1, characterized in that, The step of determining the element value of each element in the similarity matrix based on the correlation between two words in the vocabulary group corresponding to each element in the similarity matrix includes: For any element in the similarity matrix, if the two words corresponding to the element are the same, then the element value is determined to be 0. If the two words corresponding to the element are not the same, then the element value is determined to be 1.

5. The method according to claim 1, characterized in that, The step of updating the target log template set based on the similarity and a preset threshold to obtain an updated target log template set includes: If there is a target log template in the target log template set whose similarity to the log to be processed is greater than the preset threshold, then the log to be processed and the target log template are merged to obtain an updated target log template set. If the similarity between each log template in the target log template set and the log to be processed is less than or equal to the preset threshold, then the log to be processed is used as a new log template, and the new log template is added to the target log template set to obtain an updated target log template set.

6. A log template extraction device, characterized in that, The device includes: The vocabulary determination module is used to determine the target vocabulary of the log to be processed from the words based on the frequency of occurrence of each word contained in the log to be processed; the frequency of occurrence of the target vocabulary is greater than the frequency of occurrence of the words other than the target vocabulary in the words. The template query module is used to query a pre-determined set of multiple log templates based on the target vocabulary of the log to be processed, and obtain a set of target log templates corresponding to the log to be processed; the target vocabulary of each log template in each set is the same. A similarity calculation module is used to construct a similarity matrix between any log template in the target log template set and the log to be processed; the any log template contains a first number of words, and the log to be processed contains a second number of words; each element in the similarity matrix corresponds to a word group, and the word group includes one word from the any log template and one word from the log to be processed; the module determines the element value of each element in the similarity matrix based on the correlation between the two words in the word group corresponding to each element in the similarity matrix; the correlation includes whether the two words in the word group are the same or different; the module determines the normalized path distance of the similarity matrix based on the element value of each element; the module determines the target number with the larger value from the first number and the second number; and the ratio of the difference between the target number and the normalized path distance to the target number is determined as the similarity between the any log template and the log to be processed. The template update module is used to merge or add new target log templates based on the similarity and a preset threshold to obtain an updated target log template set.

7. The apparatus according to claim 6, characterized in that, The vocabulary determination module is also used to preprocess the log to be processed to obtain preprocessed log; and to perform word segmentation on the preprocessed log according to a preset delimiter to obtain multiple words contained in the log to be processed.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images