Log processing method and apparatus

Abstract

The application provides a log processing method and device, the method comprising: obtaining logs to be processed; for each second log corresponding to a first log, comparing a first field in the first log and a second field in the second log to determine the similarity and / or matching degree between the first log and the second log; wherein the first log is any one of the logs to be processed, and the second log is any one of the logs to be processed except the first log; if the similarity between the first log and the second log is greater than a preset similarity, or the matching degree between the first log and the second log is greater than a preset matching degree, it is determined that the first log and the second log belong to the same mode log, accurate judgment of the same mode log is realized, and the accuracy of the judgment of the same mode log is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a log processing method and device. Background Technology

[0002] Logs record detailed information during system operation and are an important basis for maintenance personnel to troubleshoot problems. However, with the rapid development of technology, systems are becoming more and more complex, and the amount of logs is also increasing dramatically. Therefore, in order to improve the efficiency of maintenance personnel in browsing logs, it is often necessary to filter out logs that belong to the same pattern.

[0003] In existing technology, when filtering logs of the same pattern, if two logs have different lengths, they are determined not to belong to the same pattern. If two logs have the same length, the process continues to determine whether they belong to the same pattern.

[0004] However, due to various reasons (such as non-standard logging), other words are often added to the logs, resulting in different lengths of logs belonging to the same pattern. Therefore, when judging logs based on their length first, logs with different lengths but actually belonging to the same pattern may be misjudged as not belonging to the same pattern, resulting in a low accuracy rate in judging logs of the same pattern. Summary of the Invention

[0005] This invention provides a log processing method and device to solve the technical problem of low accuracy in judging logs of the same pattern in the prior art.

[0006] In a first aspect, embodiments of the present invention provide a log processing method, including:

[0007] Get the logs to be processed;

[0008] For each second log corresponding to the first log, the first field in the first log and the second field in the second log are compared to determine the similarity and / or matching degree between the first log and the second log; wherein, the first log is any one of the logs to be processed, and the second log is any log to be processed other than the first log.

[0009] If the similarity between the first log and the second log is greater than a preset similarity, or the matching degree between the first log and the second log is greater than a preset matching degree, then it is determined that the first log and the second log belong to the same pattern log.

[0010] In one possible design, comparing the first field in the first log and the second field in the second log to determine the degree of matching between the first log and the second log includes:

[0011] Based on the preset matching score rules, the first field in the first log is matched one by one with the second field in the second log to determine the matching score corresponding to each first field in the first log;

[0012] The matching degree between the first log and the second log is determined based on the matching score corresponding to each of the first fields.

[0013] In one possible design, before matching the first field in the first log with the second field in the second log one by one based on a preset matching score rule, the following steps are also included:

[0014] Align the first log and the second log.

[0015] In one possible design, determining the matching degree between the first log and the second log based on the matching scores corresponding to each of the first fields includes:

[0016] Obtain the sum of the matching scores corresponding to each of the first fields, and determine it as the matching degree between the first log and the second log.

[0017] In one possible design, comparing a first field in the first log and a second field in the second log to determine the similarity between the first log and the second log includes:

[0018] The first field in the first log is matched one by one with the second field in the second log to determine the similarity score corresponding to each first field in the first log;

[0019] Get the log length corresponding to the first log;

[0020] The similarity between the first log and the second log is determined based on the similarity scores corresponding to each of the first fields and the log length.

[0021] In one possible design, matching the first field in the first log with the second field in the second log one by one includes:

[0022] The system retrieves a preset number of first fields from the first log in a preset order, and a preset number of second fields from the second log in a preset order.

[0023] If the first field and the second field are completely identical, then the first field in the first log and the second field in the second log will be matched one by one.

[0024] In one possible design, after obtaining the logs to be processed, the following steps are also included:

[0025] The log to be processed is subjected to regular expression replacement, and the replaced log is then split to obtain multiple fields corresponding to the log to be processed.

[0026] In a second aspect, embodiments of the present invention provide a log processing device, comprising:

[0027] The log acquisition module is used to acquire logs to be processed.

[0028] The processing module is used to compare the first field in the first log and the second field in the second log for each second log corresponding to the first log, so as to determine the similarity and / or matching degree between the first log and the second log; wherein, the first log is any one of the logs to be processed, and the second log is the log to be processed other than the first log;

[0029] The processing module is further configured to determine that the first log and the second log belong to the same pattern log when the similarity between the first log and the second log is greater than a preset similarity, or the matching degree between the first log and the second log is greater than a preset matching degree.

[0030] In one possible design, the processing module is further configured to:

[0031] Based on the preset matching score rules, the first field in the first log is matched one by one with the second field in the second log to determine the matching score corresponding to each first field in the first log;

[0032] The matching degree between the first log and the second log is determined based on the matching score corresponding to each of the first fields.

[0033] In one possible design, the processing module is further configured to:

[0034] Before matching the first field in the first log with the second field in the second log one by one based on the preset matching score rules, the first log and the second log are aligned.

[0035] In one possible design, the processing module is further configured to:

[0036] Obtain the sum of the matching scores corresponding to each of the first fields, and determine it as the matching degree between the first log and the second log.

[0037] In one possible design, the processing module is further configured to:

[0038] The first field in the first log is matched one by one with the second field in the second log to determine the similarity score corresponding to each first field in the first log;

[0039] Get the log length corresponding to the first log;

[0040] The similarity between the first log and the second log is determined based on the similarity scores corresponding to each of the first fields and the log length.

[0041] In one possible design, the processing module is further configured to:

[0042] The system retrieves a preset number of first fields from the first log in a preset order, and a preset number of second fields from the second log in a preset order.

[0043] If the first field and the second field are completely identical, then the first field in the first log and the second field in the second log will be matched one by one.

[0044] In one possible design, the processing module is further configured to:

[0045] After obtaining the log to be processed, the log to be processed is replaced with a regular expression, and the replaced log to be processed is split to obtain multiple fields corresponding to the log to be processed.

[0046] Thirdly, embodiments of the present invention provide an electronic device, comprising: at least one processor and a memory;

[0047] The memory stores computer-executed instructions;

[0048] The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the log processing method as described in the first aspect and various possible designs of the first aspect.

[0049] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the log processing method described in the first aspect and various possible designs of the first aspect.

[0050] Fifthly, embodiments of the present invention provide a computer program product, including a computer program that, when executed by a processor, implements the log processing method described in the first aspect and various possible designs of the first aspect.

[0051] This invention provides a log processing method and apparatus. When a log to be processed is obtained, it indicates that logs belonging to the same pattern need to be filtered out from the log to be processed. Then, for any log to be processed, i.e., the second log corresponding to the first log (which is the second log other than the first log in the log to be processed), the fields in the first log (i.e., the first field) and the fields in the second log (i.e., the second field) are compared to obtain the similarity and / or matching degree between the first log and the second log. When the similarity is greater than a preset similarity or the matching degree is greater than a preset matching degree, it is determined that the first log and the second log belong to the same pattern log. It is not necessary to judge whether the length of the first log and the second log are the same. Instead, it is directly determined whether the first log and the second log belong to the same pattern log based on the similarity or matching degree between the first log and the second log. This achieves accurate judgment of logs with the same pattern and improves the accuracy of judging logs with the same pattern. Thus, logs belonging to the same pattern can be accurately filtered out from the log to be processed, improving the filtering accuracy of logs with the same pattern. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 A schematic diagram of a scenario for the log processing method provided in an embodiment of the present invention;

[0054] Figure 2 A flowchart illustrating the log processing method provided in this embodiment of the invention. Figure One ;

[0055] Figure 3 A flowchart illustrating the log processing method provided in this embodiment of the invention. Figure Two ;

[0056] Figure 4 This is a schematic diagram of a tree structure provided in an embodiment of the present invention;

[0057] Figure 5 This is a schematic diagram of the structure of a log processing device provided in an embodiment of the present invention;

[0058] Figure 6 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0060] Currently, when determining whether two log entries belong to the same pattern, the Drain algorithm, based on a fixed-depth parse tree, is generally used to calculate the similarity between the two log entries. Before calculating the similarity, it is necessary to determine whether the log entries have the same length. That is, the Drain algorithm is based on the strong assumption that log entries with the same pattern have the same length. If the log entries have the same length, the similarity between the two log entries is calculated to determine whether they belong to the same pattern. If the log entries have different lengths, it is directly determined that the two log entries do not belong to the same pattern, that is, they belong to different patterns. However, in reality, due to various reasons (for example, logs with the same pattern may sometimes add or remove several parameter fields; unlike system logs, application logs are printed by code developed by programmers from different companies; due to non-standard logging, other words are often added to logs with the same pattern), logs with the same pattern may have different lengths. Therefore, when using log length to determine whether logs belong to the same pattern, logs with different lengths but actually belong to the same pattern may be misjudged as not belonging to the same pattern, resulting in a low accuracy rate for judging logs with the same pattern, and thus a low accuracy rate for filtering logs with the same pattern.

[0061] Therefore, to address the aforementioned problems, the technical concept of this invention is to eliminate the fundamental idea of ​​logs with the same pattern having the same length, thereby improving the accuracy of identifying logs with the same pattern. The Needleman-Wunsch text comparison algorithm is employed to align the text of two logs of different lengths, and the matching score (i.e., the matching degree) between the aligned two logs is calculated according to matching scoring rules. Furthermore, the similarity between the two logs is also calculated. By using both the matching degree and similarity as indicators to determine whether two logs belong to the same pattern, the limitations of relying solely on the similarity indicator are avoided, further improving the accuracy of identifying logs with the same pattern, and consequently, improving the accuracy of filtering logs with the same pattern.

[0062] The technical solutions of this disclosure and how they solve the aforementioned technical problems are explained in detail below with specific examples. These specific examples can be combined with each other, and the same or similar concepts or processes may not be repeated in some examples. Examples of this disclosure will now be described with reference to the accompanying drawings.

[0063] Figure 1 This is a schematic diagram of a scenario for the log processing method provided in an embodiment of the present invention, such as... Figure 1 As shown, electronic device 101 obtains the logs generated by system 10 or application 20 corresponding to terminal device 102 during operation, namely the logs to be processed, and determines the logs belonging to the same mode from the logs to be processed.

[0064] Among them, electronic equipment 101 includes devices with data processing capabilities such as servers and computers.

[0065] The terminal device 102 can be a server, computer, or other similar device. Furthermore, the first device 101 and the terminal device 102 can be the same device; that is, the first device 101 acquires the logs it generates during operation and identifies them as logs to be processed.

[0066] Figure 2 A flowchart illustrating the log processing method provided in this embodiment of the invention. Figure One The execution entity in this embodiment can be Figure 1 The electronic device shown. For example... Figure 2 As shown, the method includes:

[0067] S201. Obtain logs to be processed.

[0068] S202. For each second log corresponding to the first log, compare the first field in the first log and the second field in the second log to determine the similarity and / or matching degree between the first log and the second log. Here, the first log is any log to be processed, and the second log is any log to be processed other than the first log.

[0069] In this embodiment, when logs generated by the operation of relevant systems and applications are obtained, these logs are treated as logs to be processed, and logs belonging to the same pattern are filtered out from these logs to be processed.

[0070] In this embodiment, a log to be processed is randomly selected from the logs to be processed, and the selected log to be processed is determined as the first log. The logs to be processed other than the first log are taken as the second logs corresponding to the first log, so as to filter out the logs that belong to the same pattern as the first log from the second logs.

[0071] For each second log corresponding to the first log, to determine whether the first log and the second log belong to the same pattern, the first field in the first log and the second field in the second log are compared to obtain the similarity and / or matching degree between the first log and the second log, so as to determine whether the first log and the second log belong to the same pattern based on the similarity or matching degree.

[0072] S203. If the similarity between the first log and the second log is greater than the preset similarity, or the matching degree between the first log and the second log is greater than the preset matching degree, then the first log and the second log are determined to belong to the same pattern log.

[0073] In this embodiment, when the similarity between the first log and the second log is greater than a preset similarity, or the matching degree between the first log and the second log is greater than a preset matching degree, it is determined that the first log and the second log belong to the same pattern log, so as to realize the determination of logs with the same pattern.

[0074] In addition, if the similarity between the first log and the second log is less than or equal to the preset similarity, and the matching degree between the first log and the second log is less than or equal to the preset matching degree, then it is determined that the first log and the second log do not belong to the same pattern log.

[0075] As described above, when logs to be processed are obtained, it indicates that logs belonging to the same pattern need to be filtered out from these logs. For any log to be processed (i.e., the second log corresponding to the first log, which is the second log other than the first log), the fields in the first log (i.e., the first field) and the fields in the second log (i.e., the second field) are compared to obtain the similarity and / or matching degree between the first and second logs. When the similarity is greater than a preset similarity or the matching degree is greater than a preset matching degree, it is determined that the first and second logs belong to the same pattern. It is not necessary to determine whether the lengths of the first and second logs are the same; instead, the determination of whether the first and second logs belong to the same pattern is directly based on the similarity or matching degree between them. This achieves accurate judgment of logs belonging to the same pattern, improving the accuracy of such judgments. Therefore, logs belonging to the same pattern can be accurately filtered out from the logs to be processed, thus improving the accuracy of filtering logs with the same pattern.

[0076] Figure 3 A flowchart illustrating the log processing method provided in this embodiment of the invention. Figure Two In this embodiment Figure 2 Based on the previous embodiment, after obtaining the log to be processed, the variable parameters in the log can be replaced to reduce the impact of the variable parameters. This process will be described below with reference to a specific embodiment.Figure 3 As shown, the method includes:

[0077] S301. Obtain logs to be processed.

[0078] S302. Perform regular expression replacement on the log to be processed, and split the replaced log to be processed to obtain multiple fields corresponding to the log to be processed.

[0079] In this embodiment, the log is generally composed of pattern parameters, i.e., pattern fields, and variable parameters, i.e., variable fields. For example, the pattern field in the log corresponding to the login pattern is "log in", and the variable parameters include user accounts, etc. When different users log in, the user accounts in the log will also change because the user accounts are variable parameters, while the pattern field, i.e., "login", will not change, so "login" is the pattern field.

[0080] In this embodiment, to avoid the variable parameters affecting the calculation of matching degree or similarity between logs, before determining logs belonging to the same pattern from the logs to be processed, a regular expression replacement is performed on each log to be processed for each day to be processed, that is, the variable parameters in the log to be processed are replaced with preset wildcards to obtain the replaced logs to be processed.

[0081] Since the fields in the log are generally separated by log delimiters, the replaced log to be processed can be split by log delimiters to obtain the various fields included in the replaced log, i.e., the token.

[0082] Optionally, the default wildcard can be <*>, but other symbols are also possible, which will not be elaborated here.

[0083] Optional, variable parameters include common parameters such as network address (Internet Protocol), file path, and numerical value.

[0084] Optionally, log separators include spaces.

[0085] For example, log 1 to be processed includes log A, which is BLOCK*ask 10.250.17.177:50010 todelete blk_-8570780307468499817 blk_-9122557405432088649 blk_-4393063808227796056 blk_8767569714374844347 blk_7079754042611867581 blk_7608961006114219538 blk_-5017273584996436939. After performing regular expression replacement on log 1 to be processed, the resulting replaced log 1 is BLOCK*ask<*>to delete<*>. Splitting the replaced log 1, the resulting log 1 includes the fields "BLOCK*", "ask", "to", and "delete".

[0086] S303. For each second log corresponding to the first log, compare the first field in the first log and the second field in the second log to determine the similarity and / or matching degree between the first log and the second log. Wherein, the first log is any one of the logs to be processed after replacement, and the second log is any log to be processed after replacement excluding the first log.

[0087] In any embodiment, optionally, comparing a first field in a first log and a second field in a second log to determine the degree of matching between the first log and the second log includes:

[0088] Based on the preset matching score rules, the first field in the first log is matched one by one with the second field in the second log to determine the matching score corresponding to each first field in the first log;

[0089] The matching degree between the first log and the second log is determined based on the matching score corresponding to each of the first fields.

[0090] Specifically, for each second log corresponding to the first log, the fields included in the first log (i.e., the first fields) are retrieved sequentially according to their order of appearance in the log, and then the fields included in the second log (i.e., the second fields) are retrieved sequentially. For each first field, it is matched with the corresponding second field to obtain a matching score, thus achieving a one-to-one match between the first and second fields. After obtaining the matching scores for each first field, the matching score between the first log and the second log is determined based on the matching scores of the first fields, i.e., the matching degree.

[0091] Optionally, when matching the first field with the second field at the corresponding position, if the first field is the same as the second field, the matching score corresponding to the first field is determined to be the first preset score (e.g., 10 points); if the first field is different from the second field, when the second field is empty, the matching score corresponding to the first field is determined to be the second preset score (e.g., 0 points); when the second field is not empty, the matching score corresponding to the first field is determined to be the third preset score (e.g., 1 point).

[0092] Taking a specific application scenario as an example, the first preset score is 10 points, the second preset score is 0 points, and the third preset score is 1 point. The first log contains three fields: field 1, field 2, and field 3. The second log contains two fields: field 1 and field 4. The first field in the first log is field 1, and the first field in the second log is also field 1. Since they are the same, the matching score for field 1 in the first log is determined to be 10. Next, the second field in the first log is matched with the second field in the second log. Because field 2 is different from field 4, the matching score for field 2 in the first log is 1. Finally, the third field in the first log is matched with the third field in the second log. Since the third field in the second log is empty, the matching score for field 3 in the first log is 0.

[0093] Further, optionally, when determining the matching degree between the first log and the second log based on the matching scores corresponding to each first field, the sum of the matching scores corresponding to each first field is obtained and determined as the matching degree between the first log and the second log. That is, the sum of the matching scores corresponding to all first fields in the first log is calculated and the sum is used as the matching degree between the first log and the second log.

[0094] Following the above application scenario, the matching score for field 1 in the first log is 10, the matching score for field 2 is 1, and the matching score for field 3 is 0. Therefore, the matching degree between the first log and the second log is determined to be 10 + 1 + 0 = 11.

[0095] Optionally, the first log and the second log may be aligned before matching the first field in the first log with the second field in the second log one by one.

[0096] Specifically, since the lengths of the first and second logs may differ, the second log can be aligned with the first log first. After alignment, the first field in the first log can be matched with the corresponding second field in the second log.

[0097] Alternatively, the Needleman-Wunsch text comparison algorithm can be used for alignment to obtain the aligned first and second logs. Then, the matching degree between the aligned first and second logs is calculated.

[0098] Specifically, the first log is stored in a tree, with the first field, i.e., the log pattern, as the leaf node. Since log lengths may differ, the second log needs to be aligned with the log pattern. The Needleman-Wunsch text comparison algorithm is used to construct a state transition matrix. The first row and first column of the matrix are initialized using the formula gap_penalty * i, where gap_penalty is a penalty for a gap, and i is the index of the element in that row or column. Other elements in the matrix are calculated using the scoring formula, and gap_penalty is set to 0. After obtaining the state transition matrix, backtracking begins from the bottom right element. If S1(i) = S2(j), backtracking proceeds to the top left cell; if S1(i) ≠ S2(j), backtracking proceeds to the cell with the largest value among the top left, top, and left sides. If cells have the same maximum value, priority is given in the order of top left, top, and left. Based on the backtracking path, the aligned string can be written, resulting in the aligned first and second logs. The scoring formula is as follows:

[0099]

[0100] as well as,

[0101]

[0102] The number of token layers in the tree can be controlled by the tree depth parameter. For example, a tree depth of 4 results in two token layers. Specifically, the tree structure used in this application is as follows: Figure 4 As shown.

[0103] In this embodiment, optionally, comparing the first field in the first log and the second field in the second log to determine the similarity between the first log and the second log includes:

[0104] The first field in the first log is matched one by one with the second field in the second log to determine the similarity score corresponding to each first field in the first log.

[0105] Get the log length corresponding to the first log entry;

[0106] The similarity between the first log and the second log is determined based on the similarity score corresponding to each of the first fields and the log length.

[0107] Specifically, for each first field in the first log, the first field is matched with the corresponding second field in the second log to determine the similarity between the first field and the corresponding second field, thus obtaining the similarity score for the first field. After obtaining the similarity scores for each first field in the first log, the sum of the similarity scores for all first fields in the first log is calculated, and the ratio of this sum to the log length of the first log is calculated to obtain the similarity between the first log and the second log.

[0108] The log length corresponding to the first log is the number of first fields included in the first log.

[0109] Further, optionally, when matching the first field with the second field at the corresponding position in the second log, if the first field is the same as the second field at the corresponding position, the similarity score corresponding to the first field is determined as the first similarity score; if the first field is not the same as the second field at the corresponding position, the similarity score corresponding to the first field is determined as the second similarity score.

[0110] Taking a specific application scenario as an example, the first similarity score is 1, and the second similarity score is 0. The first log contains three fields: field 1, field 2, and field 3. The second log contains two fields: field 1 and field 4. The first field in the first log is field 1, and the first field in the second log is also field 1. Since they are the same, the similarity score for field 1 in the first log is determined to be 1. Next, the second field in the first log is matched with the second field in the second log to calculate the similarity score. Since field 2 is different from field 4, the similarity score for field 2 in the first log is 0. Finally, the third field in the first log is matched with the third field in the second log. The third field in the second log is empty, and it is different from the third field in the first log. Therefore, the similarity score for field 3 in the first log is determined to be 0. The log constant value for the first log is 3, and the corresponding similarity between the first and second logs is 1 / 3.

[0111] Furthermore, optionally, since the first few fields of the log are generally pattern fields, i.e. pattern parameters, before determining the similarity between the first log and the second log, i.e., before determining whether the first log and the second log belong to the same pattern log, a preset number of first fields are obtained from the first log in a preset order, and a preset number of second fields are obtained from the second log in a preset order; if all the obtained first fields and the obtained second fields are the same, then the first fields in the first log and the second fields in the second log are matched one by one.

[0112] Specifically, following a preset order—that is, the order in which the fields in the logs are arranged—a preset number of first fields are retrieved from the first log, and a preset number of second fields are retrieved from the second log. If all the retrieved first and second fields are identical, it indicates that the first and second logs may belong to the same pattern. In this case, the first fields in the first log and the second fields in the second log are matched one by one, i.e., the similarity between the first and second logs is calculated to determine whether the first and second logs belong to the same pattern. If not all the retrieved first and second fields are identical, it indicates that the first and second logs do not belong to the same pattern. In this case, there is no need to calculate the similarity between the first and second logs to determine whether they belong to the same pattern, and the process continues to determine whether the next second log and the first log belong to the same pattern.

[0113] For example, the preset number is 2. This involves determining whether the first field in the first log and the first field in the second log are the same, and also determining whether the second field in the first log and the second log are the same. If both the first and second fields in the first and second logs are the same, then a one-to-one match is performed between the first and second fields, i.e., the similarity between the first and second logs is calculated. If neither the first nor the second field in the first and second logs is the same, or if neither the first nor the second field in the first and second logs is the same, then a one-to-one match is not performed, i.e., the similarity between the first and second logs is not calculated.

[0114] S304. If the similarity between the first log and the second log is greater than the preset similarity, or the matching degree between the first log and the second log is greater than the preset matching degree, then the first log and the second log are determined to belong to the same pattern log.

[0115] For example, with a preset similarity of 0.5 and a preset match of 30, log A is BLOCK*ask10.250.17.177:50010to delete blk_-8570780307468499817 blk_-9122557405432088649 blk_-4393063808227796056 blk_8767569714374844347 blk_7079754042611867581 blk_7608961006114219538 blk_-5017273584996436939. Log B is named BLOCK*ask 10.250.10.213:50010to delete blk_4029139044660806713 blk_-5471189807977280544. Log A and Log B actually have the same pattern, "BLOCK* ask<*>to delete<*>". However, because Log A contains many variable parameters, only four fields are identical: "BLOCK*", "ask", "to", and "delete". Therefore, the similarity between Log A and Log B after text alignment is only 4 / 12 ≈ 0.33, which is less than 0.5. If only the similarity index is used to determine whether logs belong to the same pattern, Log A and Log B will be classified as two different patterns. Since the matching score between Log A and Log B is 40 points, which is greater than 30, Log A and Log B are determined to be logs of the same pattern.

[0116] However, for some shorter logs, even if they are completely identical, the matching score alone may not reach the threshold. Therefore, similarity can be used for judgment. For example, log C is "Verification succeeded for blk_-8050165297538775793"; log D is "Verification succeeded for blk_-8347949825960763812". Logs C and D have the same actual pattern, "Verification succeeded for <*>", but since there are only 3 identical fields in logs C and D, the matching score is 30, which is not greater than the preset matching score, but the similarity score reaches 0.75. Therefore, the similarity score can be used to determine that logs C and D belong to the same pattern.

[0117] In this embodiment, when calculating similarity, that is, when determining logs of the same pattern, the calculation is not based on the premise that logs of the same pattern have the same length, thereby improving the accuracy of determining logs of the same pattern.

[0118] In this embodiment, when the proportion of variable parameters in the log is large, if only a single similarity index is used, the calculation result often fails to reach the threshold, which can easily separate logs of the same pattern, resulting in a low screening accuracy of logs of the same pattern. However, this application uses the similarity or matching degree between the first log and the second log to determine whether the first log belongs to the same pattern log, instead of using only similarity to make a judgment, thus avoiding the defects of a single index and improving the screening accuracy of logs of the same pattern.

[0119] Figure 5 This is a schematic diagram of the structure of the log processing device provided in an embodiment of the present invention, as shown below. Figure 5 As shown, the log processing device 500 includes a log acquisition module 501 and a processing module 502.

[0120] The log acquisition module 501 is used to acquire logs to be processed.

[0121] Processing module 502 is used to compare the first field in the first log and the second field in the second log for each second log corresponding to the first log, in order to determine the similarity and / or matching degree between the first log and the second log. Here, the first log is any log to be processed, and the second log is any log to be processed excluding the first log.

[0122] The processing module 502 is further configured to determine that the first log and the second log belong to the same pattern log when the similarity between the first log and the second log is greater than a preset similarity, or the matching degree between the first log and the second log is greater than a preset matching degree.

[0123] In one possible design, the processing module 502 is also used for:

[0124] Based on the preset matching score rules, the first field in the first log is matched one by one with the second field in the second log to determine the matching score corresponding to each first field in the first log.

[0125] The matching degree between the first log and the second log is determined based on the matching score corresponding to each of the first fields.

[0126] In one possible design, the processing module 502 is also used for:

[0127] Before matching the first field in the first log with the second field in the second log one by one based on the preset matching score rules, the first log and the second log are aligned.

[0128] In one possible design, the processing module 502 is also used for:

[0129] Get the sum of the matching scores corresponding to each first field, and determine it as the matching degree between the first log and the second log.

[0130] In one possible design, the processing module 502 is also used for:

[0131] The first field in the first log is matched one by one with the second field in the second log to determine the similarity score corresponding to each first field in the first log.

[0132] Get the log length corresponding to the first log.

[0133] The similarity between the first log and the second log is determined based on the similarity score corresponding to each of the first fields and the log length.

[0134] In one possible design, the processing module 502 is also used for:

[0135] The system retrieves a preset number of first fields from the first log in a preset order, and a preset number of second fields from the second log in a preset order.

[0136] If the first field and the second field are identical, then the first field in the first log and the second field in the second log will be matched one by one.

[0137] In one possible design, the processing module 502 is also used for:

[0138] After obtaining the logs to be processed, regular expression replacement is performed on the logs to be processed, and the replaced logs are split to obtain multiple fields corresponding to the logs to be processed.

[0139] The log processing device provided in this embodiment of the invention can implement the log processing method of the above embodiment. Its implementation principle and technical effect are similar, and will not be described again here.

[0140] Figure 6 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. For example... Figure 6 As shown, the electronic device 600 of this embodiment includes: a processor 601 and a memory 602;

[0141] Among them, memory 602 is used to store computer-executed instructions;

[0142] The processor 601 is configured to execute computer execution instructions stored in the memory to implement the various steps performed by the receiving device in the above embodiments. For details, please refer to the relevant descriptions in the foregoing method embodiments.

[0143] Alternatively, the memory 602 can be either standalone or integrated with the processor 601.

[0144] When the memory 602 is set up independently, the electronic device also includes a bus 603 for connecting the memory 602 and the processor 601.

[0145] This invention also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the log processing method described above.

[0146] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the log processing method described above.

[0147] In the several embodiments provided by this invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0148] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0149] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The unit composed of the above modules can be implemented in hardware or in the form of hardware plus software functional units.

[0150] The integrated modules implemented as software functional modules described above can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods described in the various embodiments of this application.

[0151] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly manifested as execution by a hardware processor, or execution by a combination of hardware and software modules within the processor.

[0152] The memory may include high-speed RAM, and may also include non-volatile storage (NVM), such as at least one disk storage device, and may also be a USB flash drive, external hard drive, read-only memory, disk or optical disc, etc.

[0153] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0154] The aforementioned storage medium can be implemented from any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium can be any available medium accessible to general-purpose or special-purpose computers.

[0155] An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of the processor. Both the processor and the storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and storage medium can exist as discrete components in an electronic device or host device.

[0156] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0157] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A log processing method, characterized in that, include: Get the logs to be processed; For each second log corresponding to the first log, the first field in the first log and the second field in the second log are compared to determine the similarity and / or matching degree between the first log and the second log; wherein, the first log is any one of the logs to be processed, and the second log is any log to be processed other than the first log. If the similarity between the first log and the second log is greater than a preset similarity, or the matching degree between the first log and the second log is greater than a preset matching degree, then it is determined that the first log and the second log belong to the same pattern log. The step of comparing the first field in the first log and the second field in the second log to determine the matching degree between the first log and the second log includes: Based on the preset matching score rules, the first field in the first log is matched one by one with the second field in the second log to determine the matching score corresponding to each first field in the first log; The matching degree between the first log and the second log is determined based on the matching score corresponding to each of the first fields. The step of comparing the first field in the first log and the second field in the second log to determine the similarity between the first log and the second log includes: The first field in the first log is matched one by one with the second field in the second log to determine the similarity score corresponding to each first field in the first log; Get the log length corresponding to the first log; The similarity between the first log and the second log is determined based on the similarity scores corresponding to each of the first fields and the log length.

2. The method according to claim 1, characterized in that, Before matching the first field in the first log with the second field in the second log one by one based on the preset matching score rules, the process also includes: Align the first log and the second log.

3. The method according to claim 1, characterized in that, The step of matching the first field in the first log with the second field in the second log one by one includes: The system retrieves a preset number of first fields from the first log in a preset order, and a preset number of second fields from the second log in a preset order. If the first field and the second field are completely identical, then the first field in the first log and the second field in the second log will be matched one by one.

4. The method according to any one of claims 1 to 3, characterized in that, After obtaining the logs to be processed, the following is also included: The log to be processed is subjected to regular expression replacement, and the replaced log is then split to obtain multiple fields corresponding to the log to be processed.

5. A log processing device, characterized in that, include: The log acquisition module is used to acquire logs to be processed. The processing module is used to compare the first field in the first log and the second field in the second log for each second log corresponding to the first log, so as to determine the similarity and / or matching degree between the first log and the second log; wherein, the first log is any one of the logs to be processed, and the second log is the log to be processed other than the first log; The processing module is further configured to determine that the first log and the second log belong to the same pattern log when the similarity between the first log and the second log is greater than a preset similarity, or the matching degree between the first log and the second log is greater than a preset matching degree; based on a preset matching score rule, match the first field in the first log with the second field in the second log one by one to determine the matching score corresponding to each first field in the first log; determine the matching degree between the first log and the second log according to the matching score corresponding to each first field; and match the first field in the first log with the second field in the second log one by one to determine the similarity score corresponding to each first field in the first log. Obtain the log length corresponding to the first log; determine the similarity between the first log and the second log based on the similarity scores corresponding to each of the first fields and the log length.

6. An electronic device, characterized in that, include: At least one processor and memory; The memory stores computer-executed instructions; The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the log processing method as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by the processor, implement the log processing method as described in any one of claims 1 to 4.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the log processing method according to any one of claims 1 to 4.

Images