Log parsing method, device, server and storage medium

Abstract

The embodiment of the present invention relates to the field of network operation and maintenance, and discloses a log parsing method, device, server and storage medium. In the present invention, sample log data is clustered according to the length and the first and last keywords of each sample log in the sample log data, and the quality score of each log cluster obtained by the clustering process is determined, and then the log is parsed according to the multiple log clusters and the quality score, and the log length and the keyword are selected as the conditions for clustering, so that the speed of log parsing is greatly improved without losing the accuracy of log parsing, and the log parsing is performed according to the multiple log clusters and the quality score obtained by clustering, and the re-clustering is iteratively updated at the same time, so that the accuracy of log parsing is further improved, thereby improving the efficiency of log parsing while ensuring the accuracy of log parsing.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of network operation and maintenance, and in particular to a log parsing method. Background Art

[0002] Log parsing is to convert unstructured log text into structured log text. Log parsing has important applications in areas such as system error root cause location and anomaly detection. Based on the results of log parsing, the running order of programs in the system can be clearly understood, which can be further used to build program workflows and detect anomalies in the system.

[0003] Currently commonly used log parsing methods include: parsing log templates online in a streaming form based on the longest common subsequence. In actual application, a prefix tree is usually generated to reduce the search time; and a fixed-depth log parsing tree is constructed by assigning the first several fields of the log to different nodes on the tree to speed up the search.

[0004] The inventors have found that there are at least the following problems in the prior art: the constructed log parsing tree only uses several fields of the log, which speeds up the parsing process to a certain extent, but substantially reduces the accuracy of parsing the log. Summary of the invention

[0005] The purpose of the embodiments of the present invention is to provide a log parsing method, device, server and storage medium, so that the log parsing efficiency can be improved while ensuring the log parsing accuracy during log parsing.

[0006] To solve the above technical problems, an embodiment of the present invention provides a log parsing method, comprising: obtaining sample log data; clustering the sample log data according to the length and first and last keywords of each sample log in the sample log data to obtain multiple log clusters; determining the quality score of each log cluster in the multiple log clusters; parsing the log using the multiple log clusters and the quality scores; wherein the quality score is used to determine the adaptive similarity threshold of the log when parsing the log.

[0007] An embodiment of the present invention also provides a log parsing device, including: an acquisition module, used to acquire sample log data; a clustering module, used to cluster the sample log data according to the length and first and last keywords of each sample log in the sample log data to obtain multiple log clusters; a scoring module, used to determine the quality score of each log cluster in the multiple log clusters; a parsing module, used to parse logs using the multiple log clusters and the quality score; wherein the quality score is used to determine the adaptive similarity threshold of the log when parsing the log.

[0008] An embodiment of the present invention also provides a server, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above-mentioned log parsing method.

[0009] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program, wherein the computer program implements the above-mentioned log parsing method when executed by a processor.

[0010] Compared with the prior art, the embodiments of the present invention cluster the sample log data according to the length and the first and last keywords of each sample log in the sample log data, determine the quality score of each log cluster in the multiple log clusters obtained by the clustering process, and then parse the log according to the multiple log clusters and the quality scores. By selecting the log length and the keyword as the conditions for clustering, the speed of log parsing is greatly improved without losing the accuracy of log parsing. By performing log parsing according to the multiple log clusters and the quality scores obtained by clustering and iteratively updating the re-clustering at the same time, the accuracy of log parsing is further improved, thereby improving the efficiency of log parsing while ensuring the accuracy of log parsing. In addition, the clustering processing of the sample log data according to the length and the first and last keywords of each sample log in the sample log data includes: performing the following processing on each sample log in the sample log data: judging whether there is a log cluster matching the length of the sample log, if so, obtaining the log cluster matching the length of the sample log, if not, creating a log cluster using the sample log; judging whether there is a log cluster matching the first and last keywords of the sample log in the log cluster matching the length of the sample log, if so, obtaining the log cluster matching the first and last keywords of the sample log, if not, creating a log cluster using the sample log; determining the similarity between all log signatures in the log cluster matching the first and last keywords of the sample log and the sample log, if any similarity among all the similarities is higher than a preset threshold, inserting the sample log into the log cluster to which the log corresponding to the similarity higher than the preset threshold belongs, otherwise creating a log cluster using the sample log. By first classifying the logs according to their length, and then further completing the clustering process based on the first and last keywords of the logs and the log similarity, the speed of log parsing is greatly improved without losing the accuracy of log parsing.

[0011] In addition, determining the quality score of each log cluster in the plurality of log clusters includes: determining the quality score of each log cluster according to the compactness within each log cluster and the separation between different log clusters in all the log clusters. The product of the compactness within the cluster and the separation between clusters is used as the quality score, so that the quality score can reflect the similarity of the logs in each log cluster.

[0012] In addition, the compactness is defined as the ratio of the constant to the total length in the log, and the separability is defined as the difference between the discrete item sets of two different logs. The discrete item set is a set of discrete item pairs, and the discrete item pairs refer to all item pairs generated by two fields in each log. Specific definitions of separability and compactness are provided.

[0013] In addition, the log parsing method using the multiple log clusters and the quality score includes: obtaining the target log to be parsed; filtering out irrelevant information in the target log using regular expressions; determining a matching log cluster from the multiple log clusters according to the length and the first and last keywords of the target log, and determining the adaptive similarity between the target log and the matching log cluster; determining the adaptive similarity threshold between the target log and the matching log cluster according to the quality score; judging whether the adaptive similarity is greater than the adaptive similarity threshold, if so, inserting the target log into the matching log cluster, otherwise creating a log cluster using the target log. Several log clusters with the highest similarity to the log to be parsed are quickly determined by log length and first and last keywords, and then the log cluster that best matches the log to be parsed is determined according to the quality score of each cluster, which greatly improves the speed of log parsing while ensuring the accuracy of log parsing.

[0014] In addition, after parsing the logs using the multiple log clusters and the quality scores, the method further includes: splitting the log clusters that meet the preset splitting conditions; wherein the splitting conditions include: the number of logs in the log cluster exceeds the preset log number threshold. By splitting the log clusters whose number of logs exceeds the threshold, the quality score of each log cluster is improved, and the parsing speed is prevented from being slowed down due to too many logs in the log cluster, which is also an update and optimization of all log clusters.

[0015] In addition, the splitting of the log cluster that meets the preset splitting condition includes: determining the position in the log cluster to be split whose field number is less than the preset field number threshold as a candidate position; calculating the Gini values ​​of all the candidate positions, and splitting the log cluster according to the candidate position corresponding to the maximum Gini value in the log cluster to be split. The position for splitting the log cluster is selected according to the field number of each position in the log cluster to ensure the effectiveness of the log cluster splitting.

[0016] In addition, after splitting the log clusters that meet the preset splitting conditions, the method further includes: searching for logs in which the keyword appears continuously and other fields are the same in each log cluster obtained by the splitting, and taking them as candidate logs for deletion; and deleting all logs except the log in which the keyword appears the least in the candidate logs for deletion. By deleting duplicate logs in the logs in which the keyword appears continuously and other fields are the same, the log parsing speed is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] One or more embodiments are exemplarily described by the pictures in the corresponding drawings, and these exemplary descriptions do not constitute limitations on the embodiments.

[0018] Figure 1 is a flow chart of a log parsing method according to a first embodiment of the present invention;

[0019] Figure 2 is a flow chart of a log parsing method according to a second embodiment of the present invention;

[0020] Figure 3 is a flow chart of a log parsing method according to a third embodiment of the present invention;

[0021] Figure 4 is a flow chart of a log parsing method according to a fourth embodiment of the present invention;

[0022] Figure 5 is a flow chart of a log parsing method according to a fifth embodiment of the present invention;

[0023] Figure 6 is a block diagram of a log analysis device according to a sixth embodiment of the present invention;

[0024] Figure 7 is a block diagram of the structure of a server according to a seventh embodiment of the present invention. DETAILED DESCRIPTION

[0025] To make the purpose, technical scheme and advantages of the embodiments of the present invention clearer, the embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings. However, it will be appreciated by those skilled in the art that in the embodiments of the present invention, many technical details are proposed in order to enable the reader to better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical scheme claimed in the present application can be implemented. The division of the following embodiments is for the convenience of description, and should not constitute any limitation on the specific implementation of the present invention. The various embodiments can be combined and referenced with each other without contradiction.

[0026] The first embodiment of the present invention relates to a log parsing method. In this embodiment, sample log data is obtained; the sample log data is clustered according to the length and the first and last keywords of each sample log in the sample log data to obtain multiple log clusters; the quality score of each log cluster in the multiple log clusters is determined; the log is parsed using the multiple log clusters and the quality score; wherein the quality score is used to determine the adaptive similarity threshold of the log when parsing the log.

[0027] The following is a detailed description of the implementation details of a log parsing method of this implementation method. The following content is only for the convenience of understanding the implementation details and is not necessary for the implementation of this solution.

[0028] A log parsing method in this embodiment is as follows: Figure 1 As shown, the following steps may be specifically included:

[0029] Step 101: Obtain sample log data.

[0030] Specifically, sample log data is obtained, and the obtained sample log data includes a plurality of sample logs, wherein the sample logs are filtered out of irrelevant information through regular expressions.

[0031] Step 102: clustering the sample log data according to the length and the first and last keywords of each sample log to obtain multiple log clusters.

[0032] Specifically, the length and the first and last keywords of the sample log are used as classification conditions to cluster the sample log data, and multiple log clusters are obtained after clustering. A log cluster is a special data structure used to store a set of log signatures, a row index list (used to store the log ID number belonging to the log cluster, and the log length and the first and last keywords are used as the row index in this application), the number of constant fields of each log signature, and the similarity threshold that can be classified into the log cluster.

[0033] Specifically, when multiple log clusters are obtained, the length and the first and last keywords of the logs in the log cluster are used as keys, and the logs in the log cluster are used as values ​​for cache processing.

[0034] Step 103: Determine a quality score of each log cluster in the plurality of log clusters.

[0035] Specifically, the quality score is used to determine the adaptive similarity threshold of the log when parsing the log; determine the quality score of each log cluster in the multiple log clusters, including: counting the compactness within each log cluster in all the log clusters and the separability between different log clusters, and using the normalized product of the compactness and the separability as the quality score of each log cluster in the multiple log clusters.

[0036] Furthermore, compactness refers to the ratio of the constant to the total length in the log signature. The compactness is defined as g, and its calculation formula is:

[0037]

[0038] Among them, ct r represents the number of constant fields in the log signature r, l r Indicates the length of r.

[0039] Separability refers to the difference between discrete item sets of different log signatures, where discrete item sets include multiple discrete item pairs. Discrete item pairs refer to all item pairs generated by deleting wildcards from logs. Separability is generally obtained by counting the Jaccard distances of discrete item pairs between different log clusters. The specific calculation formula of the Jaccard distance J is as follows:

[0040]

[0041] Among them, A is all field pairs generated by the compared log signature, and B is all field pairs generated by the existing log signature.

[0042] In a specific example, the log signature "RASKERNEL*generating core" in the log cluster contains a wildcard *, then its compactness The discrete item pairs of the log signature include (RAS, KERNEL), (RAS, generating), (RAS, core), (KERNEL, generating), (KERNEL, core) and (generating, core). The Jaccard distance of the discrete item pairs corresponding to each log cluster is counted to obtain the separability of the log cluster from other log clusters.

[0043] Step 104: Parse the logs using the multiple log clusters and quality scores.

[0044] Specifically, the logs to be parsed are input one by one in a streaming manner, and the input logs are compared and analyzed with multiple existing log clusters. During the analysis process, the quality score is used to determine the adaptive similarity threshold between the input log and the compared log cluster, thereby completing the parsing of the input log.

[0045] Compared with the prior art, this embodiment clusters the sample log data according to the length and the first and last keywords of each sample log in the sample log data, determines the quality score of each log cluster in the multiple log clusters obtained by the clustering process, and then parses the log according to the multiple log clusters and the quality scores. By selecting the log length and keywords as the conditions for clustering, the speed of log parsing is greatly improved without losing the accuracy of log parsing. By performing log parsing according to the multiple log clusters and quality scores obtained by clustering and iteratively updating the re-clustering at the same time, the accuracy of log parsing is further improved, thereby improving the efficiency of log parsing while ensuring the accuracy of log parsing.

[0046] The second embodiment of the present invention relates to a log parsing method. The second embodiment is substantially the same as the first embodiment, with the main difference being that the second embodiment of the present invention specifically provides a method for clustering sample log data according to the length and the first and last keywords of each sample log in the sample log data to obtain multiple log clusters.

[0047] A log parsing method in this embodiment is as follows: Figure 2 As shown, the following steps may be specifically included:

[0048] Step 201: Obtain sample log data. This step is similar to step 101 in the first embodiment and will not be described in detail here.

[0049] Step 202: Obtain a single sample log in the sample log data.

[0050] Step 203: Determine whether there is a log cluster that matches the length of the sample log. If yes, proceed to step 204; otherwise, proceed to step 205.

[0051] Specifically, multiple log clusters having the same log length as the sample log are determined from existing log clusters. If there is no log cluster matching the length of the sample log, a log cluster is created using the sample log.

[0052] In a specific example, the sample log is "RAS KERNEL INFO generating core", and a plurality of log clusters with "length=5" are determined.

[0053] Step 204 : Determine whether there is a log cluster matching the first and last keywords of the sample log among the log clusters matching the length of the sample log. If yes, proceed to step 206 ; otherwise, proceed to step 205 .

[0054] Specifically, first analyze whether the first and last keywords of the sample log contain any special characters. If they contain any special characters such as "_", the keyword at that position is replaced by the wildcard "*". Otherwise, the keyword is used as the judgment condition. In the log clusters that have been determined to match the log length of the sample log, multiple log clusters that match the first and last keywords of the sample log are determined. If there are no log clusters that match the first and last keywords of the sample log, a log cluster is created with the sample log.

[0055] In a specific example, the sample log is "RAS KERNEL INFO generating core". In this case, multiple log clusters whose first and last keywords of the log signature are "RAS" and "core" are determined in the log cluster of "length=5".

[0056] Step 205: Create a log cluster using a single piece of sample data.

[0057] Step 206 : Determine whether there is a log in the log cluster that matches the first and last keywords of the sample log whose similarity with the sample log is higher than a preset threshold. If yes, proceed to step 207 ; otherwise, proceed to step 205 .

[0058] Step 207: insert the sample log into the log cluster to which the log corresponding to the similarity higher than the preset threshold belongs.

[0059] Specifically, the similarity between the sample log and the existing logs in the log cluster is compared to determine whether the similarity exceeds the specified threshold. The sample log is then compared with all the logs in the log cluster in turn. If the similarity between the two is greater than the threshold, the sample log is classified as the cluster with the highest similarity among all matching log clusters. If any similarity is lower than the specified threshold, a new log cluster is created with the sample log.

[0060] Furthermore, the similarity sim can be determined by the following calculation formula:

[0061]

[0062] Among them, seq is the sample log, tem is the log in the log cluster to be compared, cost(seq i ,tem i ) indicates an exclusive OR operation, that is, if seq i with,tem i The same, the cost value is 1, if seq i with,tem i If they are not the same, the cost value is 0, where seq i is the i-th field of seq, tem iis the i-th field of tem, l is the log length of the sample log, n c The number of constant fields in the log currently being compared in this log cluster.

[0063] Furthermore, the preset threshold t can be determined by the following calculation formula:

[0064]

[0065] Among them, diff represents the different count values ​​at each position of the sample log and the log in the compared log cluster, float represents forcibly converting the variable data type to a floating point type, and n c The number of constant fields in the log currently being compared in this log cluster.

[0066] In a specific example, the sample log is "RAS KERNEL INFO generating core", and there is a log "RAS KERNEL*generating core" in the log cluster. Then the similarity is The similarity threshold is It is inserted into the log cluster.

[0067] Step 208: Determine whether all sample logs in the sample log data have been inserted into the log cluster. If yes, proceed to step 209; otherwise, proceed to step 202.

[0068] Step 209: Determine a quality score of each log cluster in the plurality of log clusters.

[0069] Step 210: Parse the logs using multiple log clusters and quality scores.

[0070] Step 209 and step 210 of this embodiment are similar to step 103 and step 104 in the first embodiment, and are not described in detail here.

[0071] Compared with the prior art, this embodiment performs clustering processing on the sample log data according to the length and the first and last keywords of each sample log in the sample log data, determines the quality score of each log cluster in the multiple log clusters obtained by the clustering processing, and then parses the log according to the multiple log clusters and the quality scores. By selecting the log length and keywords as the conditions for clustering processing, the speed of log parsing is greatly improved without losing the accuracy of log parsing.

[0072] The third embodiment of the present invention relates to a log parsing method. In this embodiment, the process of parsing logs using the multiple log clusters and the quality score is further refined: obtaining the target log to be parsed; using regular expressions to filter out irrelevant information in the target log; determining a matching log cluster from the multiple log clusters according to the length and the first and last keywords of the target log, and determining the adaptive similarity between the target log and the matching log cluster; determining the adaptive similarity threshold between the target log and the matching log cluster according to the quality score; judging whether the adaptive similarity is greater than the adaptive similarity threshold, if so, inserting the target log into the matching log cluster, otherwise creating a log cluster using the target log.

[0073] A log parsing method in this embodiment is as follows: Figure 3 As shown, the following steps may be specifically included:

[0074] Step 301: Obtain sample log data.

[0075] Step 302: clustering the sample log data according to the length and the first and last keywords of each sample log to obtain multiple log clusters.

[0076] Step 303: Determine the quality score of each log cluster in the plurality of log clusters.

[0077] Steps 301 to 303 of this embodiment are similar to steps 101 to 103 in the first embodiment, and are not described in detail here.

[0078] Step 304: Obtain the target log to be parsed.

[0079] Step 305: Use regular expressions to filter out irrelevant information in the target log.

[0080] Specifically, this step mainly includes the following two aspects: The first aspect is mainly the processing of position-independent items: In the original log, there are usually some fixed items, and these items appear in the same position in the same source log, for example, the first field in the log is usually the timestamp generated by the log; or these items change but have the same attributes, for example, in nova data, a request ID is generated at a fixed position. By analyzing the log features, meaningless data at these fixed positions can be filtered out, such as the records in the first column in this example are all timestamp attributes; The second aspect is mainly the processing of position-indeterminate related items: There are also some items in the log text that have the same attributes, but the positions where they appear are not fixed, and they are important parts of the log and cannot be deleted at will. Although the log text formats generated by different systems are different, some fields in the log also follow "international practices", such as IP addresses, file directory structures, etc. Therefore, obtaining pre-processed sample log data, that is, using a pre-set regular expression method, can greatly reduce the cost of parsing logs.

[0081] In a specific example, the original log to be parsed is "2017-10-07 12:00:13 RAS KERNELINFO generating core", and the processed log is "RAS KERNEL INFO generating core", that is, after irrelevant information in the target log is filtered out by using a regular expression, the log to be parsed becomes "RASKERNEL INFO generating core".

[0082] Step 306: Determine a matching log cluster from multiple log clusters according to the length of the target log and the first and last keywords.

[0083] Specifically, the length and the first and last keywords of the log to be parsed are directly used as the key, and a log cluster whose log length and the first and last keywords match the log to be parsed is obtained from multiple log clusters through cache query.

[0084] Step 307: Determine the adaptive similarity between the target log and the matching log cluster and the threshold of the adaptive similarity.

[0085] Specifically, the calculation method of the adaptive similarity is the same as the similarity calculation method in the second embodiment of the present invention; the adaptive similarity threshold is obtained by multiplying the similarity threshold in the second embodiment of the present invention by the quality score.

[0086] Furthermore, the adaptive similarity sim can be determined by the following calculation formula:

[0087]

[0088] Among them, seq is the log to be parsed, tem is the log in the matching log cluster, cost(seq i ,tem i ) indicates an exclusive OR operation, that is, if seq i with,tem i The same, the cost value is 1, if seq i with,tem i If they are not the same, the cost value is 0, where seq i is the i-th field of seq, tem i is the i-th field of tem, l is the length of the log to be parsed, n c The number of constant fields in the logs to be matched in this log cluster.

[0089] Furthermore, the adaptive similarity threshold t′ can be determined by the following calculation formula:

[0090]

[0091] Among them, diff represents the different count values ​​at each position of the log to be parsed and the log in the matching log cluster, float() represents forcibly converting the variable data type to floating point type, n c is the number of constant fields of the log currently being compared in the log cluster, and q is the quality score of the matching log cluster.

[0092] Step 308 : Determine whether the adaptive similarity is greater than the adaptive similarity threshold, if so, proceed to step 309 , otherwise proceed to step 310 .

[0093] Step 309: Insert the target log into the matching log cluster.

[0094] In a specific example, the log to be parsed is "RAS KERNEL INFO generating core", and there is a log "RAS KERNEL*generating core" in the log cluster. Then the adaptive similarity is The adaptive similarity threshold is Among them, q < 1, that is, the adaptive similarity is greater than the adaptive similarity threshold, then it is inserted into the log cluster.

[0095] Step 310: Create a log cluster using the target log.

[0096] In a specific example, parsing logs using multiple log clusters and quality scores includes: obtaining the entire data set, processing the target logs in turn, the original log to be parsed is "2017-10-07 12:00:13 Deleteblock blk_2342", and there is an existing log cluster "Delete block*". After regular expression filtering, the log to be parsed becomes "Delete block blk_2342". At this time, according to the length of the log "length=3", and the first and last keywords are "Delete" and "*" respectively ("blk_2342" contains the special character "_", which is replaced by "*"), the log cluster "Deleteblock*" is determined to be a matching log cluster, and the adaptive similarity between the log and the logs in the log cluster is compared. The adaptive similarity is The adaptive similarity threshold is Where q < 1, so the log can be directly inserted into the log cluster.

[0097] It is worth mentioning that while parsing the target log, the multiple log clusters that have been obtained can be iteratively re-clustered to recalculate the quality score of each log cluster. The log clusters are continuously updated according to the real-time log parsing process, thereby further improving the accuracy of log parsing during the parsing process.

[0098] Compared with the prior art, the present embodiment obtains sample log data; clusters the sample log data according to the length and the first and last keywords of each sample log in the sample log data to obtain multiple log clusters; determines the quality score of each log cluster in the multiple log clusters; and parses the log using the multiple log clusters and the quality score; wherein, parsing the log using the multiple log clusters and the quality score includes: obtaining the target log to be parsed; filtering out irrelevant information in the target log using a regular expression; determining a matching log cluster in the multiple log clusters according to the length and the first and last keywords of the target log, and determining the adaptive similarity between the target log and the matching log cluster; determining the adaptive similarity threshold between the target log and the matching log cluster according to the quality score; judging whether the adaptive similarity is greater than the adaptive similarity threshold, if so, inserting the target log into the matching log cluster, otherwise creating a log cluster using the target log. Several log clusters with the highest similarity to the log to be parsed are quickly determined by the log length and the first and last keywords, and then the log cluster that best matches the log to be parsed is determined according to the quality score of each cluster, and iteratively updating the re-clustering and recalculating the log score of each cluster at the same time, which greatly improves the speed of log parsing while ensuring the accuracy of log parsing.

[0099] The fourth embodiment of the present invention relates to a log parsing method. This embodiment is further improved on the basis of the first embodiment: after parsing the logs using multiple log clusters and the quality score, it also includes: splitting the log clusters that meet the preset splitting conditions; wherein the splitting conditions include: the number of logs in the log cluster exceeds the preset log number threshold.

[0100] A log parsing method in this embodiment is as follows: Figure 4 As shown, the following steps may be specifically included:

[0101] Step 401: Obtain sample log data.

[0102] Step 402: clustering the sample log data according to the length and the first and last keywords of each sample log to obtain multiple log clusters.

[0103] Step 403: Determine the quality score of each log cluster in the plurality of log clusters.

[0104] Step 404: Parse the logs using the multiple log clusters and quality scores.

[0105] Steps 401 to 404 of this embodiment are similar to steps 101 to 104 of the first embodiment, and are not described in detail here.

[0106] Step 405: Determine the log cluster to be split.

[0107] Specifically, each log cluster in which the number of logs in all the log clusters exceeds the preset log number threshold is split into multiple log clusters. If the number of logs in the log cluster is less than the preset log number threshold, that is, the number of logs in the currently parsed log cluster is too small, and the information stored in the parameters of each position of the log in the log cluster is limited, the log cluster will not be split temporarily. If the number of logs in the log cluster is greater than the preset log number threshold, that is, the number of logs in the currently parsed log cluster is large, which affects the speed of log parsing. The preset log number threshold can be set by the technician according to the actual situation.

[0108] Step 406: Determine the candidate location of the log cluster to be split.

[0109] Specifically, the position where the number of fields in each of the log clusters to be split is less than a preset field number threshold and greater than 1 is determined as a candidate position. If the number of fields at a position in the log exceeds the preset field number threshold, it means that there are many types of parameters at the position, which means that the possibility that the position is a parameter is higher to a certain extent. In this case, the log cluster is not split according to the position, and the field number threshold can be set by the technician according to the actual situation.

[0110] Step 407: Determine the optimal position according to the Gini value of each candidate position.

[0111] Specifically, the Gini values ​​of all candidate positions are calculated (the Gini value is an indicator for selecting the optimal features when the classic decision tree CART is used for classification problems), and the Gini value of each candidate position is compared with the maximum Gini value of the log parameter. Among the candidate positions whose comparison result is less than the maximum Gini value of the log parameter, the Gini value with the largest value is selected as the optimal position. The Gini value of each position can be determined by the following calculation formula:

[0112]

[0113] Among them, l is the total number of unique fields that appear in the current position. is the number of occurrences of the corresponding unique field, and cnt is the total number of fields appearing at that position.

[0114] Specifically, because only when the entropy value of the parameter at a certain position in the log exceeds the minimum entropy value of the log parameter, the position can be used as a candidate position for splitting, the maximum Gini value of the log parameter is set. When the Gini value of the candidate position is less than the maximum Gini value of the log parameter, it means that the entropy value of the log parameter is greater than the minimum entropy value of the log parameter. The maximum Gini value of the log parameter can be set by the technician according to the actual situation.

[0115] Step 408: Split the log cluster to be split into multiple log clusters according to the optimal position.

[0116] Specifically, different fields at the optimal position are assigned to different log clusters, and the fields at other positions remain unchanged, so that the log cluster to be split is split into multiple log clusters.

[0117] In a specific example, suppose a given log cluster contains "RAS KERNEL INFO generating core" and "RAS KERNEL FAILED generating core", first determine whether the number of logs in the log cluster exceeds the preset log number threshold template, if not, skip the splitting of the log cluster, if it exceeds the threshold (such as the log cluster contains 100 logs, the threshold is set to 50), determine the number of fields at each position of the log signature of the log cluster, where there is only one field RAS at position 1, only one field KERNEL at position 2, and two different fields INFO and FAILED at position 3. The position with only one field is skipped and not processed, and those positions with the number of fields less than the preset field number threshold are taken as candidate positions (here position 2 and position 3 are selected as candidate positions), the Gini value of each candidate position is calculated, and it is compared with the maximum Gini value of the log parameter, and the one with the largest Gini value is selected from each candidate position as the optimal position, and the field at the position is split, such as splitting the log cluster according to the fields INFO and FAILED at position 3, then the corresponding log clusters "RAS KERNEL INFO generatingcore" and "RAS KERNEL FAILED generating core" are obtained after splitting.

[0118] Compared with the prior art, this embodiment obtains sample log data; clusters the sample log data according to the length and the first and last keywords of each sample log in the sample log data to obtain multiple log clusters; determines the quality score of each log cluster in the multiple log clusters; parses the logs using the multiple log clusters and the quality scores; splits the log clusters that meet the preset splitting conditions; wherein the splitting conditions include: the number of logs in the log cluster exceeds the preset log number threshold. . By splitting the log clusters whose number of logs exceeds the threshold, the quality score of each log cluster is improved, and the slow parsing speed due to the excessive number of logs in the log cluster can be prevented, which is also an update and optimization of all log clusters.

[0119] The fifth embodiment of the present invention relates to a log parsing method. This embodiment is further improved on the basis of the fourth embodiment: after each log cluster in which the number of logs exceeds a preset log number threshold is split into multiple log clusters, it also includes: in each log cluster obtained by splitting, searching for logs in which the keyword appears continuously and other fields are the same as candidate deletion logs; in the candidate deletion logs, deleting all the logs except the log in which the keyword appears the least number of times.

[0120] A log parsing method in this embodiment is as follows: Figure 5 As shown, the following steps may be specifically included:

[0121] Step 501: Obtain sample log data.

[0122] Step 502: clustering the sample log data according to the length and the first and last keywords of each sample log to obtain multiple log clusters.

[0123] Step 503: Determine the quality score of each log cluster in the plurality of log clusters.

[0124] Step 504: Parse the logs using the multiple log clusters and quality scores.

[0125] Steps 501 to 504 of this embodiment are similar to steps 101 to 104 in the first embodiment, and are not described in detail here.

[0126] Step 505: Split each log cluster in which the number of logs exceeds a preset log number threshold in all log clusters into multiple log clusters.

[0127] Step 506: Delete all logs except the log with the least number of keyword appearances among the logs in which the keyword appears continuously but other fields are the same.

[0128] Specifically, this step is mainly aimed at the situation where logs with the same information have different lengths, mainly considering the following two situations:

[0129] (1) Merge consecutive identical preprocessing variables into one preprocessing variable, such as normalizing "delete block blk_objectblk_object" and "delete block blk_objectblk_objectblk_object" into "delete block blk_object";

[0130] (2) Consecutive parameter wildcards are merged into one parameter wildcard, such as normalizing "AB**C" and "AB***C" into "AB*C".

[0131] Compared with the prior art, this embodiment obtains sample log data; clusters the sample log data according to the length and the first and last keywords of each sample log in the sample log data to obtain multiple log clusters; determines the quality score of each log cluster in the multiple log clusters; parses the logs using the multiple log clusters and the quality score; splits each log cluster in which the number of logs exceeds the preset log number threshold in all log clusters into multiple log clusters; in each log cluster obtained by the splitting, searches for logs in which the keyword appears continuously and other fields are the same as candidate deletion logs; in the candidate deletion logs, delete all the logs except the log in which the keyword appears the least number of times. By deleting duplicate logs in logs in which the keyword appears continuously and other fields are the same, the log parsing speed is improved.

[0132] The step division of the above methods is only for the purpose of clear description. When implemented, they can be combined into one step or some steps can be split and decomposed into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this patent; adding insignificant modifications to the algorithm or process or introducing insignificant designs without changing the core design of the algorithm and process are all within the scope of protection of this patent.

[0133] The sixth embodiment of the present invention relates to a log parsing device, including: an acquisition module 601, a clustering module 602, a scoring module 603 and a parsing module 604, the specific structure of which is as follows: Figure 6 As shown:

[0134] The acquisition module 601 is used to acquire sample log data.

[0135] The clustering module 602 is used to perform clustering processing on the sample log data according to the length and the first and last keywords of each sample log in the sample log data to obtain multiple log clusters.

[0136] The scoring module 603 is used to determine the quality score of each log cluster in the multiple log clusters;.

[0137] The parsing module 604 is configured to parse the log using the multiple log clusters and the quality scores.

[0138] In a specific example, clustering the sample log data according to the length and first and last keywords of each sample log in the sample log data includes: performing the following processing on each sample log in the sample log data: determining whether there is a log cluster matching the length of the sample log, if so, obtaining the log cluster matching the length of the sample log, if not, creating a log cluster using the sample log; determining whether there is a log cluster matching the first and last keywords of the sample log in the log cluster matching the length of the sample log, if so, obtaining the log cluster matching the first and last keywords of the sample log, if not, creating a log cluster using the sample log; determining the similarity between all log signatures in the log cluster matching the first and last keywords of the sample log and the sample log, if any similarity among all the similarities is higher than a preset threshold, inserting the sample log into the log cluster to which the log corresponding to the similarity higher than the preset threshold belongs, otherwise creating a log cluster using the sample log.

[0139] In a specific example, determining the quality score of each log cluster among the multiple log clusters includes: determining the quality score of each log cluster according to the compactness within each log cluster and the separability between different log clusters among all the log clusters.

[0140] In a specific example, in addition, the log parsing using the multiple log clusters and the quality score includes: obtaining a target log to be parsed; filtering out irrelevant information in the target log using a regular expression; determining a matching log cluster from the multiple log clusters based on the length and first and last keywords of the target log, and determining an adaptive similarity between the target log and the matching log cluster; determining an adaptive similarity threshold between the target log and the matching log cluster based on the quality score; determining whether the adaptive similarity is greater than the adaptive similarity threshold, and if so, inserting the target log into the matching log cluster, otherwise creating a log cluster using the target log.

[0141] In a specific example, after parsing the logs using the multiple log clusters and the quality scores, the method further includes: splitting the log clusters that meet a preset splitting condition; wherein the splitting condition includes: the number of logs in the log cluster exceeds a preset log number threshold.

[0142] In a specific example, splitting the log cluster that meets the preset splitting conditions includes: determining the position in the log cluster to be split where the number of fields is less than a preset field number threshold as a candidate position; calculating the Gini values ​​of all the candidate positions, and splitting the log cluster according to the candidate position corresponding to the maximum Gini value in the log cluster to be split.

[0143] In a specific example, after splitting the log clusters that meet the preset splitting conditions, it also includes: in each log cluster obtained by the split, searching for logs in which the keyword appears continuously and other fields are the same, as candidate deletion logs; among the candidate deletion logs, deleting all the logs except the log in which the keyword appears the least number of times.

[0144] It is not difficult to find that this embodiment is a device embodiment corresponding to the first embodiment, and this embodiment can be implemented in conjunction with the first embodiment. The relevant technical details mentioned in the first embodiment are still valid in this embodiment, and in order to reduce repetition, they are not repeated here. Accordingly, the relevant technical details mentioned in this embodiment can also be applied in the first embodiment.

[0145] It is worth mentioning that all modules involved in this embodiment are logic modules. In practical applications, a logic unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. In addition, in order to highlight the innovative part of the present invention, this embodiment does not introduce units that are not closely related to solving the technical problem proposed by the present invention, but this does not mean that there are no other units in this embodiment.

[0146] A seventh embodiment of the present invention relates to a server, such as Figure 7 As shown, the server includes at least one processor 701; and a memory 702 communicatively connected to the at least one processor 701; and a communication component 703 communicatively connected to the log parsing device, and the communication component 703 receives and sends data under the control of the processor 701; wherein the memory 702 stores instructions that can be executed by at least one processor 701, and the instructions are executed by at least one processor 701 to implement the above-mentioned log parsing method embodiment.

[0147] Specifically, the electronic device includes: one or more processors 701 and a memory 702, Figure 7 A processor 701 is taken as an example. The processor 701 and the memory 702 may be connected via a bus or other means. Figure 7 The example of the bus connection is taken as an example. The memory 702 is a computer-readable storage medium that can be used to store computer software programs, computer executable programs and modules. The processor 701 executes various functional applications and data processing of the device by running the computer software programs, instructions and modules stored in the memory 702, that is, realizing the above-mentioned log parsing method.

[0148] The memory 702 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and applications required for at least one function; the data storage area may store a list of options, etc. In addition, the memory 702 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 702 may optionally include a memory remotely arranged relative to the processor 701, and these remote memories may be connected to an external device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0149] One or more modules are stored in the memory 702, and when executed by one or more processors 701, the log parsing method in any of the above method implementations is executed.

[0150] The above-mentioned product can execute the method provided in the implementation mode of this application, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not fully described in this implementation mode, please refer to the method provided in the implementation mode of this application.

[0151] Among them, the memory and the processor are connected in a bus manner, and the bus may include any number of interconnected buses and bridges, and the bus connects various circuits of one or more processors and memories together. The bus can also connect various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be one element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices on a transmission medium. The data processed by the processor is transmitted on a wireless medium via an antenna, and further, the antenna also receives data and transmits the data to the processor.

[0152] The processor is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory can be used to store data used by the processor when performing operations.

[0153] The eighth embodiment of the present invention relates to a computer-readable storage medium storing a computer program, which implements the above-mentioned log parsing method embodiment when executed by a processor.

[0154] That is, those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a program, and the program is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0155] Those skilled in the art will appreciate that the above-mentioned embodiments are specific examples for implementing the present invention, and in actual applications, various changes may be made thereto in form and detail without departing from the spirit and scope of the present invention.

Claims

1. A log parsing method, characterized in that: include: Get sample log data; Clustering the sample log data according to the length and the first and last keywords of each sample log in the sample log data to obtain multiple log clusters; Determine the quality score of each log cluster according to the compactness within each log cluster and the separability between different log clusters in all log clusters; wherein the compactness is defined as the ratio of the number of constants in the log to the total length, and the separability is defined as the difference between discrete item sets of two different logs; wherein the discrete item set is a set of discrete item pairs, and the discrete item pairs refer to all item pairs generated by two fields in each log; parsing logs using the plurality of log clusters and the quality scores; The quality score is used to determine an adaptive similarity threshold of the log when parsing the log.

2. The log parsing method according to claim 1, characterized in that: The clustering process of the sample log data according to the length and the first and last keywords of each sample log in the sample log data includes: Each sample log in the sample log data is processed as follows: Determine whether there is a log cluster matching the length of the sample log, if yes, obtain the log cluster matching the length of the sample log, if no, create a log cluster using the sample log; Determine whether there is a log cluster matching the first and last keywords of the sample log in the log cluster matching the length of the sample log, if yes, obtain the log cluster matching the first and last keywords of the sample log, if not, create a log cluster using the sample log; Determine the similarity between all log signatures in the log cluster that matches the first and last keywords of the sample log and the sample log; if any similarity among all the similarities is higher than a preset threshold, insert the sample log into the log cluster to which the log corresponding to the similarity higher than the preset threshold belongs; otherwise, create a log cluster using the sample log.

3. The log parsing method according to claim 1, characterized in that: The utilizing the plurality of log clusters and the quality score to parse the logs comprises: Get the target log to be parsed; Use regular expressions to filter out irrelevant information in the target log; Determine a matching log cluster from the multiple log clusters according to the length and the first and last keywords of the target log, and determine the adaptive similarity between the target log and the matching log cluster; Determining an adaptive similarity threshold between the target log and the matching log cluster according to the quality score; It is determined whether the adaptive similarity is greater than the adaptive similarity threshold; if so, the target log is inserted into the matching log cluster; otherwise, a log cluster is created using the target log.

4. The log parsing method according to claim 1, characterized in that: After parsing the logs using the multiple log clusters and the quality scores, the method further includes: Split the log clusters that meet the preset splitting conditions; The splitting condition includes: the number of logs in the log cluster exceeds a preset log number threshold.

5. The log parsing method according to claim 4, characterized in that: The splitting of the log cluster that meets the preset splitting condition includes: Determine a position in the log cluster to be split where the number of fields is less than a preset field number threshold as a candidate position; The Gini values ​​of all the candidate positions are calculated, and the log cluster is split according to the candidate position corresponding to the maximum Gini value in the log cluster to be split.

6. The log parsing method according to claim 4, characterized in that: After splitting the log cluster that meets the preset splitting condition, the method further includes: In each log cluster obtained by splitting, the logs with consecutive keyword appearances and the same other fields are selected as candidate logs for deletion; Among the candidate deletion logs, all the logs except the log in which the keyword appears the least number of times are deleted.

7. A log analysis device, characterized in that: include: The acquisition module is used to obtain sample log data; A clustering module, used for clustering the sample log data according to the length and the first and last keywords of each sample log in the sample log data to obtain multiple log clusters; A scoring module, used for determining the quality score of each log cluster according to the compactness within each log cluster and the separability between different log clusters in all log clusters; wherein the compactness is defined as the ratio of the number of constants in the log to the total length, and the separability is defined as the difference between discrete item sets of two different logs; wherein the discrete item set is a set of discrete item pairs, and the discrete item pairs refer to all item pairs generated by two fields in each log; A parsing module, configured to parse logs using the plurality of log clusters and the quality scores; The quality score is used to determine an adaptive similarity threshold of the log when parsing the log.

8. A server, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the log parsing method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the log parsing method according to any one of claims 1 to 6 is implemented.

Images