Log classification method, electronic device and computer readable storage medium
The log classification method clusters log data into groups using similarity-based feature vectors and vector edit distances, addressing inefficiencies in existing methods by reducing data volume and enhancing classification efficiency and real-time monitoring.
Patent Information
- Application Number
- CN202111350287.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-11-15
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2041-11-15
AI Technical Summary
Existing log classification methods are inefficient due to the rapid increase in log data volume and diversity, leading to unstructured text data processing challenges, lack of real-time notification, and manual keyword searches, which fail to provide timely alerts and effective classification strategies.
A log classification method that clusters log data into groups based on similarity, using feature vectors and vector edit distances to reduce data volume and improve classification efficiency, followed by final classification into predefined sets with reduced similarity calculations.
This approach significantly enhances log classification efficiency by reducing data volume and improving accuracy through fewer similarity calculations, enabling real-time monitoring and early warning prompts.
Smart Images

Figure CN114090850B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of log processing, and in particular to a log classification method, an electronic device, and a computer-readable storage medium. Background Art
[0002] As more and more application systems are added, each system has logs, which play a very important role in various business services. The growth of business volume has led to a rapid increase in traffic. The number of logs recorded by the system is increasing, and each system records a variety of logs. The log content will be diverse, and the amount of data stored in the logs has also increased dramatically.
[0003] System logs are usually unstructured text data. The related existing technical solutions are to use keywords to query or filter all logs based on rules after an exception occurs in the application, and there is no good log classification strategy. Summary of the invention
[0004] In order to solve the above problems, the present application provides a log classification method, an electronic device and a computer-readable storage medium, which can reduce the number of similarity calculations performed on each type of log in the classified final log classification set, thereby improving the classification efficiency of log data.
[0005] To solve the above technical problems, a technical solution adopted in the present application is: 1. providing a log classification method, the method comprising: obtaining log data; clustering the log data according to the similarity between multiple logs in the log data to obtain multiple log groups; according to the similarity between the logs in each log group and each type of log in the classified final log classification set, clustering the logs in the log group whose similarity meets a preset threshold into the corresponding final log classification set.
[0006] Wherein, obtaining log data includes: obtaining original log data; preprocessing the original log data to obtain log data; wherein the data volume of the log data is smaller than the data volume of the original log data, and the log data includes multiple logs and the quantity corresponding to each log.
[0007] The original log data includes multiple original logs; the original log data is preprocessed to obtain log data, including: processing multiple original logs in the original log data according to preset rules to obtain multiple rule logs; merging the same logs in the multiple rule logs to obtain log data.
[0008] Among them, multiple original logs in the original log data are processed according to preset rules to obtain multiple rule logs, including: extracting log attributes of multiple original logs in the original log data, the log attributes including at least one of log time, log source, log level, log thread, log class, log event and log abnormal state; according to the preset rule log template and the log attribute of each log, multiple rule logs are generated accordingly.
[0009] The method comprises: clustering the log data according to the similarity between multiple logs in the log data to obtain multiple log groups, including: determining the feature vectors of multiple logs in the log data; determining the vector edit distance between the multiple logs according to the feature vectors of the multiple logs; clustering the multiple logs according to the vector edit distance between the multiple logs to obtain multiple log groups.
[0010] The log data includes a feature field, and determining feature vectors of multiple logs in the log data includes: determining a feature vector corresponding to each log according to the feature field and text features of each log in the log data.
[0011] Among them, according to the similarity between the logs in each log group and each type of log in the classified final log classification set, the logs in the log group whose similarity meets the preset threshold are clustered into the corresponding final log classification set, including: obtaining the source of each log group in multiple log groups; and obtaining the final log classification set with the same source as each log group; calculating the vector edit distance between the logs in each log group and each type of log in the final log classification set, and clustering the logs in each log group into a target log classification set, which is the final log classification set with the smallest vector edit distance with the corresponding log in the log group.
[0012] The method further includes: obtaining the increment of each type of log in the final log classification set within a preset time; if the increment exceeds a preset value, issuing an early warning prompt.
[0013] To solve the above technical problems, another technical solution adopted in the present application is: to provide an electronic device, which includes a processor and a memory coupled to the processor, the memory stores a computer program, and the processor is used to execute the computer program to implement the method provided by the above technical solution.
[0014] To solve the above technical problem, another technical solution adopted by the present application is: providing a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, it implements the method provided by the above technical solution.
[0015] The beneficial effect of the embodiment of the present application is: different from the prior art, the log classification method provided by the present application includes: obtaining log data; clustering the log data according to the similarity between multiple logs in the log data to obtain multiple log groups; clustering the logs in the log group whose similarity meets the preset threshold into the corresponding final log classification set according to the similarity between the logs in each log group and the various logs in the classified final log classification set. In the above manner, the log data is first clustered into multiple log groups, and then the final classification is performed with the corresponding classified final log classification set based on fewer log groups, which can reduce the number of similarity calculations with various logs in the classified final log classification set, thereby improving the classification efficiency of the log data. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:
[0017] Figure 1 This is a flow chart of the first embodiment of the log classification method provided by the present application;
[0018] Figure 2 It is a flow chart of the second embodiment of the log classification method provided by the present application;
[0019] Figure 3 It is a flowchart of an embodiment of step 22 provided by the present application;
[0020] Figure 4 It is a flowchart of an embodiment of step 221 provided by the present application;
[0021] Figure 5 It is a flowchart of the third embodiment of the log classification method provided by the present application;
[0022] Figure 6 It is a flowchart of an embodiment of step 52 provided by the present application;
[0023] Figure 7 is a flowchart of an embodiment of step 55 provided by the present application;
[0024] Figure 8 It is a structural schematic diagram of an embodiment of an electronic device provided by the present application;
[0025] Fig. 9 It is a structural schematic diagram of an embodiment of a computer-readable storage medium provided by the present application;
[0026] Fig.10 It is an application schematic diagram of the related technology provided by this application;
[0027] Fig.11 It is an application schematic diagram of the technical solution of this application provided by this application. DETAILED DESCRIPTION
[0028] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It will be understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for ease of description, only some but not all structures related to the present application are shown in the drawings. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.
[0029] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.
[0030] As the company's business develops, there are more and more systems, and each system has a log. Logs play a very important role in various business services. Usually, logs are mainly used to record key execution points, information when program execution errors occur, etc. When a system fails, developers can locate the cause of the failure through error logs.
[0031] The growth of business volume has led to a rapid increase in traffic. The number of logs recorded by the system continues to increase. Each system records a variety of logs. The log content will be diverse, and the amount of stored logs has also increased dramatically. The relevant technical solution is to record all logs in the storage system, such as Elasticsearch. The disadvantages are very obvious: developers cannot know how many logs are recorded. For example, they cannot know in real time which application has recorded the most logs, nor do they know which application in the company has recorded the most logs, which ones need to be recorded, and which ones do not need to be recorded.
[0032] System logs are usually unstructured text data. The related existing technical solutions are to use keywords to query or filter all logs based on rules after an exception occurs in the application. When an exception occurs, the business party is not notified immediately, and there is no timely alarm function. Traditional methods are highly dependent on experience. When new types of logs appear, keyword matching may not be applicable. At this time, a set of methods is needed to automatically find new logs and know how much the same type of logs have changed year-on-year and month-on-month. In the scenario of real-time update of massive logs, log changes require manual keyword search, which takes a long time and cannot meet the log alarm processing requirements.
[0033] Based on this, the present application solves at least one of the above-mentioned problems by first clustering log data into multiple log groups, and then performing final classification based on fewer log groups and the corresponding classified final log classification set. For details, please refer to any of the following embodiments.
[0034] See also Figure 1 , Figure 1 This is a flow chart of an embodiment of a log classification method provided by the present application. The method includes:
[0035] Step 11: Get log data.
[0036] Among them, each system has log data, and log data occupies a very important position in various business services. Such as application logs, security logs, system logs, Scheduler service logs, FTP logs, WWW logs, DNS server logs, etc. In general, log data is used to record information about hardware, software and system problems in the system, and can also monitor events that occur in the system. Users can use it to check the cause of the error or find traces left by the attacker when attacked. For example, if someone performs an IPC probe on the system, the system will quickly record the IP, time, user name, etc. used by the prober in the security log. After using FTP to probe, the IP, time, user name used for the probe, etc. will be recorded in the FTP log. For example, it is also used to record key execution points, information when program execution errors occur, etc., such as recording every user operation, such as, at XX time, the file was downloaded; at YY time, the file was uploaded, and for example, recording program execution errors, such as, at zz time, the file download failed. When a system fails, relevant personnel can locate the cause of the failure through error log data.
[0037] When a system has many users, there is a phenomenon that multiple log data are generated at the same time.
[0038] Step 12: Cluster the log data according to the similarity between multiple logs in the log data to obtain multiple log groups.
[0039] In some embodiments, the log data can be clustered using K-Means clustering, such as first randomly determining at least one log as the initial center point among multiple logs, then calculating the distance between the remaining logs and at least one initial center point, and dividing the log into the log group corresponding to the nearest initial center point. After preliminary classification, the center point is determined again from each log group, the distance between the remaining logs and at least one initial center point is recalculated, and the logs are re-divided, and iterated until the change of the log group tends to be flat, then the log group at this time is determined to be the clustered log group. Among them, the log group is a collection of logs, and there can be at least one log in a collection.
[0040] For example, there are 20 logs in total, such as log A-log T. First, three initial center points are determined, namely log A, log B and log C. Then, the distances between the remaining logs and the three initial center points are calculated to obtain three log groups, namely the first log group, the second log group and the third log group. Among them, the first log group includes log A and log D-log I, a total of 7 logs; the second log group includes log B and log J-log O, a total of 7 logs; the third log group includes log C and log P-log T, a total of 6 logs.
[0041] In the second iterative calculation, the center point is re-determined from the first log group, the second log group, and the third log group, and the distance between the remaining 17 logs and the new center point is recalculated to obtain three new log groups. In this way, the final log group is determined by iterating multiple times.
[0042] In other embodiments, the center point can be randomly determined each time to obtain the corresponding log group, and a log group can be obtained after each iteration according to the number of iterations. Then, each log group is calculated and the best log group is selected as the final log group.
[0043] In other embodiments, data clustering can be performed by using any of a mean shift clustering method, a density-based clustering method, a maximum expectation clustering method using a Gaussian mixture model, an agglomerative hierarchical clustering method, and a graph group detection method.
[0044] Step 13: Based on the similarity between the logs in each log group and each type of log in the classified final log classification set, the logs in the log group whose similarity meets a preset threshold are clustered into the corresponding final log classification set.
[0045] In this embodiment, there are multiple classified final log classification sets, and each final log classification set has different types of logs. For example, there are final log classification set A, final log classification set B, and final log classification set C. Final log classification set A has logs of types A1, A2, and A3, final log classification set B has logs of types B1, B2, and B3, and final log classification set C has logs of types C1, C2, and C3.
[0046] In step 13, the logs in each log group can be traversed and compared with the various logs in the final log classification set to determine the similarity between the logs in each log group and the various logs in the final log classification set. When the similarity between the log in the log group and the target type log in the final log classification set meets the preset threshold, the log in the log group and the log of the target type are classified into one category. If the similarity between the log and the various logs in the final log classification set does not meet the preset threshold, the log is treated as a separate category and added to the final log classification set, so that the logs of each log group are classified in the corresponding final log classification set.
[0047] In this embodiment, log data is obtained; the log data is clustered according to the similarity between multiple logs in the log data to obtain multiple log groups; the logs in each log group are clustered into the corresponding final log classification set according to the similarity between the logs in each log group and the various logs in the corresponding final log classification set. The log data is first clustered into multiple log groups, and then the final classification is performed with the corresponding final log classification set based on fewer log groups. This can reduce the number of similarity calculations with various logs in the final log classification set, thereby improving the classification efficiency of the log data.
[0048] See also Figure 2 , Figure 2 This is a flow chart of an embodiment of a log classification method provided by the present application. The method includes:
[0049] Step 21: Get the original log data.
[0050] The original log data includes multiple original logs. In this embodiment, the original log is a non-structured log, and each log contains a timestamp, application, log level, and class name. The system will generate tens of thousands of logs like this in a day.
[0051] Step 22: Preprocess the original log data to obtain log data.
[0052] The data volume of the log data is smaller than the data volume of the original log data, and the log data includes a plurality of logs and a quantity corresponding to each log.
[0053] For example, the logs with the same application and content in the original log data can be merged to obtain new log data, and the log data can be counted. These logs with the same application and content only differ in the time of generation, so they can be merged. Since the original log data of multiple logs are merged, the amount of log data is less than the amount of original log data.
[0054] In some embodiments, keyword extraction may be performed on data in the original log data, and the extracted keywords may be used to form a new log, ie, log data. Due to keyword extraction, the amount of new log data is reduced.
[0055] In some embodiments, see Figure 3 , step 22 may be the following process:
[0056] Step 221: Process multiple original logs in the original log data according to preset rules to obtain multiple rule logs.
[0057] Since the original log is an unstructured log, multiple original logs in the original log data are processed using preset rules to obtain multiple rule logs.
[0058] For example, the content of an original log is "2021-06-11 11:32:01.579[vayhapi][application wapi]ERROR[http-nio-8080-exec-74]SignChecker: Determine whether it is a student, the ID number is wrong: null exception".
[0059] The preset rule can be "Time+application name+log level+thread+class name+*ID number error*abnormal". The content of the original log is extracted according to the preset rule to obtain multiple rule logs. In this process, the content is extracted, which can reduce the data volume of the rule log.
[0060] In some embodiments, see Figure 4 , step 221 may be the following process:
[0061] Step 2221: extract log attributes of multiple original logs in the original log data.
[0062] The log attributes include at least one of log time, log source, log level, log thread, log class, log event and log abnormal state.
[0063] Extract the content corresponding to the log attributes in the original log data, such as the log time is the specific time when the log is generated, the log source is the application or source where the log is generated, the log level can be advanced, intermediate and elementary, the log thread is the thread that generates the log, the log class can be the class name method applied when the log is generated, the log event can be an event associated with the log generation, such as input error event, input correct event, and the log exception status indicates the result obtained after executing the log class, such as abnormal or normal.
[0064] Take the content of an original log as "2021-06-11 11:32:01.579[vayhapi][application wapi]ERROR[http-nio-8080-exec-74]SignChecker: Determine whether it is a student, ID number error: null exception". The extracted log attributes are: log time is 2021-06-11 11:32:01.579, log source is application wapi, log level is ERROR, log thread is http-nio-8080-exec-74, log class is SignChecker: Determine whether it is a student, log event is ID number error: null, and log exception status is exception.
[0065] Step 2222: Generate multiple rule logs according to the preset rule log template and the log attribute of each log.
[0066] According to the preset rule log template, the content of the log attribute of each log is filled in accordingly to generate corresponding multiple rule logs.
[0067] Step 222: Merge the same logs in multiple rule logs to obtain log data.
[0068] Since the log data has been adjusted to a regular log, it is possible to determine whether there are identical logs. The identical logs here refer to logs with identical contents except for the time of generation. For example, the log generated at the first time has the content "abnormal". The log generated at the second time has the content "abnormal". These two logs can be determined to be identical and merged. Merging here means retaining only one log content and counting the number of logs.
[0069] Step 23: Cluster the log data according to the similarity between multiple logs in the log data to obtain multiple log groups.
[0070] In this embodiment, the log feature fields of multiple logs in the log data can be calculated and the statistical method of text frequency can be used to extract the feature words in each log. The more frequently a word appears in the log, the more it can represent the log feature, such as the application name. The higher the frequency of a specific anomaly, the more it can represent that the log feature appearing in the log is abnormal.
[0071] Based on the TF-IDF (term frequency–inverse document frequency) algorithm, it is used to evaluate the importance of a word to a document set or a document in a corpus. The importance of a word increases in direct proportion to the number of times it appears in a document, but it decreases in inverse proportion to the frequency of its appearance in the corpus.
[0072] It can be expressed by the following formula:
[0073] Among them, TF w Represents the frequency of term w.
[0074] The features that can be extracted from the logs themselves include: word count, number of non-repeated occurrences, length, number of stop words, number of punctuation marks, number of capitalized words, number of title words, and average word length.
[0075] The word count indicates the number of words in each line of log text.
[0076] The number of unique words refers to the number of words that appear only once in each line of text.
[0077] The length indicates the length of each line of text and how much storage space it occupies.
[0078] The number of stop words represents statistics: the number of words such as between, but, about, very, etc.
[0079] The number of punctuation marks indicates the number of punctuation marks contained in each line of text.
[0080] The number of uppercase words means counting the number of uppercase words.
[0081] The number of title words refers to the number of words whose first letter is capitalized and the rest of the letters are lowercase.
[0082] The average length of words is the average length of each word in each line of text.
[0083] Then extract the feature vector of the feature words. For example, suppose all the feature word sets are S = {fw1, fw2, ..., fwn}, and they form an n-dimensional vector space in sequence. For multiple logs L and their feature field set WL in the log data, the feature vector is defined as: VL = [v1, v2, ..., vn]; where: vi = 1, fwi∈WL, i = 1, 2, ..., n; vi = 0,
[0084] The eigenvector of each log in the log data is calculated. After the eigenvector is calculated, one of the logs can be expressed as: L1 = [1, 0, 0, 1, 0, 1].
[0085] Then the vector edit distance between the multiple logs is determined based on the feature vectors of the multiple logs.
[0086] The following is an explanation of the string edit distance:
[0087] String edit distance is a method for calculating the similarity between strings. Given two strings S and T, the set of deletion, insertion, and replacement operations required to convert S to T is called the edit path from S to T. The shortest edit path is called the edit distance between strings S and T. The operation set contains three operations: deleting a node, inserting a node, and replacing the labels of two nodes. a→Λ represents deleting node a, and Λ represents an empty node; Λ→b represents inserting node b; and a→b represents replacing the label of node a with the label of node b.
[0088] Each of the three operations mentioned above has a cost, and the cost functions λ(a→Λ), λ(Λ→b), and λ(a→b) represent the cost of deleting node a, the cost of inserting node b, and the cost of replacing the label of node a with the label of node b. The cost is a non-negative real number. The value of the cost function is basically customized by the user.
[0089] To convert string S into string T, we need to go through {e1 e2 e3 e4...en} operations, where each ei (i = 1, 2, ..., n) is an operation. Let E = {e1 e2 e3 e4...en} be a continuous sequence of operations. Therefore, the total cost of converting S into T through the operation set E is λ(E) = ∑λ(ei) = λ(e1) + i = 1λ(e2) + ... + λ(en). This formula can be used to calculate the cost of converting S into T through E. The edit distance from S to T can be defined as: stringdist(S, T) = min{λ(E)|E is one of the paths}.
[0090] It can be seen that for characters S and T, the editing path required to convert S into T is positively correlated with the length of S and T. The longer S and T are, the longer it takes to calculate the editing path and the lower the efficiency.
[0091] Dynamic programming can be used to calculate the edit distance between two characters. For characters S and T, the time complexity can be quickly calculated to be O(|S|×|T|). It takes a lot of time to calculate irrelevant text areas, and the time efficiency is relatively low. The feature vector method proposed in this embodiment can extract feature fields from S and T, and then convert the feature fields to 0 and 1, and obtain feature vectors again, which can greatly reduce the length of S and T. Since the length of S and T is greatly reduced, the time for calculating the edit distance can be greatly reduced, thereby greatly improving the calculation efficiency.
[0092] The feature vector is converted into a constant character, and brought into the edit distance algorithm to calculate the edit distance. In this embodiment, for two different logs, their edit distances are first calculated. If they are greater than the (similarity) threshold, they are connected, and all logs that are connected between each other are grouped into one group. For example, the threshold can be 80%, and those exceeding 80% are grouped into one group.
[0093] Step 24: Based on the similarity between the logs in each log group and each type of log in the classified final log classification set, the logs in the log group whose similarity meets a preset threshold are clustered into the corresponding final log classification set.
[0094] In this embodiment, the log data is obtained by preprocessing the original log data using a regular log template, thereby reducing the amount of log data. Regularized log data can reduce the corresponding number of calculations when calculating similarity, improve calculation efficiency, and regularized log data can improve the accuracy of similarity calculation when calculating similarity, and then use fewer log groups as a benchmark to perform final classification with the corresponding classified final log classification set, which can reduce the number of similarity calculations with various types of logs in the final log classification set, thereby improving the classification efficiency of log data.
[0095] See also Figure 5 , Figure 5 This is a flow chart of an embodiment of a log classification method provided by the present application. The method includes:
[0096] Step 51: Obtain log data.
[0097] The log data may be log data obtained by the technical solution in any of the above embodiments. For example, the log data in this embodiment is the pre-processed log data in the above implementation.
[0098] Step 52: Determine feature vectors of multiple logs in the log data.
[0099] The log data includes characteristic fields.
[0100] In some embodiments, see Figure 6 , step 52 may be the following process:
[0101] Step 521: Determine the characteristic fields of the log data.
[0102] Step 522: Determine a feature vector corresponding to each log according to the feature fields and text features of each log in the log data.
[0103] Step 53: Determine the vector edit distance between the multiple logs according to the feature vectors of the multiple logs.
[0104] In this embodiment, the log feature fields of multiple logs in the log data can be calculated and the statistical method of text frequency can be used to extract the feature words in each log. The more frequently a word appears in the log, the more it can represent the log feature, such as the application name. The higher the frequency of specific anomalies, the more it can represent that the feature appearing in the log is abnormal.
[0105] Based on the TF-IDF (term frequency–inverse document frequency) algorithm, it is used to evaluate the importance of a word to a document set or a document in a corpus. The importance of a word increases in direct proportion to the number of times it appears in a document, but it decreases in inverse proportion to the frequency of its appearance in the corpus.
[0106] It can be expressed by the following formula:
[0107] Among them, TF w Represents the frequency of term w.
[0108] Features that can be extracted based on the log itself include: word count, number of non-repeated occurrences, length, number of stop words, number of punctuation marks, number of capitalized words, number of title words, and average word length.
[0109] The word count indicates the number of words in each line of log text.
[0110] The number of unique words refers to the number of words that appear only once in each line of text.
[0111] The length indicates the length of each line of text and how much storage space it occupies.
[0112] The number of stop words represents statistics: the number of words such as between, but, about, very, etc.
[0113] The number of punctuation marks indicates the number of punctuation marks contained in each line of text.
[0114] The number of uppercase words means counting the number of uppercase words.
[0115] The number of title words refers to the number of words whose first letter is capitalized and the rest of the letters are lowercase.
[0116] The average length of words is the average length of each word in each line of text.
[0117] Then extract the feature vector of the feature words. For example, suppose all the feature word sets are S = {fw1, fw2, ..., fwn}, and they form an n-dimensional vector space in sequence. For multiple logs L and their feature field set WL in the log data, the feature vector is defined as: VL = [v1, v2, ..., vn]; where: vi = 1, fwi∈WL, i = 1, 2, ..., n; vi = 0,
[0118] The eigenvector of each log in the log data is calculated. After the eigenvector is calculated, one of the logs can be expressed as: L1 = [1, 0, 0, 1, 0, 1].
[0119] Then the vector edit distance between the multiple logs is determined based on the feature vectors of the multiple logs.
[0120] The following is an explanation of the string edit distance:
[0121] String edit distance is a method for calculating the similarity between strings. Given two strings S and T, the set of deletion, insertion, and replacement operations required to convert S to T is called the edit path from S to T. The shortest edit path is called the edit distance between strings S and T. The operation set contains three operations: deleting a node, inserting a node, and replacing the labels of two nodes. a→Λ represents deleting node a, and Λ represents an empty node; Λ→b represents inserting node b; and a→b represents replacing the label of node a with the label of node b.
[0122] Each of the three operations mentioned above has a cost, and the cost functions λ(a→Λ), λ(Λ→b), and λ(a→b) represent the cost of deleting node a, the cost of inserting node b, and the cost of replacing the label of node a with the label of node b. The cost is a non-negative real number. The value of the cost function is basically customized by the user.
[0123] To convert string S into string T, we need to go through {e1 e2 e3 e4...en} operations, where each ei (i = 1, 2, ..., n) is an operation. Let E = {e1 e2 e3 e4...en} be a continuous sequence of operations. Therefore, the total cost of converting S into T through the operation set n E is λ(E) = ∑λ(ei) = λ(e1) + i = 1λ(e2) + ... + λ(en). From this formula, we can calculate the cost of converting S into T through E. Then the edit distance from S to T can be defined as: stringdist(S, T) = min{λ(E)|E is one of the paths}.
[0124] It can be seen that for characters S and T, the editing path required to convert S into T is positively correlated with the length of S and T. The longer S and T are, the longer it takes to calculate the editing path and the lower the efficiency.
[0125] Dynamic programming can be used to calculate the edit distance between two characters. For characters S and T, the time complexity can be quickly calculated to be O(|S|×|T|). It takes a lot of time to calculate irrelevant text areas, and the time efficiency is relatively low. The feature vector method proposed in this embodiment can extract feature fields from S and T, and then convert the feature fields to 0 and 1, and obtain feature vectors again, which can greatly reduce the length of S and T. Since the length of S and T is greatly reduced, the time for calculating the edit distance can be greatly reduced, thereby greatly improving the calculation efficiency.
[0126] Step 54: Cluster the multiple logs according to the vector edit distance between the multiple logs to obtain multiple log groups.
[0127] The feature vector is converted into a constant character, and brought into the edit distance algorithm to calculate the edit distance. In this embodiment, for two different logs, their edit distances are first calculated to determine the similarity between the two. If the similarity is greater than a threshold, they are connected, and all logs that are connected between each other are grouped into one group. For example, the threshold can be 80%, and those exceeding 80% are grouped into one group.
[0128] Step 55: Cluster the logs in each log group into the corresponding final log classification set according to the similarity between the logs in each log group and the logs of each type in the corresponding classified final log classification set.
[0129] In some embodiments, see Figure 7 , step 55 may be the following process:
[0130] Step 551: Obtain the source of each log group in the plurality of log groups.
[0131] In some embodiments, the source of each log group can be defined as the application that generates the log, such as application A, application B, and application C.
[0132] Step 552: Obtain a final log classification set that is the same as the source of each log group.
[0133] The final log classification set is also differentiated by application.
[0134] Step 553: Calculate the vector edit distance between the logs in each log group and each type of log in the final log classification set, and cluster the logs in each log group into a target log classification set, which is the final log classification set with the smallest vector edit distance with the corresponding logs in the log group.
[0135] Specifically, the logs that need to participate in the calculation are vectorized to obtain the corresponding feature vectors. For example, log A in the target log group is vectorized to obtain the first feature vector. Log B in the target type log is vectorized to obtain the second feature vector.
[0136] According to the calculation method of any of the above embodiments, the vector edit distance of the first feature vector and the second feature vector is calculated. In this form, the vector edit distance of each log in each log group is calculated with each type of log in the log classification set to obtain a plurality of vector edit distances. That is, each log in each log group corresponds to a plurality of vector edit distances.
[0137] Determine the minimum vector edit distance from several vector edit distances. Cluster the log into the corresponding log class of the target log classification set corresponding to the minimum vector edit distance. It should be noted that the smaller the edit distance, the greater the similarity between the two. The feature vector method proposed in this embodiment can extract the log and convert it into a vector, and then obtain the edit distance again, so that the time for calculating the edit distance can be greatly reduced, thereby greatly improving the calculation efficiency.
[0138] For a log of an application, for each specific log line, the vector edit distance is calculated with the specific log line of the final classification set of the application.
[0139] In one application scenario, the calculated vector edit distance is used as the score of the corresponding similarity. If the calculated score S1 between the logs exceeds the threshold F, the two logs are classified into one category, and the cumulative number of the log is increased by 1.
[0140] If the score S of any specific log line in each type of log in the final log classification set is lower than the threshold F, it is equivalent to a new type. A row of records is added to the final log classification set, and the cumulative count is 1.
[0141] The above process is repeated until all log lines are executed and the log classification set D'i = [x1, x2...xN] is calculated.
[0142] By calculating the complete log, we can get the clustering result P'i = [D1, D2...DN] of all applications.
[0143] In this embodiment, the log data is obtained by preprocessing the original log data using a regular log template, thereby reducing the amount of log data. Regularized log data can reduce the corresponding number of calculations when calculating similarity, improve calculation efficiency, and regularized log data can improve the accuracy of similarity calculation when calculating similarity, and then use fewer log groups as a benchmark to perform final classification with the corresponding classified final log classification set, which can reduce the number of similarity calculations with various types of logs in the final log classification set, thereby improving the classification efficiency of log data.
[0144] The method further includes: obtaining the increment of each type of log in the final log classification set within a preset time; if the increment exceeds a preset value, issuing an early warning prompt.
[0145] See also Figure 8 , Figure 8 1 is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. The electronic device 80 includes a processor 81 and a memory 82 coupled to the processor 81. The memory 82 stores a computer program. The processor 81 is used to execute the computer program to implement the following method:
[0146] Obtain log data; cluster the log data according to the similarity between multiple logs in the log data to obtain multiple log groups; cluster the logs in the log group whose similarity meets a preset threshold into the corresponding final log classification set according to the similarity between the logs in each log group and each type of log in the classified final log classification set.
[0147] It can be understood that the processor 81 is also used to execute a computer program to implement a method in any of the above embodiments. Please refer to any of the above technical solutions for details, which will not be elaborated here.
[0148] See also Fig. 9 , Fig. 9 1 is a schematic diagram of a computer readable storage medium according to an embodiment of the present application. The computer readable storage medium 90 stores a computer program 91. When the computer program 91 is executed by a processor, the following method is implemented:
[0149] Obtain log data; cluster the log data according to the similarity between multiple logs in the log data to obtain multiple log groups; cluster the logs in the log group whose similarity meets a preset threshold into the corresponding final log classification set according to the similarity between the logs in each log group and each type of log in the classified final log classification set.
[0150] It can be understood that when the computer program 91 is executed by the processor, it is also used to implement the method of any of the above embodiments. Please refer to any of the above technical solutions for details, which will not be repeated here.
[0151] In one application scenario, combined with Fig.10 and Fig.11 To explain:
[0152] In order to verify the effectiveness of the algorithm, this application implements a verification tool. The verification tool first downloads the full log files of the business system and classifies the logs using any of the above-mentioned technical solutions of this application to prove that the technical solution of this application can greatly improve the computing efficiency and that clustering the logs is effective.
[0153] First, the business logs of the virtual card batch system, virtual card interface system, data platform service system, external data source access service system, and payment deduction routing system are selected from the business system as experimental data sources, and the relevant technical solutions and the technical solutions of this application are compared respectively.
[0154] like Fig.10 The figure shows the business log test results of 5 core business systems using relevant technical solutions. The average log length is 180-310, and the calculation speed is about 60,000 logs per second. The log length and processing speed are roughly negatively correlated, that is, as the log length increases, the processing speed becomes slower.
[0155] like Fig.11 As shown in the figure, the business log test results of 5 core business systems using the technical solution of this application are shown, among which the number of log entries and Fig.10 The difference is that after the conversion of the technical solution of this application, the length of the original log is reduced to 1 / 10, so the calculation speed can reach about 300,000 logs per second, which greatly improves the processing speed and greatly improves the efficiency of full log clustering. It also reflects that the log length is negatively correlated with the clustering speed. The shorter the length, the faster the processing speed. However, it is far better than the related technology, and the clustering speed is greatly improved.
[0156] In one application scenario, all application log data streams are obtained based on Flink, and the log cluster Kafka consumes all logs. For example, the original content of the log is collected to obtain data as follows:
[0157] 1.2021-10-21 11:32:01.579[vayhapi][application w api]ERROR[http-nio-8080-exec-74]SignChecker: Determine whether it is a student, ID number error: null exception.
[0158] 2.2021-10-21 11:33:03.579[vayhapi][application w api]ERROR[http-nio-8080-exec-74]SignChecker: Determine whether it is a student, ID number error: null exception.
[0159] 3.2021-10-21 11:33:23.579 [vayhapi][application w api]ERROR [http-nio-8080-exec-74] AfEventUploadAspect: Anti-fraud 2.0 report get parameter message id: msgid error: ReadTimeOut timeout exception.
[0160] 4.2021-10-21 11:33:13.579[crm][Customer Relationship System]INFO[http-nio-8080-exec-74]CouponService: The user received 1 coupon for the 618 promotion.
[0161] Then, we use Flink to clean the original log data in real time, remove the wrong data, keep only the useful logs, and remove the information that does not need to be displayed in each log. The cleaned logs are as follows:
[0162] 1.2021-10-21 11:32:01.579 [vayhapi][application w api] ERROR SignChecker: Determine whether it is a student, ID number error: null exception.
[0163] 2.2021-10-21 11:33:03.579 [vayhapi][application w api] ERROR SignChecker: Determine whether it is a student, ID number error: null exception.
[0164] 3.2021-10-21 11:33:21.579 [vayhapi][application w api]ERRORAfEventUploadAspect: Anti-fraud 2.0 reporting to get parameters message id: msgid error: ReadTimeOut timeout exception.
[0165] 4.2021-10-21 11:33:13.579[crm][Customer Relationship System]INFO CouponService: The user received 1 coupon for the 618 promotion.
[0166] Then put the cleaned logs into the queue to be processed, waiting for the core thread to process them.
[0167] Then clustering is performed according to the technical solution of any of the above embodiments. Logs after clustering:
[0168] 1.2021-10-21 11:32:01.579[vayhapi][application w api]ERROR SignChecker: Determine whether it is a student, ID card number error: null exception; cumulative number of entries: 2.
[0169] 2.2021-10-21 11:33:21.579 [vayhapi][application w api] ERRORAfEventUploadAspect: Anti-fraud 2.0 report obtain parameter message id: msgid error: ReadTimeOut timeout exception; cumulative number of items: 1.
[0170] 3.2021-10-21 11:33:13.579[crm][Customer Relationship System]INFO CouponService: The user received 1 coupon for the 618 promotion; the total number of coupons: 1.
[0171] The calculated results are then grouped by application, and each application data is sent to Elasticsearch for storage, which is used for subsequent viewing of log volume rankings and year-on-year and month-on-month alerts. The specific grouping can be as follows:
[0172] 1. Application: [vayhapi] [application w api] Log type: ERROR SignChecker: Determine whether it is a student, ID number error: null exception; cumulative number of entries: 2.
[0173] 2. Application: [vayhapi][application w api] Log type: ERROR AfEventUploadAspect: Anti-fraud 2.0 report to obtain parameters message id: msgid error: ReadTimeOut timeout exception; cumulative number of entries: 1.
[0174] 3. Application: [crm] [Customer Relationship System] Log type: INFO CouponService: The user received 1 coupon for the 618 promotion; the total number of items: 1.
[0175] According to the number of changes in each application log, configure the alarm threshold, alarm type, and alarm period, and send an alarm notification when the threshold is triggered. For example, the alarm rules are as follows:
[0176] 1. Application w alarm - query interface abnormal year-on-year change rate [determine whether it is a student, ID number is wrong].
[0177] 2. Customer relationship system alert - number of coupons received by users - month-on-month ratio in the past hour.
[0178] When an alarm event is received by the alarm channel, the person in charge of the business system will be notified by phone, SMS, WeChat, or email according to the alarm level. If a WeChat notification is received:
[0179] 1. Compared with the previous day, the application w log alarm-query interface abnormality increased by 50% year-on-year [determining whether it is a student, the ID number is wrong] in the last five minutes, and the number of logs in the last five minutes: 2.
[0180] 2. Customer relationship system alert - the number of users who received coupons - [users received coupons for the 618 promotion] - decreased by 80% month-on-month in the past hour.
[0181] In other embodiments, a large visualization screen is used to display the ranking of the number of log entries for each application, and the aggregated logs are queried in real time to display each application log, arranged from large to small.
[0182] For example, the current 5-minute log ranking of application w:
[0183] 1.2021-10-21 11:32:01.579[vayhapi][application w api]ERROR SignChecker: Determine whether it is a student, ID number error: null exception; cumulative number of entries: 98757.
[0184] 2.2021-10-21 11:33:21.579 [vayhapi][application w api]ERRORAfEventUploadAspect: Anti-fraud 2.0 report obtain parameter message id: msgid error: ReadTimeOut timeout exception; cumulative number of items: 8765.
[0185] In the above flink real-time calculation, the flink streaming computing engine is used to implement it: first create a FlinkKafkaConsumer010 object to consume indicator data from kafka; then add the log object as a data source to the flink execution environment to obtain DataStreamSource; call flink's flatMap method to clean the kafka message data into log data that can be calculated, and the data type is Tuple2<String,Map> . Call the assignTimestampsAndWatermarks method of the flink object to uniformly use the timestamp field timestamp as the time watermark. All time-related calculations are based on this field of the indicator. Call the keyBy(0) method of the flink object to split the data using the application name field of each indicator to ensure that the data calculated in each data stream is the data of the same application. Call the timeWindow(Time.minutes(5), Time.minutes(1)) method of the flink object to create a sliding time window based on the application stream data. The window length is 5 minutes and the window sliding length is 1 minute. That is, the data within 5 minutes of the application will be aggregated into a window. After the calculation is completed, the window time will move forward by one minute. Call the apply() method of the flink object to implement the accumulation of the number of each log according to the log clustering method and the string similarity method in the method. Call the map() method of the flink object to encapsulate the data in the method and then send it to elasticsearch.
[0186] In summary, the log classification method, electronic device and computer-readable storage medium provided by the present application obtain log data; cluster the log data according to the similarity between multiple logs in the log data to obtain multiple log groups; cluster the logs in each log group into the corresponding final log classification set according to the similarity between the logs in each log group and the various logs in the corresponding final log classification set. The log data is first clustered into multiple log groups, and then the final classification is performed with the corresponding final log classification set based on fewer log groups. This can reduce the number of similarity calculations with various logs in the final log classification set, thereby improving the classification efficiency of the log data.
[0187] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only illustrative, for example, the division of the modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.
[0188] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0189] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0190] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made according to the description and drawings of the present application, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.
Claims
1. A log classification method, characterized in that: The method comprises: Get log data; Clustering the log data according to similarities between multiple logs in the log data to obtain multiple log groups; For any log group among the multiple log groups, obtaining a final log classification set having the same source as the log group; According to the similarity between the logs in the log group and each type of log in the final log classification set, the logs in the log group whose similarity meets a preset threshold are clustered into the corresponding final log classification set.
2. The method according to claim 1, characterized in that The obtaining of log data includes: Get the original log data; The original log data is preprocessed to obtain the log data; wherein the data volume of the log data is smaller than the data volume of the original log data, and the log data includes a plurality of logs and a quantity corresponding to each of the logs.
3. The method according to claim 2, characterized in that The original log data includes a plurality of original logs; The preprocessing of the original log data to obtain the log data includes: Processing multiple original logs in the original log data according to preset rules to obtain multiple rule logs; The same logs in the multiple rule logs are merged to obtain the log data.
4. The method according to claim 3, characterized in that The processing of the multiple original logs in the original log data according to the preset rules to obtain multiple rule logs includes: Extracting log attributes of a plurality of original logs in the original log data, wherein the log attributes include at least one of log time, log source, log level, log thread, log class, log event, and log abnormal state; According to the preset rule log template and the log attribute of each of the logs, a plurality of rule logs are correspondingly generated.
5. The method according to claim 1, characterized in that The step of clustering the log data according to the similarities between the multiple logs in the log data to obtain multiple log groups includes: Determine feature vectors of the plurality of logs in the log data; Determine a vector edit distance between the plurality of logs according to the feature vectors of the plurality of logs; The multiple logs are clustered according to the vector edit distances between the multiple logs to obtain multiple log groups.
6. The method according to claim 5, characterized in that The log data includes a characteristic field; The determining the feature vectors of the plurality of logs in the log data comprises: According to the characteristic fields and text features of each of the logs in the log data, a characteristic vector corresponding to each of the logs is determined.
7. The method according to claim 5, characterized in that The method further comprises: Get the source of the log group; The step of clustering the logs in the log group whose similarity meets a preset threshold into the corresponding final log classification set according to the similarity between the logs in the log group and the various logs in the final log classification set includes: Calculate the vector edit distance between the logs in the log group and each type of log in the final log classification set, and cluster the logs in the log group into a target log classification set, where the target log classification set is the final log classification set with the smallest vector edit distance with the corresponding logs in the log group.
8. The method according to claim 1, characterized in that The method further comprises: Obtaining the increment of each type of log in the final log classification set within a preset time; If the increment exceeds a preset value, an early warning prompt is issued.
9. An electronic device, characterized in that: The electronic device comprises a processor and a memory coupled to the processor, wherein a computer program is stored in the memory, and the processor is configured to execute the computer program to implement the method according to any one of claims 1 to 8.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the computer program implements the method according to any one of claims 1 to 8.
