Log simplification method, device, electronic device and computer readable medium
By clustering and deduplication of system logs, the redundancy problem of log duplication is solved, the fault location efficiency is improved, and the operation and maintenance work is simplified.
Patent Information
- Application Number
- CN202010864698.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-08-25
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2040-08-25
AI Technical Summary
The data logs generated during the system operation contain many duplicate content, resulting in waste of memory space and increased complexity of operation and maintenance fault location.
By clustering all logs in the first log collection, multiple types of logs are obtained, where each type of log includes a log with a similarity greater than the preset threshold, and a log is obtained from each type of log of the multi-class log to form The second log collection, the number of logs in the second log collection is the same as the number of categories of multi-class logs, and the duplicate logs are removed.
Reduces the number of logs, improves the efficiency of fault location, simplifies the workflow of operation and maintenance personnel, and reduces the difficulty of checking.
Smart Images

Figure CN112100025B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of big data technology, and in particular to a log simplification method, device, electronic device and computer-readable medium. Background Art
[0002] With the advent of the big data era, the system will generate a large amount of data logs during operation, which will help operation and maintenance personnel locate system failures. Currently, the data logs generated by the system contain a lot of duplicate content. Redundant data logs waste memory space and increase the complexity of fault location for operation and maintenance personnel. Summary of the invention
[0003] The purpose of the embodiments of the present application is to provide a log simplification method, device, electronic device and computer-readable medium to solve the problem of log duplication and redundancy. The specific technical solution is as follows:
[0004] In a first aspect, a log simplification method is provided, the method comprising:
[0005] Performing a clustering operation on all logs in the first log set to obtain multiple types of logs, wherein each type of log includes logs whose similarity is greater than a preset threshold;
[0006] A log is obtained from each type of log of the multiple types of logs to obtain a second log set, wherein the number of logs in the second log set is the same as the number of categories of the multiple types of logs.
[0007] Optionally, performing a clustering operation on all logs in the first log set includes obtaining each type of log in the following manner:
[0008] When there is an unmarked log in the first log set, obtaining the unmarked log as a first target log;
[0009] Determine a similarity between the first target log and a candidate log in the first log set, wherein the candidate log is an unmarked log in the first log set except the first target log;
[0010] If there is a second target log in the candidate logs and the similarity between the second target log and the first target log is greater than the preset threshold, marking the second target log in the first log set, and treating the first target log and the second target log as one type of log;
[0011] When there is no second target log in the candidate logs and the similarity between the second target log and the first target log is greater than the preset threshold, the first target log is regarded as a type of log.
[0012] Optionally, the first log set includes a plurality of logs that are sequentially arranged and are initially unmarked, wherein obtaining the unmarked log as the first target log includes:
[0013] In the case that this is the first time to obtain logs, the first log in the first log set is obtained as the first target log;
[0014] When this acquisition is not the first log acquisition, the unmarked log in the first log set that is located after the third target log is acquired as the first target log, wherein the third target log is the log acquired from the first log set last time.
[0015] Optionally, after obtaining the second log set, the method further includes:
[0016] determining the number of the second target logs;
[0017] generating a target chart according to the number of the second target logs in the second log set and the first target log;
[0018] The target graph is output.
[0019] Optionally, determining the similarity between the first target log and the candidate logs in the first log set includes:
[0020] Obtaining a first text vector of the first target log and a second text vector of a candidate log in the first log set;
[0021] determining a cosine value of an angle between the first text vector and the second text vector;
[0022] When the cosine value of the angle is greater than a preset threshold, it is determined that the first target log is similar to the candidate log.
[0023] Optionally, obtaining a first text vector of the first target log includes:
[0024] Performing a word segmentation operation on the first target log to obtain a plurality of first words;
[0025] Performing word vector processing on the first vocabulary to obtain a first word vector;
[0026] An average value of a plurality of the first word vectors in the first target log is determined, and the average value is used as the first text vector.
[0027] In a second aspect, a log simplification device is provided, the device comprising:
[0028] A clustering module, used for performing a clustering operation on all logs in the first log set to obtain multiple types of logs, wherein each type of log includes logs whose similarity is greater than a preset threshold;
[0029] The acquisition module is used to acquire a log from each type of log of the multiple types of logs to obtain a second log set, wherein the number of logs in the second log set is the same as the number of categories of the multiple types of logs.
[0030] Optionally, the clustering module includes:
[0031] an acquiring unit, configured to acquire, when there is an unmarked log in the first log set, the unmarked log as a first target log;
[0032] a determining unit, configured to determine a similarity between the first target log and a candidate log in the first log set, wherein the candidate log is an unmarked log in the first log set except the first target log;
[0033] a marking unit, configured to mark the second target log in the first log set and treat the first target log and the second target log as one type of log if there is a second target log in the candidate logs and the similarity between the second target log and the first target log is greater than the preset threshold;
[0034] as a unit, configured to treat the first target log as a type of log when there is no second target log in the candidate logs and the similarity between the second target log and the first target log is greater than the preset threshold.
[0035] In a third aspect, an electronic device is provided, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;
[0036] Memory, used to store computer programs;
[0037] The processor is used to implement any of the method steps described above when executing a program stored in the memory.
[0038] In a fourth aspect, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, any of the method steps described above is implemented.
[0039] Beneficial effects of the embodiments of the present application:
[0040] The embodiments of the present application provide a log simplification method, device, electronic device, and computer-readable medium. The present application performs a clustering operation on all logs in a first log set to obtain multiple types of logs, wherein each type of log includes logs whose similarity is greater than a preset threshold, and then obtains a log from each type of log in the multiple types of logs to obtain a second log set, wherein the number of logs in the second log set is the same as the number of categories of the multiple types of logs. The present application obtains a second log set, which includes logs of various categories, and the number of logs in each category is 1, and no longer contains duplicate logs, thereby reducing the number of logs. Operation and maintenance personnel locate faults through the simplified logs, thereby improving the efficiency of fault location.
[0041] Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0043] Figure 1 A flow chart of a log simplification method provided in an embodiment of the present application;
[0044] Figure 2 A flow chart of a method for obtaining each type of log provided in an embodiment of the present application;
[0045] Figure 3 A schematic diagram of obtaining a target chart provided in an embodiment of the present application;
[0046] Figure 4 A flow chart of a log simplification method provided in an embodiment of the present application;
[0047] Figure 5 A schematic diagram of the structure of a log simplification device provided in an embodiment of the present application;
[0048] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0049] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0050] An embodiment of the present application provides a log simplification method, which can be applied to a processor to simplify a large number of duplicate logs.
[0051] The following will describe in detail a log simplification method provided by an embodiment of the present application in combination with a specific implementation method. Figure 1 As shown, the specific steps are as follows:
[0052] Step 101: performing a clustering operation on all logs in a first log set to obtain multiple types of logs.
[0053] Each type of log includes logs whose similarity is greater than a preset threshold.
[0054] In the embodiment of the present application, the system generates a large number of logs during operation, and these logs are included in the first log set. The processor obtains all logs in the first log set and performs clustering operations on the logs to obtain multiple types of logs. Each type of log includes one log or multiple logs. For a type of log including multiple logs, the similarity of multiple logs in the type of log is greater than a preset threshold.
[0055] Step 102: Obtain a log from each type of log of the multiple types of logs to obtain a second log set.
[0056] The number of logs in the second log set is the same as the number of categories of the multi-category logs.
[0057] In an embodiment of the present application, the processor obtains a log from each of the multiple types of logs, and organizes the obtained logs into a second log set. Since the second set contains a log from each of the multiple types of logs, each log in the second log set has a different category, and the number of logs is the same as the number of log categories. Therefore, the number of logs in the second log set is the same as the number of categories of the multiple types of logs.
[0058] As an optional implementation, Figure 2 As shown, clustering all logs in the first log set includes obtaining each type of log in the following manner:
[0059] Step 201: when there is an unmarked log in the first log set, the unmarked log is obtained as a first target log.
[0060] In an embodiment of the present application, if the semantic similarity of two logs reaches a preset threshold, one of the logs will be marked and the other will not be marked. Therefore, if there are multiple logs in the first log set whose semantic similarity reaches the preset threshold, one log will not be marked and the other logs will be marked. Exemplarily, it can be marked as a duplicate log. If there is an unmarked log in the first log set, the processor obtains an unmarked log as the first target log.
[0061] Step 202: Determine whether the similarity between the first target log and the candidate log is greater than a preset threshold.
[0062] The candidate logs are unmarked logs in the first log set except the first target log.
[0063] In an embodiment of the present application, the first log set includes marked logs and unmarked logs, and may also include marked logs. Among them, the unmarked logs include the first target log and the candidate logs other than the first target log, and the processor determines the similarity between the first target log and the candidate logs in the first log set, and determines whether the similarity is greater than a preset threshold. If the processor determines that the similarity between the first target log and the candidate log is greater than the preset threshold, step 203 is executed; if the processor determines that the similarity between the first target log and the candidate log is not greater than the preset threshold, step 204 is executed.
[0064] Step 203: Mark the second target log in the first log set, and treat the first target log and the second target log as one type of log.
[0065] In an embodiment of the present application, if the processor determines that the similarity between the first target log and the candidate log is greater than a preset threshold, the second target log whose similarity with the first target log is greater than the preset threshold is marked in the candidate log. Since the similarity between the first target log and the second target log is greater than the preset threshold, the first target log and the second target log are regarded as the same type of logs.
[0066] Exemplarily, the logs in the first log set are Arr = [A, B, C, A1, D, A2, B1], where Arr is the first log set, and A, B, C, A1, D, A2, and B1 are logs in the first log set. If all the logs in the current Arr are unmarked, and the first target log is A, then the candidate logs include B, C, A1, D, A2, and B1. If the processor determines that the similarity between A and A1 and A2 is higher than a preset threshold, then the processor uses A1 and A2 as the second target logs and marks A1 and A2. A, A1, and A2 are logs of the same type.
[0067] If A1 and A2 in the current Arr are marked, and the first target log is B, then the candidate logs include A, C, D, and B1. The processor determines that the similarity between B and B1 is higher than the preset threshold, then the processor takes B1 as the second target log and marks B1. B and B1 are the same type of logs.
[0068] Step 204: The first target log is regarded as a type of log.
[0069] In an embodiment of the present application, if the processor determines that the similarity between the first target log and the candidate log is not greater than a preset threshold, it indicates that there is no log in the candidate log that is repeated with the first target log, and the marked log in the first log set is not repeated with the first target log, then the first target log is treated as a separate type of log.
[0070] Exemplarily, the logs in the first log set are Arr = [A, B, C, A1, D, A2, B1], where Arr is the first log set, and A, B, C, A1, D, A2, B1 are logs in the first log set. If A1, A2, and B1 in the current Arr are marked, and the first target log is C, then the candidate logs include A, B, and D. If the processor determines that there is no log in the candidate log whose similarity with C is greater than a preset threshold, the processor treats C as a separate type of log.
[0071] As an optional implementation, the first log set includes a plurality of logs that are sequentially arranged and initially unmarked, wherein obtaining the unmarked log as the first target log includes: when this acquisition is the first acquisition of logs, obtaining the first log in the first log set as the first target log; when this acquisition is not the first acquisition of logs, obtaining the unmarked log in the first log set that is located after the third target log as the first target log. The third target log is the log obtained from the first log set last time.
[0072] In the embodiment of the present application, logs are generated one by one during the operation of the system, and the processor obtains multiple logs in the order in which the logs are generated. Therefore, the initial first log set includes multiple logs that are arranged in sequence and are initially unmarked. If the processor obtains the first target log for the first time, the logs in the first log set are not marked, and the processor uses the first log received first as the first target log. If the processor does not obtain the first target log at one time, the processor uses the marked logs in the first log set and the logs that are repeated with the marked logs as the third log, and the processor uses the logs after the third log as the first target log.
[0073] Exemplarily, all logs in the first log set are Arr = [A, B, C, A1, D, A2, B1], where Arr is the first log set, and A, B, C, A1, D, A2, B1 are logs in the first log set. If A1, A2, and B1 in the current Arr are marked, and the first target log is C, then it is not the first time that the processor obtains the first target log. The processor uses the marked logs A1, A2, B1, the log A repeated with A1, and the log B repeated with B1 as the third target log, and the processor uses the log C after the third log as the first target log.
[0074] This application simplifies the logs, allowing operation and maintenance personnel to analyze the deduplicated and simplified logs, quickly access key information, reduce the difficulty of access, and improve work efficiency. This application has a wide range of applications and is applicable to logs generated by various application systems, middleware, databases, etc.
[0075] As an optional implementation, after obtaining the second log set, the method further includes: determining the number of second target logs; generating a target chart according to the number of second target logs in the second log set and the first target log; and outputting the target chart.
[0076] In an embodiment of the present application, after obtaining the second log set, the processor determines the number of second target logs, and then generates a chart based on the number of second target logs in the second log set and the first target log. The process of the processor determining the number of second target logs is as follows: when the processor performs a clustering operation on all logs in the first log set, each time a second target log is determined, the number of second target logs is increased by 1. After obtaining the second log set, the number of second target logs is no longer increased by 1, and the result at this time is the final number of second target logs. The processor can use JavaScript chart libraries such as Chart.js, Google Charts, and Highcharts to generate target charts, and the present application does not specifically limit the method of generating target charts.
[0077] The processor generates a target chart through HTML (HyperTextMarkup Language) or Echarts according to the name of the first target log and the number of the second target log. The target chart includes but is not limited to a pie chart, a list, etc. Figure 3 As shown, Figure 3 is a schematic diagram of a target chart. The pie chart includes three first target logs, namely (a), (b) and (c), and the number of second target logs corresponding to (a), (b) and (c) are 28, 34 and 17 respectively. The proportion of the first target log can also be determined according to the ratio of the number of first target logs to the total number of logs.
[0078] In this application, the generated target chart is clear and intuitive, which is convenient for operation and maintenance personnel to determine key information based on the target chart, without having to consult a large number of logs, thereby improving work efficiency. In addition, the target chart can provide comprehensive information, so that the operation and maintenance personnel have a comprehensive understanding of all logs, reducing the difficulty of analysis for the operation and maintenance personnel. The target chart can be used repeatedly, without the need for the operation and maintenance personnel to re-analyze the logs when they need to determine the information, saving the operation and maintenance personnel's time.
[0079] As an optional implementation, determining the similarity between the first target log and the candidate logs in the first log set includes: obtaining a first text vector of the first target log and a second text vector of the candidate logs in the first log set; determining a cosine value of an angle between the first text vector and the second text vector; and determining that the first target log is similar to the candidate log when the cosine value of the angle is greater than a preset threshold.
[0080] In an embodiment of the present application, the processor obtains the first text vector of the first target log and the second text vector of the candidate log in the first log set, and then calculates the cosine value of the angle between the first text vector and the second text vector, and determines the similarity between the first target log and the candidate log according to the cosine value of the angle. Among them, the cosine value of the angle is proportional to the similarity, and the larger the cosine value of the angle, the higher the similarity. Therefore, when the processor determines that the cosine value of the angle is greater than the preset threshold, the first target log is similar to the candidate log.
[0081] Exemplarily, let the first text vector be a, and the second text vector be b, |a| and |b| are the moduli of the first text vector and the second text vector, respectively, cos<a,b> That is, the cosine value between the first text vector and the second text vector, and the cosine value of the angle between the first text vector and the second text vector
[0082] As an optional implementation, obtaining the first text vector of the first target log includes: performing a word segmentation operation on the first target log to obtain multiple first words; performing word vector processing on the first words to obtain a first word vector; determining an average value of the multiple first word vectors in the first target log, and using the average value as the first text vector.
[0083] In an embodiment of the present application, if the processor wants to obtain the first text vector of the first target log, it first performs a word segmentation operation on the first target log to obtain multiple first words after the word segmentation operation, and then performs word vector processing on the first word through word2Vec to obtain the first word vector. The processor determines the average value of the multiple first word vectors in the first target log and uses the average value as the first text vector. The generation process of the second text vector of the candidate log is the same as the generation process of the first text vector, and this application will not elaborate on it. The word segmentation tool for the word segmentation operation can be jieba, Ansj, etc., and this application does not specifically limit the word segmentation tool.
[0084] As an optional implementation, before the processor performs word vector processing on the first vocabulary through word2Vec, for the second vocabulary that does not have a word vector in the first target log, the processor can first obtain the word vector of the second vocabulary. At this time, all the vocabulary in the first target log has a corresponding word vector, and the vocabulary with the corresponding word vector is used as the first vocabulary. Then, the processor performs word vector processing on the first vocabulary in the first target log through word2Vec.
[0085] Exemplarily, the processor may train the second vocabulary through a deep neural network to finally obtain a word vector of the second vocabulary. The deep neural network includes a Skip-gram model, a CBOW model, or a DNN model.
[0086] The present application expands the number and range of first words with corresponding word vectors through a deep neural network, making the first text vector more accurate.
[0087] As an optional implementation, the processor determines that the first target log is similar to the candidate log, and can also use the following methods: benchmark method, word shift distance, Smooth Inverse Frequency, TF-IDF (term frequency-inverse document frequency), edit distance calculation, Jaccard coefficient calculation and pre-trained encoder.
[0088] Optionally, the present application embodiment also provides a processing flow of a log simplification method, such as Figure 4 As shown, the specific steps are as follows:
[0089] Step 401: Obtain logs and generate a first log set.
[0090] Step 402: traverse the first log set.
[0091] Step 403: Determine whether the first target log is a marked log. If it is determined not to be a marked log, execute step 404. If it is determined to be a marked log, return to step 402.
[0092] Step 404: Add the second log set.
[0093] Step 405: Determine whether the remaining logs have been traversed. If the traversal has been completed, return to step 402; if the traversal has not been completed, execute step 406.
[0094] Step 406: Determine whether the remaining log is a marked log. If it is determined that it is not a marked log, execute step 407. If it is determined that it is a marked log, return to step 405.
[0095] Step 407: Determine whether the similarity between the first target log and the candidate log is greater than a preset threshold. If so, execute step 408; if not, return to step 405.
[0096] Step 408: Mark as the second target log, and increase the number of the second target log by 1.
[0097] Step 409: When the traversal of the first log set is finished, a summary chart is generated.
[0098] Exemplarily, the logs in the first log set are Arr=[A, B, C, A1, D, A2, B1], where Arr is the first log set, and A, B, C, A1, D, A2, and B1 are respectively logs in the first log set. If all logs in Arr are unmarked and the first target log is A, then the candidate logs include B, C, A1, D, A2, and B1. If the processor determines that A is an unmarked log, A is included in the second log set, and the processor traverses the remaining logs [B, C, A1, D, A2, B1]. If the processor determines that B is not a marked log, then the similarity between A and B is compared to see if it is greater than a preset threshold. If the processor determines that the similarity between A and B is not higher than the preset threshold, then the remaining logs are returned to see if C is a marked log; if the processor determines that C is not a marked log and the similarity between A and C is not higher than the preset threshold, then the remaining logs are returned to see if A1 is a marked log; if the processor determines that A1 is not a marked log, but the similarity between A and A1 is higher than the preset threshold, then A1 is marked as the second target log, the value of the second target log is increased by 1, and then the remaining logs are returned to see if D is a marked log. After traversing the remaining logs in the above manner, the processor obtains that the second target logs are A1 and A2, and the processor returns to traverse the first log set [B, C, A1, D, A2, B1].
[0099] The first target log is B. The processor determines that B is an unmarked log, so B is included in the second log set. The processor traverses the remaining logs [C, A1, D, A2, B1]. The processor determines that B is not a marked log, then compares whether the similarity between B and C is greater than the preset threshold. The processor determines that the similarity between B and C is not higher than the preset threshold, then returns the remaining logs to determine whether A1 is a marked log; the processor determines that A1 is a marked log, then returns the remaining logs to determine whether D is a marked log. The processor determines that D is not a marked log, and the similarity between D and B is not higher than the preset threshold, then returns the remaining logs to determine whether A2 is a marked log; the processor determines that A2 is a marked log, then returns the remaining logs to determine whether B1 is a marked log. The processor determines that B1 is not a marked log, but the similarity between B and B1 is higher than the preset threshold, then marks B1 as the second target log. After traversing the remaining logs in the above manner, the processor obtains that the second target logs are A1, A2 and B1, and the processor returns to the first log set traversal [C, A1, D, A2, B1].
[0100] The first target log is C. The processor determines that C is an unmarked log, so C is included in the second log set. The processor traverses the remaining logs [A1, D, A2, B1]. A1 is a marked log, so A1 is skipped and D is traversed. If the similarity between D and C is not higher than the preset threshold, A2 and B1 are traversed in turn. However, both A2 and B1 are marked logs, so A2 and B1 are skipped and the traversal is completed.
[0101] The second log set after the traversal is completed includes A, B, C, and D, among which A has two duplicate logs A1 and A2, and B has one duplicate log B1.
[0102] Based on the same technical concept, the embodiment of the present application also provides a log simplification device, such as Figure 5 As shown, the device comprises:
[0103] A clustering module 501 is used to perform a clustering operation on all logs in the first log set to obtain multiple types of logs, wherein each type of log includes logs whose similarity is greater than a preset threshold;
[0104] The acquisition module 502 is used to acquire a log from each type of log of the multiple types of logs to obtain a second log set, wherein the number of logs in the second log set is the same as the number of types of the multiple types of logs.
[0105] Optionally, the clustering module 501 includes:
[0106] an acquisition unit, configured to acquire the unmarked log as the first target log when there is an unmarked log in the first log set;
[0107] A determination unit, configured to determine a similarity between a first target log and a candidate log in the first log set, wherein the candidate log is an unmarked log in the first log set except the first target log;
[0108] a marking unit, configured to mark the second target log in the first log set if there is a second target log in the candidate log whose similarity with the first target log is greater than a preset threshold, and regard the first target log and the second target log as logs of the same type;
[0109] As a unit, it is used to treat the first target log as a type of log when there is no second target log in the candidate logs whose similarity with the first target log is greater than a preset threshold.
[0110] Optionally, the first log set includes a plurality of logs which are arranged in sequence and are initially unmarked, wherein the obtaining unit includes:
[0111] A first acquisition subunit is used to acquire the first log in the first log set as the first target log when the current acquisition is the first acquisition of logs;
[0112] The second acquisition subunit is used to acquire the unmarked log in the first log set after the third target log as the first target log when this acquisition is not the first log acquisition, wherein the third target log is the log acquired from the first log set last time.
[0113] Optionally, the device comprises:
[0114] A determination module, used to determine the number of second target logs;
[0115] A generating module, configured to generate a target chart according to the number of second target logs in the second log set and the first target log;
[0116] Output module, used to output the target chart.
[0117] Optionally, the determining unit includes:
[0118] A third acquisition subunit is used to acquire a first text vector of the first target log and a second text vector of the candidate log in the first log set;
[0119] A first determining subunit, used to determine a cosine value of an angle between a first text vector and a second text vector;
[0120] The second determining subunit is configured to determine that the first target log is similar to the candidate log when the cosine value of the angle is greater than a preset threshold.
[0121] Optionally, the third acquisition subunit includes:
[0122] A word segmentation submodule, used for performing a word segmentation operation on the first target log to obtain a plurality of first words;
[0123] A processing submodule, used for performing word vector processing on the first word to obtain a first word vector;
[0124] The determination submodule is used to determine an average value of multiple first word vectors in the first target log, and use the average value as the first text vector.
[0125] The embodiments of the present application provide a log simplification method, device, electronic device, and computer-readable medium. The present application performs a clustering operation on all logs in a first log set to obtain multiple types of logs, wherein each type of log includes logs whose similarity is greater than a preset threshold, and then obtains a log from each type of log in the multiple types of logs to obtain a second log set, wherein the number of logs in the second log set is the same as the number of categories of the multiple types of logs. The present application obtains a second log set, which includes logs of various categories, and the number of logs in each category is 1, thereby reducing the number of logs. Operation and maintenance personnel perform fault location through the simplified logs, thereby improving the efficiency of fault location.
[0126] Based on the same technical concept, an embodiment of the present invention further provides an electronic device, such as Figure 6 As shown, it includes a processor 601, a communication interface 602, a memory 603 and a communication bus 604, wherein the processor 601, the communication interface 602, and the memory 603 communicate with each other through the communication bus 604.
[0127] Memory 603, used for storing computer programs;
[0128] The processor 601 is used to implement the above steps when executing the program stored in the memory 603.
[0129] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.
[0130] The communication interface is used for communication between the above electronic device and other devices.
[0131] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.
[0132] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
[0133] In another embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above methods are implemented.
[0134] In another embodiment of the present invention, a computer program product including instructions is provided. When the computer program product is run on a computer, the computer executes any one of the methods in the above embodiments.
[0135] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.
[0136] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0137] The above description is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest range consistent with the principles and novel features applied for herein.
Claims
1. A log simplification method, characterized in that: The method comprises: Performing a clustering operation on all logs in the first log set to obtain multiple types of logs, wherein each type of log includes logs whose similarity is greater than a preset threshold; Obtain a log from each type of log of the multiple types of logs to obtain a second log set, wherein the number of logs in the second log set is the same as the number of types of the multiple types of logs; The clustering operation on all logs in the first log set includes obtaining each type of log in the following manner: When there is an unmarked log in the first log set, obtaining the unmarked log as a first target log; Determine a similarity between the first target log and a candidate log in the first log set, wherein the candidate log is an unmarked log in the first log set except the first target log; If there is a second target log in the candidate logs and the similarity between the second target log and the first target log is greater than the preset threshold, marking the second target log in the first log set, and treating the first target log and the second target log as one type of log; When there is no second target log in the candidate logs and the similarity between the second target log and the first target log is greater than the preset threshold, the first target log is regarded as a type of log.
2. The method according to claim 1, characterized in that The first log set includes a plurality of logs that are sequentially arranged and are initially unmarked, wherein obtaining the unmarked log as the first target log includes: In the case that this is the first time to obtain logs, the first log in the first log set is obtained as the first target log; When this acquisition is not the first log acquisition, the unmarked log in the first log set that is located after the third target log is acquired as the first target log, wherein the third target log is the log acquired from the first log set last time.
3. The method according to claim 1, characterized in that After obtaining the second log set, the method further includes: determining the number of the second target logs; generating a target chart according to the number of the second target logs in the second log set and the first target log; The target graph is output.
4. The method according to claim 1, characterized in that: The determining the similarity between the first target log and the candidate logs in the first log set includes: Obtaining a first text vector of the first target log and a second text vector of a candidate log in the first log set; determining a cosine value of an angle between the first text vector and the second text vector; When the cosine value of the angle is greater than a preset threshold, it is determined that the first target log is similar to the candidate log.
5. The method according to claim 4, characterized in that The obtaining a first text vector of the first target log includes: Performing a word segmentation operation on the first target log to obtain a plurality of first words; Performing word vector processing on the first vocabulary to obtain a first word vector; An average value of a plurality of the first word vectors in the first target log is determined, and the average value is used as the first text vector.
6. A log simplification device, characterized in that: The device comprises: A clustering module, used for performing a clustering operation on all logs in the first log set to obtain multiple types of logs, wherein each type of log includes logs whose similarity is greater than a preset threshold; An acquisition module, configured to acquire a log from each type of log of the plurality of types of logs to obtain a second log set, wherein the number of logs in the second log set is the same as the number of types of the plurality of types of logs; Wherein, the clustering module includes: an acquiring unit, configured to acquire, when there is an unmarked log in the first log set, the unmarked log as a first target log; a determining unit, configured to determine a similarity between the first target log and a candidate log in the first log set, wherein the candidate log is an unmarked log in the first log set except the first target log; a marking unit, configured to mark the second target log in the first log set and treat the first target log and the second target log as one type of log if there is a second target log in the candidate logs and the similarity between the second target log and the first target log is greater than the preset threshold; as a unit, configured to treat the first target log as a type of log when there is no second target log in the candidate logs and the similarity between the second target log and the first target log is greater than the preset threshold.
7. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1 to 5 when executing a program stored in a memory.
8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 1 to 5 are implemented.
Citation Information
Patent Citations
Log clustering method based on graph structure
CN105468677A
Repetitive log filtering method for terminal
CN107171825A
Log reporting method, device, electronic device and computer-readable storage medium
CN109359026A
Log query method and device, computer device and storage medium
CN110096485A
