A fault diagnosis method and device based on density clustering

Abstract

The application discloses a kind of fault diagnosis method and device based on density clustering, it is related to fault diagnosis field, when server fails, the first log data of the log file of each equipment in server in a period of time before and after current time is acquired, and the second log data of log file in a period of time before and after once failure is acquired, finally the current working parameter of each equipment in server is acquired again, these three parameters are sent to clustering model, to determine the cause of server this time failure and its solution strategy, the cause of failure and failure solution strategy are sent to user terminal. By the mode of obtaining multiple log data, multiple log files can be associated, and the cause of failure can be determined more accurately. Moreover, the clustering model itself has a simple structure and low application difficulty, making it suitable for wide application. In addition, the clustering model directly provides the failure solution strategy, eliminating the need for manual diagnosis and reducing the technical threshold of staff.

Description

[0001] Technology Neighborhood

[0002] This invention relates to the field of fault diagnosis, and in particular to a fault diagnosis method and apparatus based on density clustering. Background Technology

[0003] Various components within a server continuously generate logs during operation. As the number of servers increases, the amount of logs generated within the same timeframe also increases, increasing the difficulty of daily server maintenance for staff. When a server malfunctions, the sheer volume of logs, including many irrelevant ones, makes it difficult for staff to quickly locate the fault and determine the appropriate solution, further complicating maintenance. Current technologies often incorporate neural network models to improve efficiency. These models learn to filter out faulty logs from all server logs for staff review. While this method improves efficiency, simply filtering faulty logs fails to uncover connections between multiple logs, making it impossible to accurately determine the fault type based on these relationships. Furthermore, the complex structure of neural network models hinders widespread adoption due to their high application difficulty. Summary of the Invention

[0004] The purpose of this invention is to provide a fault diagnosis method and apparatus based on density clustering, which can link multiple log files together to more accurately determine the cause of the fault. Moreover, it is easy to apply and can be widely used. It does not require manual diagnosis, thus lowering the technical threshold for staff.

[0005] To address the aforementioned technical problems, this invention provides a fault diagnosis method based on density clustering, comprising:

[0006] Determine whether the server is malfunctioning at the current moment based on the log files generated by each device in the server;

[0007] If so, obtain the first log data of all the log files within a first preset time range centered on the current time;

[0008] Obtain second log data from the log file within a second preset time range centered on each historical moment of failure;

[0009] Obtain the current operating parameters of all devices in the server;

[0010] All the first log data, the second log data, and all the current working parameters are sent to a preset clustering model in order to determine the fault type and fault resolution strategy at the current moment.

[0011] sending the fault type and the fault resolution strategy to a user terminal.

[0012] Preferably, determining whether the server has a fault at the current time according to log files generated by each device in the server comprises:

[0013] obtaining log files generated by all the devices in the server;

[0014] for any of the log files, determining whether a log line containing a preset keyword is generated at the current time in the log file;

[0015] if yes, determining that the server has a fault at the current time.

[0016] Preferably, obtaining second log data in a second preset time length range centered on each historical time of fault in the log file comprises:

[0017] for any of the log files generating the preset keyword at the current time, obtaining all second log data in a second preset time length range centered on the time of generating the preset keyword in the log file.

[0018] Preferably, in the preset clustering model, determining the fault type and the fault resolution strategy at the current time comprises:

[0019] for any of the log files generating the preset keyword at the current time, determining a first distance between the first log data in the log file and the first log data of all other log files;

[0020] for any of the log files generating the preset keyword at the current time, determining a second distance between the first log data in the log file and the second log data in all other log files generating the preset keyword at the current time;

[0021] clustering the first log data and all log data with a distance close to a corresponding preset distance to obtain a first clustering cluster corresponding to the first log data of each of the log files generating the preset keyword at the current time;

[0022] determining the fault type and the fault resolution strategy at the current time according to each of the first clustering cluster and the current working parameter.

[0023] Preferably, before obtaining first log data of all the log files in a first preset time length range centered on the current time, the method further comprises:

[0024] For any one of the log files, delete the log lines without time stamps in the log file as a whole.

[0025] Preferably, before sending all the first log data, the second log data and all the current working parameters into the preset clustering model, further comprising:

[0026] Carry out word segmentation on all the log data respectively to obtain a plurality of words;

[0027] Determine whether the similarity between each of the log data is greater than a preset similarity according to the words;

[0028] If the similarity is greater than the preset similarity, change the neighborhood parameter corresponding to one of the log data, and keep the neighborhood parameter corresponding to another of the log data as a default parameter;

[0029] Send all the first log data, the second log data and all the current working parameters into a preset clustering model, comprising:

[0030] Send all the first log data, the second log data, the neighborhood parameters of the two kinds of log data and all the current working parameters into a preset clustering model.

[0031] Preferably, determining whether the similarity between each of the log data is greater than a preset similarity according to the words, comprising:

[0032] In any one of the log data, determine the proportion between each of the words in the log data and the total number of words in the log data;

[0033] For any two of the log data, the following steps are performed:

[0034] Determine the proportion difference between the proportions of each of the same words between the two log data;

[0035] Take all the words with a proportion difference less than a preset difference as similar words between the two log data;

[0036] Determine whether the sum of the proportions of all the similar words is greater than a preset proportion;

[0037] If yes, determine that the similarity between the two log data is greater than a preset similarity.

[0038] Preferably, in the preset clustering model, determine the fault type and the fault solving strategy of the server in this failure, comprising:

[0039] cluster all the first log data and the second log data, multiply the clustering result by a preset multiple and by the neighborhood parameter corresponding to the two log data, to obtain a plurality of second clustering clusters;

[0040] determine the fault type and the fault resolution strategy of the current moment according to each second clustering cluster and the working parameter.

[0041] Preferably, the pre-training of the preset clustering model comprises:

[0042] obtain N log files;

[0043] divide the N log files into M log packs, N and M are positive integers and M is not greater than N;

[0044] corresponding to the current working parameter obtained each time the preset keyword is detected is added to the log pack in which the log file generating the preset keyword is located;

[0045] obtain the fault type and the corresponding fault resolution strategy sent by a user and add them to a designated log pack;

[0046] for any log file, M log packs are respectively taken as M training sets of the log file to train the log file.

[0047] The application also provides a fault diagnosis device based on density clustering, comprising:

[0048] a memory for storing a computer program;

[0049] a processor for executing the computer program to realize the steps of the fault diagnosis method based on density clustering as described above.

[0050] The application provides a fault diagnosis method and device based on density clustering, relating to the field of fault diagnosis. When a server fails, the first log data of the log file of each device in the server within a period of time before and after the current moment is obtained, the second log data of the log file within a period of time before and after the time when the log file has ever failed is obtained, and finally the current working parameter of each device in the server is obtained. The three parameters are sent to a clustering model together to determine the cause of the current failure of the server and the resolution strategy thereof, and the cause of the failure and the resolution strategy of the failure are sent to a user terminal. By obtaining multiple log data, multiple log files can be associated, and the cause of the failure can be determined more accurately. Moreover, the clustering model itself has a simple structure and low application difficulty, and is conducive to wide application. In addition, the clustering model directly gives the resolution strategy of the failure, and manual diagnosis is not needed, thereby reducing the technical threshold of the staff. BRIEF DESCRIPTION OF DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the accompanying drawings needed to be used in the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0052] Figure 1 A flow chart of a fault diagnosis method based on density clustering provided by the present application;

[0053] Figure 2 A structural schematic diagram of a fault diagnosis model provided by the present application;

[0054] Figure 3 A structural diagram of a fault diagnosis device based on density clustering provided by the present application. DETAILED DESCRIPTION

[0055] The core of the present application is to provide a fault diagnosis method and device based on density clustering, which can associate multiple log files, can more accurately determine the fault cause, has low application difficulty, is conducive to wide use, and does not need manual diagnosis, thereby reducing the technical threshold of the staff.

[0056] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings of the embodiments of the present application. Obviously, the described embodiments are some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without any creative effort fall within the protection scope of the present application.

[0057] With the advent of the big data era, the storage demand for services is becoming larger and larger, and the number of servers is becoming more and more, which makes the daily operation and maintenance of the server cluster more and more difficult. Because server operation and maintenance must pay attention to the log in the server, but because the server generates more logs, more and more servers will result in a huge amount of logs in the whole server cluster, and manual work is difficult to undertake the increasingly difficult server operation and maintenance work.

[0058] In order to improve the operation and maintenance efficiency and improve the operation and maintenance speed of a single server, the prior art usually adds a neural network model to replace manual operation, uses the powerful processing and analysis capability of the neural network model to select logs indicating faults from a large number of logs in real time, classifies the fault logs and normal logs, and the staff only needs to focus on the fault logs, thereby reducing the amount of logs that the staff needs to face. Although this method can improve efficiency, it only performs classification and does not provide the cause of the current fault, nor does it diagnose the current fault in combination with multiple log files. The generation of a server problem is a complex process, and it is impossible to diagnose whether the server has a problem through logs in a file. Therefore, the prior art either cannot accurately determine the fault or needs to consume a large number of training samples to train the model. The former has low accuracy, and the latter is difficult to widely apply.

[0059] To solve the above technical problems, please refer to Figure 1 , Figure 1 A flowchart of a fault diagnosis method based on density clustering provided by the present application comprises:

[0060] S1: determining whether the server has a fault at the current time according to log files generated by each device in the server;

[0061] In order to timely solve the fault in the server, first, the log files of each device in the server need to be continuously or periodically acquired. Each device in the server has a corresponding log file. When the device is working, the device writes various data and actions such as its own actions, task execution results, and current state into the log file. These log files usually include two types: one is a file in which the device continuously writes, and the other is multiple log files classified according to different actions and tasks of the device, that is, the content in these log files usually includes multiple actions at different times in a period of time. Therefore, when determining whether the server has a fault at the current time according to these log files, it is mainly determined whether there is content indicating a fault in the latest log in the existing log file and whether there is content indicating a fault in the log file newly generated at the current time.

[0062] S2: if yes, acquiring first log data of all log files in a first preset time range centered on the current time;

[0063] S3: acquiring second log data in a second preset time range centered on each historical time of the fault in the log file;

[0064] In order to accurately determine the fault of the server, the fault cannot be determined unilaterally according to a single log file, for example, the fault cannot be determined according to a log file in which a line of log generated most recently contains content indicating a fault. Therefore, when the fault of the server is determined, two kinds of log data need to be obtained: the first log data is all log data of all log files in the server within a period of time before and after the current time, for example, data within 3 minutes before and after the current time can be obtained, and a log file usually contains about 100 lines of log data; the second log data is previous log data, because a log file contains multiple logs within a period of time, a log file usually contains multiple fault logs, and the second log data is all log data within a period of time before and after the time at which each fault log in the log file occurs. In summary, the essence of the first log data and the second log data is that when the server fails each time, each log file obtains each log within a period of time before and after the time at which the failure occurs each time. Further, when the second log data is obtained, in order to reduce the amount of data processing, only the second log data of the log file in which the content of the fault at the current time can be obtained, that is, only the second log data of the log file in which the content of the fault at the current time is obtained.

[0065] S4: Obtain the current working parameters of all devices in the server;

[0066] In order to further accurately determine the fault of the server, the current working parameters of each device in the server also need to be obtained, which refers to the model, working temperature, frequency, voltage and some actual working related parameters (such as the speed of the fan or the occupancy rate of the processor) of the device. It can be understood that when the server fails, some devices on the server must be abnormal, at this time, the working parameters of the device are the embodiment of the "symptoms" of the device. For the specific steps of obtaining the working parameters, the working parameters can be obtained through various sensors arranged in the server, or the working parameters can be obtained through the BMC (Baseboard Management Controller) in the server, and the present application does not limit this.

[0067] S5: Send all the first log data, the second log data and all the current working parameters to a preset clustering model, so as to determine the fault type and the fault solving strategy at the current time;

[0068] S6: Send the fault type and the fault solving strategy to a user terminal.

[0069] The clustering model is mainly a density-based spatial clustering of applications with noise (DBSCAN), and similar log data is clustered into a cluster. When the model is trained in advance, the staff diagnoses the server failure according to the log files and collected working parameters, and uploads the log files, working parameters, diagnosed failure types and failure resolution strategies to the model. When the actual failure type is determined, the log data obtained at the current time is clustered into a cluster, and the cluster corresponding to the log file with the failure content at the current time is determined. If the working parameters are similar to the working parameters obtained at the current time, the pre-diagnosed failure type and failure resolution strategy can be sent to the staff; if the working parameters are not similar, the pre-diagnosed failure type and failure resolution strategy can also be sent to the staff, and the model is optimized according to the feedback content of the staff. If the staff feedback content indicates that the failure type and failure resolution strategy are correct, the current working parameters can also correspond to the cluster, otherwise a new cluster needs to be generated.

[0070] It should be noted that the obtained first log data and second log data must contain normal logs. When the same failure occurs multiple times, if the log files of a certain device contain normal logs in the log data of the several failures, but the several pieces of log data are very similar or even identical, it indicates that the several pieces of log data are related to the failure. Therefore, the normal log data also needs to be considered. Further, a certain keyword that frequently appears in the normal log data can be used as a new failure identifier to enrich the failure detection method.

[0071] In summary, when the server fails, the first log data of the log files of each device in the server within a certain period of time before and after the current time is obtained, the second log data of the log files within a certain period of time before and after the failure is obtained, and finally the current working parameters of each device in the server are obtained. The three parameters are sent to the clustering model to determine the cause of the server failure and the resolution strategy, and the failure cause and the failure resolution strategy are sent to the user terminal. By obtaining multiple log data, multiple log files can be associated to more accurately determine the failure cause. Moreover, the clustering model has a simple structure and low application difficulty, and is suitable for wide application. In addition, the clustering model directly provides the failure resolution strategy, which does not require manual diagnosis, thereby reducing the technical threshold and workload of the staff.

[0072] Based on the above embodiment:

[0073] As a preferred embodiment, the log files generated by the devices in the server are used to determine whether the server fails at the current time, comprising:

[0074] obtaining log files generated by all devices in the server;

[0075] for any log file, judging whether a log line containing a preset keyword is generated in the log file at the current time;

[0076] if yes, determining that the server has a failure at the current time.

[0077] In order to simply determine whether the server has a failure, in the present application, it is considered that each device will mark the type of the log when generating the log. For example, info, error, fail, waring, debug, etc., wherein the marks of error, fail and waring all indicate that there is a problem in the log line to some extent, and therefore, when judging whether the server has a failure, it is only necessary to judge whether there is an error, fail and waring mark indicating a problem in the latest log line in the existing log file, and whether there is also such a mark in the log file newly generated at the current time. Based on this, by detecting the keyword, it can be simply determined whether the server has a failure.

[0078] As a preferred embodiment, the second log data in the log file within a second preset time length range centered on each historical time of failure includes:

[0079] For any log file generating a preset keyword at the current time, all second log data in the log file within a second preset time length range centered on the time of generating the preset keyword is obtained.

[0080] In order to reduce the data processing amount, in the present application, some second log data with small failure correlation will not be obtained. Specifically, the purpose of obtaining the first log data is to judge whether there is a correlation between various log files at the current time, and therefore, the first data of each log file needs to be obtained; and the purpose of obtaining the second log data is mainly to judge whether there is a relationship between the previous failure and the current failure, and therefore, when obtaining the second log data, only the second log data of the log file with failure content at the current time is obtained. Based on this, the data processing amount can be effectively reduced.

[0081] As a preferred embodiment, in the preset clustering model, determining the failure type and the failure solution strategy at the current time includes:

[0082] For any log file generating preset keyword at the current time, determine the first distance between the first log data in the log file and the first log data of all other log files;

[0083] For any log file generating preset keyword at the current time, determine the second distance between the first log data in the log file and the second log data of all other log files generating preset keyword at the current time;

[0084] Cluster the first log data and all log data with distance close to the corresponding preset distance to obtain the first clustering cluster corresponding to the first log data of each log file generating preset keyword at the current time;

[0085] Determine the fault type and fault resolution strategy of the current time according to each first clustering cluster and the current working parameter.

[0086] In order to cluster the log data, in the present application, the clustering is mainly based on the distance between two log data. Here, the log file with fault content at the current time is called fault log file, and the log file without fault content at the current time is called normal log file. Specifically, the first distance between the first log data of the fault log and the first log data of each other fault+normal log file needs to be calculated, and the second distance between the first log data of the fault log and the second log data of each other fault log file needs to be calculated. The first distance is equivalent to calculating the relevance between different log files at the current time, and the second distance is equivalent to determining whether the influence on the equipment when the fault occurs each time is similar or not. Further, the third distance between the first log data when the fault occurs this time and the second log data of the fault log file itself can also be calculated for each fault log file, which is equivalent to seeing whether the fault occurring this time has occurred before.

[0087] In actual clustering, the first preset distance can be set smaller, such as 0.4, and the second preset distance needs to be set larger, such as 0.6. This will accurately analyze whether the historical fault log data of different files is related to the current occurring fault, and the log data at the current time is necessarily related to the current occurring fault, so the judgment accuracy of the first log data is lower than that of the second log data.

[0088] As a preferred embodiment, before obtaining the first log data of all log files within the first preset time length range centered on the current time, it further comprises:

[0089] For any log file, delete the log line without timestamp in the log file in the whole line.

[0090] In order to obtain effective log data, in the present application, considering that there are various types of logs generated by the server, some of which do not have a timestamp (such as logs that simply report the execution status of a task), not only cannot be used for server operation and maintenance, but also cannot be used to speculate when the log was generated. For fault detection, it has no reference. Therefore, before obtaining log data, each log in the log file without a timestamp needs to be deleted. When obtaining the first log data, the range of the first preset time period is expanded in response to the number of deleted logs to avoid too few log entries. Based on this, by deleting the number of log entries without a timestamp, the part without reference in the log file is removed, and effective log data can be obtained.

[0091] As a preferred embodiment, before sending all the first log data, the second log data and all the current working parameters to the preset clustering model, it further includes:

[0092] segmenting all the log data respectively to obtain a plurality of words;

[0093] determining whether the similarity between the log data is greater than a preset similarity according to the words;

[0094] if the similarity is greater than the preset similarity, changing the neighborhood parameter corresponding to one of the log data, and keeping the neighborhood parameter corresponding to the other log data as a default parameter;

[0095] sending all the first log data, the second log data and all the current working parameters to the preset clustering model, including:

[0096] sending all the first log data, the second log data, the neighborhood parameters of the two types of log data and all the current working parameters to the preset clustering model.

[0097] In order to ensure the diversity of the cluster, in the present application, considering that there may be too similar or even identical log files between different log files, a plurality of identical cluster clusters may be generated based on these identical log files, which leads to the existence of useless clusters in the generated cluster clusters, therefore, the neighborhood parameter of one of the log files needs to be adjusted, so that the clustering algorithm can also obtain different cluster clusters when clustering similar or identical log files. Specifically, the similarity of two log files is judged from two dimensions: one dimension is to perform word segmentation on the log files to obtain a word set corresponding to the log files, if the word sets of the two log files are highly repetitive, it means that the two log files are similar, for example, if the similarity between the two log files is more than 50%, it is considered to be close; one dimension is from the perspective of rows, if there are some rows of data in the two log files that are completely identical, it means that the two log files are similar. For two similar log files, the neighborhood parameter of one of the log files remains unchanged, still the default value, and the neighborhood parameter of the other log file needs to be changed, so as to distinguish the two log files. Based on this, the diversity of the cluster can be ensured.

[0098] As a preferred embodiment, whether the similarity between the log data is greater than the preset similarity is judged according to the word, including:

[0099] In any log data, the proportion of each word in the log data to the total number of words in the log data is determined;

[0100] For any two log data, the following steps are performed:

[0101] The proportion difference between the proportions of each same word between the two log data is determined;

[0102] All words with a proportion difference less than a preset difference are regarded as similar words between the two log data;

[0103] Whether the sum of the proportions of all similar words is greater than a preset proportion is judged;

[0104] If yes, it is determined that the similarity of the two log data is greater than the preset similarity.

[0105] In order to accurately determine the similarity between two log files, in the present application, firstly, the proportion of each word in the log file where it is located is calculated, which refers to the proportion of the number of occurrences of the word to the total number of words in the log file; secondly, the proportions of the same word in different log files are compared, if the proportions of a certain word in two log files are similar, it means that the word is a similar word of the two log files, based on this, all similar words between the two log files are calculated. Then, the sum of the proportions of all similar words is calculated, that is, the sum of the proportions of all similar words to the total number of words in the log file, because the proportions of the similar words in the two log files are similar, only the proportion of these words in one of the log files needs to be calculated. If the sum of the proportions of all similar words is greater than a preset proportion, it means that the two log files are similar. For example, if the proportion of all similar words is greater than 50% of the entire log file, it means that at least half of the two log files are similar. Based on this, the similarity between the two log files can be accurately determined.

[0106] As a preferred embodiment, in the preset clustering model, the fault type and the fault resolution strategy of the server in this failure are determined, comprising:

[0107] Clustering all the first log data and the second log data, and multiplying the clustering result by a preset multiple and multiplying it by the neighborhood parameter corresponding to the two log data, to obtain a plurality of second clustering clusters;

[0108] According to each second clustering cluster and the working parameter, the fault type and the fault resolution strategy at the current time are determined.

[0109] In order to accurately cluster, in the present application, for each log file, the DBSCAN density clustering algorithm is used to initialize the neighborhood parameter (ε, MinPts) to calculate the closeness between two log data. The basic formula is min(X i ,X j )*0.4, wherein Xi is the log data in one log file, Xj is the log data in another log file, and 0.4 is a preset multiple. Further, for two non-similar log data, the neighborhood parameter P is 1, so the final result is min(X i ,X j )*0.4; for two similar log data, the neighborhood parameter P of a certain log data is not 1, for example, it may be 0.8, so the final result is min(X i ,X j*0.4*0.8. Similarly, for two log data, if the two log data are similar, then ε = ε * η, where η is the sum of the proportions of all similar words described above; otherwise, ε remains the default value.

[0110] After clustering, each time the distance between the log data is determined to be close, the log data is clustered together, and subsequent other clustering is not calculated as a simple example, assuming that there are five log data A, B, C, D and E to be clustered, first calculate the distance between A and BCDE, if B is close to A, then A and B are clustered, and the distance between C and DE is calculated directly in the second clustering, without considering A and B. Based on this, multiple clustering clusters are formed, until each log data is clustered, and the clustering clusters are labeled according to the fault type and fault solution strategy uploaded by the operation and maintenance personnel. Based on this, accurate clustering can be performed.

[0111] As a preferred embodiment, the pre-training of the preset clustering model includes:

[0112] Obtain N log files;

[0113] Divide the N log files into M log packs, N and M are positive integers and M is not greater than N;

[0114] When the preset keyword is detected each time, the current working parameter obtained is added to the log pack where the log file generating the preset keyword is located;

[0115] Obtain the fault type and corresponding fault solution strategy sent by the user and add them to the designated log pack;

[0116] For any log file, M log packs are used as M training sets of the log file to train the log file.

[0117] In order to improve the accuracy of the clustering model, in the present application, some log files can be randomly selected from all historical log files, and these log files can be packaged into multiple log packs, and the log data contained in each log pack can be different, for example, several million log files can be randomly selected, and these log files can be irregularly packaged into 100,000 log packs. For each log file in the several million log files, the log pack is used as a training set of the log file, that is, each log file has 100,000 training sets, and the fault reason and fault solution strategy are used as the association.

[0118] Please refer to Figure 2 , Figure 2A structure diagram of a fault diagnosis model provided by the application is shown in the figure. The left side block diagram is a training step. The log package in the server is input into the log analysis device as a training set, that is, into the device where the clustering model is located. Then, the useless data in the log package is cleaned, the time stamp is standardized, the logs are segmented to obtain a plurality of word sets, and then a corresponding model is established for each log file. The model is trained using the log package as the training set of the model to determine the association rules between the logs and the corresponding fault causes and solution strategies. The right side block diagram is an actual use process. The logs in the server are collected. If a fault is found in the logs, the various log data in the above embodiment is sent to the diagnosis device. The diagnosis device can be the same device as the log analysis device or another device. The diagnosis device calculates and further clusters the log data obtained this time and the previous training results based on the clustering model to find the corresponding fault causes and solution strategies of the fault this time as the diagnosis results output to the staff.

[0119] For specific training steps and actual application steps, a log file indicating the existence of a fault is taken as an example. When a new log file indicating the existence of a fault is added to a log package at a certain time, the log file and all other log files in the same log package are clustered first. Then, the other log packages are observed to see whether there are log files added to other log packages at the same time as the log file. If so, the log file and each log file in the other log packages are clustered again. Of course, in order to reduce the amount of calculation, the log file can be clustered only with the log files in the log package where the log file indicating the existence of a fault is added to other log packages at the same time. As a simple example, it is assumed that there are four log files A, B, C, and D in total. A and B are in a log package x, C and D are in another log package y and a log package z respectively. If a log file E indicating the existence of a fault is added to a log package x at a certain time, and a log file F indicating the existence of a fault is added to a log package z, E and AB are clustered first, and then E and DF are clustered.

[0120] Please refer to Figure 3 , Figure 3 A structure diagram of a fault diagnosis device based on density clustering provided by the application is shown in the figure. The device includes:

[0121] A memory 21 for storing a computer program;

[0122] A processor 22 for executing the computer program to implement the steps of the fault diagnosis method based on density clustering as described above.

[0123] For a detailed introduction of the fault diagnosis device based on density clustering provided by the present application, please refer to the above-mentioned embodiments of the fault diagnosis method based on density clustering, which will not be repeated here.

[0124] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the difference from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the device disclosed by the embodiments, since it corresponds to the method disclosed by the embodiments, the description is relatively simple, and the related parts can be referred to the method part.

[0125] It should also be noted that in the specification, the relationship terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between the entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0126] The above description of the disclosed embodiments enables a person skilled in the art to implement or use the present application. Various modifications to the embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A density clustering-based fault diagnosis method characterized by, The method comprises the following steps: determining whether the server has failed at the current time according to log files generated by each device in the server; if so, obtaining first log data of all the log files within a first preset time range centered on the current time; obtaining second log data within a second preset time range centered on each historical time of failure in the log files; obtaining current working parameters of all the devices in the server; sending all the first log data, the second log data and all the current working parameters to a preset clustering model to determine the type of failure at the current time and a failure resolution strategy; wherein, before sending all the first log data, the second log data and all the current working parameters to the preset clustering model, the method further comprises the following steps: performing word segmentation on all the log data respectively to obtain a plurality of words; determining whether the similarity between each of the log data is greater than a preset similarity according to the words; if the similarity is greater than the preset similarity, changing the neighborhood parameter corresponding to one of the log data and keeping the neighborhood parameter corresponding to another of the log data as a default parameter; sending all the first log data, the second log data and all the current working parameters to the preset clustering model comprises: sending all the first log data, the second log data, the neighborhood parameters of the two types of log data and all the current working parameters to the preset clustering model; in the preset clustering model, determining the type of failure and the failure resolution strategy of the server in this failure comprises: clustering all the first log data and the second log data, multiplying the clustering result by a preset multiple and by the neighborhood parameters corresponding to the two types of log data to obtain a plurality of second clustering clusters; determining the type of failure at the current time and the failure resolution strategy according to each of the second clustering clusters and the working parameters; sending the type of failure and the failure resolution strategy to a user terminal.

2. The density-based clustering based fault diagnosis method of claim 1, wherein, determining whether the server has failed at the current time according to log files generated by each device in the server comprises: obtaining log files generated by all the devices in the server; for any one of the log files, determining whether a log line containing a preset keyword is generated in the log file at the current time; if so, determining that the server has failed at the current time.

3. The density-based clustering based fault diagnosis method of claim 2, wherein, obtaining second log data within a second preset time range centered on each historical time of failure in the log files comprises: for any one of the log files generating the preset keyword at the current time, obtaining second log data within a second preset time range centered on the time of generating the preset keyword in the log file.

4. The density-based clustering based fault diagnosis method of claim 3, wherein, in the preset clustering model, determining the type of failure at the current time and the failure resolution strategy comprises: For any of the log files generating the preset keyword at the current moment, determining a first distance between the first log data in the log file and the first log data of all other log files; For any of the log files generating the preset keyword at the current moment, determining a second distance between the first log data in the log file and the second log data in all other log files generating the preset keyword at the current moment; Clustering the first log data and all log data with distances close to the corresponding preset distance to obtain a first clustering cluster corresponding to the first log data of each of the log files generating the preset keyword at the current moment; Determining a fault type and a fault resolution strategy at the current moment according to each of the first clustering cluster and the current working parameter.

5. The density-based clustering based fault diagnosis method of claim 1, wherein, Before obtaining all of the first log data of the log files in a first preset time range centered on the current moment, further comprising: For any of the log files, deleting log lines without timestamps in the log file in a whole line.

6. The density-based clustering based fault diagnosis method of claim 1, wherein, According to the word, determining whether the similarity between each log data is greater than a preset similarity, comprising: In any of the log data, determining the proportion between each of the words in the log data and the total number of words in the log data; For any two of the log data, the following steps are performed: Determining the proportion difference between the proportions of each same word between the two log data; Taking all of the words with the proportion difference less than a preset difference as similar words between the two log data; Determining whether the sum of the proportions of all of the similar words is greater than a preset proportion; If yes, determining that the similarity of the two log data is greater than a preset similarity.

7. The density-based clustering based fault diagnosis method according to any one of claims 1 to 6, characterized in that, The pre-training of the preset clustering model comprises: Obtaining N log files; Dividing N log files into M log packs, N and M are positive integers and M is not greater than N; Corresponding to the current working parameter obtained each time the preset keyword is detected, adding the log file generating the preset keyword to the log pack where the log file is located; Obtaining the fault type and the corresponding fault resolution strategy sent by the user and adding them to the specified log pack; For any of the log files, M log packs are respectively taken as M training sets of the log file to train the log file.

8. A density clustering-based failure diagnosis apparatus characterized by comprising: Comprise: A memory for storing a computer program; A processor for executing the computer program to realize the steps of the fault diagnosis method based on density clustering according to any one of claims 1 to 7.

Images