Fault judgment method and device, electronic equipment and storage medium

Abstract

The invention provides a fault judgment method and device, electronic equipment and a storage medium. The method comprises the steps of obtaining target log data within a preset duration based on fault occurrence time of a target server; extracting target keywords in the target log data based on preset keywords, and determining a target log vector corresponding to the target log data based on the TF-IDF value of each target keyword; and predicting the fault type of the target server based on the target daily vector and the sample fault machine data set by using a KNN algorithm. The log vector is determined through the TF-IDF value based on the keyword, the fault type is judged based on the log vector by using the KNN algorithm, automatic judgment of the fault type is realized, and the TF-IDF can reflect the unique contribution of the keyword to the log data, so that the log vector can reflect the unique contribution of different keywords, the original log data features are better reflected, and the reliability of the log data is improved. Based on the daily vector data judgment, a fault judgment result can be obtained more accurately, and the efficiency and accuracy of fault judgment are improved.

Description

Technical Field

[0001] The present invention relates to the field of operation and maintenance technology, and in particular to a fault judgment method, device, electronic equipment and storage medium. Background Art

[0002] The stable operation of the server is crucial to the enterprise. As the server operating system operation and maintenance personnel, we will receive various server failures on a daily basis, such as machine downtime, machine resource overload, machine login failure, machine memory, motherboard, CPU, disk, motherboard, power failure, etc. At present, fault handling usually relies on manual experience for analysis, and after determining the type of machine failure, corresponding processing is performed, such as capacity expansion, restarting the gate god agent used for login, replacing faulty parts, etc. The current fault analysis and processing technology mainly relies on the combination of operation and maintenance personnel experience + monitoring, which is not automated and intelligent enough. Summary of the invention

[0003] In view of this, an embodiment of the present invention provides a fault diagnosis method, device, electronic device and storage medium to improve the efficiency and accuracy of server fault diagnosis.

[0004] According to one aspect of the present invention, a fault judgment method is provided, the method comprising:

[0005] Determine the fault information of the target server to be judged, wherein the fault information includes the time when the fault occurs;

[0006] Acquire target log data within a preset time period based on the fault occurrence time;

[0007] Based on preset keywords, extract each target keyword in the target log data, and determine a target log vector corresponding to the target log data based on a word frequency-inverse text frequency index value of each target keyword;

[0008] The KNN algorithm is used to predict the fault type of the target server based on the target log vector and the sample fault machine data set.

[0009] In a possible embodiment, the sample fault machine data set is pre-tagged with the TF-IDF vocabulary, and the method further includes:

[0010] Calculating the occurrence frequency TF of each target keyword and calculating the inverse document frequency IDF of each target keyword based on the TF-IDF vocabulary;

[0011] The product of the occurrence frequency and the inverse document frequency is used as the TF-IDF value of the keyword;

[0012] The determining the target log vector corresponding to the target log data based on the TF-IDF value of each target keyword includes:

[0013] Based on the order of appearance of each target keyword in the target log data, the TF-IDF value of the target keyword is used as one-dimensional data in a target log vector to obtain a target log vector, wherein the dimension of the target log vector is the sum of the target keywords.

[0014] In a possible embodiment, the TF-IDF vocabulary of the sample faulty machine dataset is pre-obtained by the following steps:

[0015] Selecting sample faulty servers, and obtaining log data of each of the sample faulty servers within a preset time period;

[0016] Segmenting each of the log data, and obtaining a target segmentation result from each segmentation result based on a preset filtering rule, wherein the preset filtering rule includes a preset keyword;

[0017] A TF-IDF vocabulary is constructed based on each of the target word segmentation results. The TF-IDF vocabulary includes the occurrence frequency of each of the target word segmentation results and a log identifier including the target word segmentation result.

[0018] In a possible embodiment, the TF-IDF vocabulary includes a log vector of each sample faulty machine in a sample faulty machine data set, and the log vector of the sample faulty machine is determined based on a TF-IDF value of a keyword contained in the log of the sample faulty machine;

[0019] The using of the KNN algorithm to predict the fault type of the target server based on the target log vector and the sample fault machine data set includes:

[0020] Calculating the similarity between the target log vector and the sample log vectors of each of the sample faulty machines;

[0021] Determine a preset number of sample log vectors with the highest similarity to the target log vector as comparison logs;

[0022] It is determined that the fault type that occurs most frequently among the fault types corresponding to the comparison logs is the fault type corresponding to the target log vector.

[0023] In a possible embodiment, the preset number is determined by the following steps:

[0024] Dividing a preset fault data set into a plurality of subsets, wherein the preset fault data set includes logs of a machine having a fault and a fault type label thereof;

[0025] For each preset K value, a KNN algorithm is used to predict the predicted fault type of each log in a training set based on the preset K value, so as to evaluate the performance index of the preset K value in the training set; wherein the training set is selected from the subset for the preset K value, and the performance index includes the evaluation accuracy;

[0026] In the case where the evaluation accuracy is higher than the preset accuracy threshold, cross-validating the preset K value based on a validation set preset for the preset K value; wherein the validation set is the remaining subsets in each of the subsets except the training set;

[0027] In a case where the preset K value passes the cross validation, the preset K value is determined to be the preset number.

[0028] In a possible embodiment, the method further includes: encapsulating the KNN algorithm into a detection tool, wherein the detection tool is used to determine a fault type corresponding to the log vector based on the input log vector.

[0029] According to another aspect of the present invention, a fault judgment device is provided, characterized in that the device comprises:

[0030] A determination module, used to determine the fault information of the target server to be determined, wherein the fault information includes the time when the fault occurs;

[0031] An acquisition module, used to acquire target log data within a preset time period based on the fault occurrence time;

[0032] An extraction module, configured to extract each target keyword in the target log data based on preset keywords, and determine a target log vector corresponding to the target log data based on a word frequency-inverse text frequency index value of each target keyword;

[0033] The judgment module is used to predict the fault type of the target server based on the target log vector and the sample fault machine data set by using the KNN algorithm.

[0034] In a possible embodiment, the sample fault machine data set is pre-tagged with the TF-IDF vocabulary, and the device further includes:

[0035] A calculation module, used to calculate the occurrence frequency TF of each target keyword and calculate the inverse document frequency IDF of each target keyword based on the TF-IDF vocabulary; and use the product of the occurrence frequency and the inverse document frequency as the TF-IDF value of the keyword;

[0036] The determining the target log vector corresponding to the target log data based on the TF-IDF value of each target keyword includes:

[0037] Based on the order of appearance of each of the target keywords in the target log data, the TF-IDF value of the target keyword is used as one-dimensional data in a target log vector to obtain a target log vector, wherein the dimension of the target log vector is the sum of the target keywords;

[0038] The TF-IDF vocabulary of the sample fault machine dataset is pre-obtained by the following steps:

[0039] Selecting sample faulty servers, and obtaining log data of each of the sample faulty servers within a preset time period;

[0040] Segmenting each of the log data, and obtaining a target segmentation result from each segmentation result based on a preset filtering rule, wherein the preset filtering rule includes a preset keyword;

[0041] Building a TF-IDF vocabulary based on each of the target word segmentation results, wherein the TF-IDF vocabulary includes the occurrence frequency of each of the target word segmentation results and the log identifier containing the target word segmentation result;

[0042] The TF-IDF vocabulary includes the log vector of each sample faulty machine in the sample faulty machine data set, and the log vector of the sample faulty machine is determined based on the TF-IDF value of the keyword contained in the log of the sample faulty machine;

[0043] The using of the KNN algorithm to predict the fault type of the target server based on the target log vector and the sample fault machine data set includes:

[0044] Calculating the similarity between the target log vector and the sample log vectors of each of the sample faulty machines;

[0045] Determine a preset number of sample log vectors with the highest similarity to the target log vector as comparison logs;

[0046] Determine that the fault type that occurs most frequently among the fault types corresponding to the comparison logs is the fault type corresponding to the target log vector;

[0047] The preset number is determined by the following steps:

[0048] Dividing a preset fault data set into a plurality of subsets, wherein the preset fault data set includes logs of a machine having a fault and a fault type label thereof;

[0049] For each preset K value, a KNN algorithm is used to predict the predicted fault type of each log in a training set based on the preset K value, so as to evaluate the performance index of the preset K value in the training set; wherein the training set is selected from the subset for the preset K value, and the performance index includes the evaluation accuracy;

[0050] In the case where the evaluation accuracy is higher than the preset accuracy threshold, cross-validating the preset K value based on a validation set preset for the preset K value; wherein the validation set is the remaining subsets in each of the subsets except the training set;

[0051] In a case where the preset K value passes the cross validation, determining the preset K value to be the preset number;

[0052] The device also includes a packaging module, which is used to package the KNN algorithm into a detection tool, and the detection tool is used to determine the fault type corresponding to the log vector based on the input log vector.

[0053] According to another aspect of the present invention, there is provided an electronic device, comprising:

[0054] Processor; and

[0055] Memory for storing programs,

[0056] Wherein, the program includes instructions, and when the instructions are executed by the processor, the processor executes any of the above-mentioned fault judgment methods.

[0057] According to another aspect of the present invention, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute any of the above-mentioned fault judgment methods.

[0058] One or more technical solutions provided in the embodiments of the present invention perform keyword extraction on log data within a preset time period before a fault occurs, and determine a log vector corresponding to the log data based on the TF-IDF value of the keyword, thereby using the KNN algorithm to judge the fault type based on the log vector, thereby realizing automatic judgment of the fault type. Moreover, TF-IDF can reflect the unique contribution of keywords to the log data. Accordingly, the log vector can reflect the unique contribution of different keywords, so that the log vector can better reflect the original log data characteristics. When the KNN algorithm is subsequently used to determine the fault type, the fault judgment result can also be obtained more accurately based on the log vector data judgment, thereby improving the efficiency and accuracy of fault judgment. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Further details, features and advantages of the invention are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0060] Figure 1 A schematic diagram of a flow chart of a fault diagnosis method provided by an embodiment of the present invention;

[0061] Figure 2 A schematic diagram of another flow chart of a fault judgment method provided by an embodiment of the present invention;

[0062] Figure 3 A schematic diagram of the structure of a fault judgment device provided by an embodiment of the present invention;

[0063] Figure 4 A block diagram of an exemplary electronic device that can be used to implement an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0064] Embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments of the present invention are only for exemplary purposes and are not intended to limit the scope of protection of the present invention.

[0065] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0066] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the following description. It should be noted that the concepts of "first", "second", etc. mentioned in the present invention are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0067] It should be noted that the modifications of "one" and "plurality" mentioned in the present invention are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0068] The names of the messages or information exchanged between multiple devices in the embodiments of the present invention are only used for illustrative purposes, and are not used to limit the scope of these messages or information.

[0069] In the related art, server fault handling takes a long time from user reporting to operation and maintenance personnel intervening to locate and analyze the fault, and there is a possibility that the operation and maintenance personnel on duty cannot respond immediately. It also takes time for the operation and maintenance personnel to log in to the machine to analyze and operate the fault. Therefore, the time-consuming uncertainty of current server fault handling will cause further losses to the business. In addition, the current server fault location and analysis rely on the professional knowledge of the operation and maintenance personnel, and the fault handling time is proportional to the experience and skills of the operator. Ordinary users lack the professional knowledge of operation and maintenance and cannot analyze, judge and handle the current fault.

[0070] In order to improve the efficiency of fault diagnosis, the embodiments of the present invention provide a fault diagnosis method, an apparatus, an electronic device and a storage medium. The fault diagnosis method can be applied to any electronic device with a fault diagnosis function, and the electronic device can be a server, a computer or a mobile terminal, etc. The scheme of the present invention is described below with reference to the accompanying drawings:

[0071] Figure 1 A flowchart of a fault judgment method provided by an embodiment of the present invention may include the following steps:

[0072] S101, determining fault information of a target server to be determined, wherein the fault information includes a time when the fault occurs;

[0073] S102, acquiring target log data within a preset time period based on the fault occurrence time;

[0074] S103, based on preset keywords, extracting each target keyword in the target log data, and determining a target log vector corresponding to the target log data based on the TF-IDF value of each target keyword;

[0075] S104: Predict the fault type of the target server based on the target log vector and the sample faulty machine data set using a KNN algorithm.

[0076] By applying the embodiment of the present invention, keywords are extracted from log data within a preset time period before the occurrence of a fault, and a log vector corresponding to the log data is determined based on the TF-IDF value of the keyword, so that the KNN algorithm is used to judge the fault type based on the log vector, thereby realizing automatic judgment of the fault type. TF-IDF can reflect the unique contribution of keywords to the log data, and accordingly, the log vector can reflect the unique contribution of different keywords, so that the log vector can better reflect the original log data characteristics. When the KNN algorithm is subsequently used to determine the fault type, the fault judgment result can also be obtained more accurately based on the log vector data judgment, thereby improving the efficiency and accuracy of fault judgment.

[0077] The following is an exemplary description of S101-S104:

[0078] The target server to be judged in S101 can be any server that has a fault. The fault specifically refers to a fault in the server or a service running in the server. In a possible embodiment, a monitoring tool can be used to monitor the running status of the machine and each service. When a fault occurs during operation, an error message can be thrown to prompt that the machine has a fault. The fault information may include the identification of the machine that has a fault and the time when the fault occurs. The above monitoring tool can be flexibly selected according to the actual scenario, and the present invention does not specifically limit this.

[0079] When the error message is received, the log of the machine within the preset time period can be obtained to make a fault judgment. The preset time period can be selected according to the actual application scenario, such as 10 minutes, 30 minutes or 6 hours before the fault occurs. The machine log usually records the operating status of the machine, including the access information, performance indicators, error information and security events of the machine, etc., where the access information may include the client IP address of the server accessing the server, the access time, etc., the performance indicators may include the resource usage in the server, including the CPU resource utilization rate, the memory utilization rate, etc., the error information is the abnormality that occurs in the server during operation, including the abnormal event and the corresponding abnormal occurrence time, and the security event may include the permission change record and login record in the server, etc. For example, the log may be the operating system log of the server.

[0080] The log usually records the full amount of data of the machine operation, and the amount of data is relatively large. Therefore, in a possible embodiment, the server log can be cleaned after obtaining the log, such as cleaning data that is not related to the fault. As a possible implementation method, the data information that needs to be eliminated can be pre-set, such as log timestamps and system check logs, etc. The data information that needs to be eliminated can be set in the form of field identifiers. The log data can store different data information through different fields. Based on the field identifier, the corresponding data information in the log can be determined and eliminated, thereby obtaining the target log data. The above-mentioned data information that needs to be eliminated can also be in the form of a regular expression, and the present invention does not make specific limitations on this. As a possible implementation method, the data information that needs to be retained can also be set, and the log is filtered based on the data information that needs to be retained to obtain the target log data.

[0081] After obtaining the target log data, it can be segmented. In a possible embodiment, the target log data can be segmented according to the word length. For example, if the log identifies fault-related information in English, the segmentation length can be set to 1 word, that is, each segmentation result contains one word. In other possible embodiments, a word list can be preset, and the preset word list includes multiple preset words, which can be set according to the actual business scenario. Accordingly, the target log data can be segmented based on the preset word list to obtain the segmentation result.

[0082] In a possible embodiment, NTLK (Natural Language Toolkit) may be used to segment the target log.

[0083] Afterwards, preset keywords can be obtained from each word segmentation result. The preset keywords refer to words that are highly relevant to fault detection, such as Out of Memory, Hardware Error, Failed, Warning, Crash, Reboot, Exception, Resource limit reached, etc.

[0084] After determining each preset keyword, the TF-IDF (Term Frequency-Inverse Document Frequency) value of the keyword can be calculated, where TF refers to the frequency of occurrence of the keyword. The higher the TF of the keyword, the more representative the keyword is of the content of the log, and the more important it is. IDF refers to the frequency of occurrence of the keyword in the prediction library, which can be historical log data. The higher the IDF of the keyword, the more it is shared by many log data, and the less important it is because it cannot well represent the unique content of the target log. TF-IDF is the product of the keyword's TF and IDF.

[0085] In a possible embodiment, the TF and IDF of a keyword may be calculated by the following formula:

[0086] TF(t,d) = number of occurrences of word t in log d / total number of words in document d

[0087] IDF(t) = log(total number of logs / (number of logs containing word t + 1))

[0088] Wherein, t is a keyword, d represents a target log, the total number of words in document d is all word segmentation results contained in the target log, and the total number of logs refers to the number of all logs contained in the sample faulty machine data set. The sample faulty machine data set is annotated by a TF-IDF table. The TF-IDF vocabulary includes the log vectors of each sample faulty machine in the sample faulty machine data set. The log vectors of the sample faulty machines are determined based on the TF-IDF values ​​of the keywords contained in the logs of the sample faulty machines. The method for obtaining the TF-IDF table will be described in detail below. In the IDF calculation formula, +1 is used to avoid division by 0.

[0089] Through the above formula, the TF-IDF value of the keyword can be calculated. By extracting each keyword contained in the target log and calculating the TF-IDF value of each keyword, the log vector corresponding to the target log data can be obtained. As a possible implementation method, the TF-IDF value corresponding to the keyword can be used as one-dimensional data in the vector according to the order of appearance of each keyword in the target log, and the number of keywords contained in the log is the dimension of the log vector.

[0090] For example, a log contains five keywords, namely failed, reboot, warning, crash, and error. The TF-IDF value of failed is 0.1, the TF-IDF value of reboot is 0.8, the TF-IDF value of warning is 0, the TF-IDF value of crash is 0, and the TF-IDF value of error is 0.4. The log vector corresponding to the log is [0.1, 0.8, 0, 0, 0.4].

[0091] A higher TF-IDF value means that the keyword contributes more to the uniqueness of the log, and thus has a larger weight in the vector. In the subsequent fault judgment process, the keyword with a larger weight in the vector contributes more to the judgment result, thereby improving the accuracy of fault judgment.

[0092] The present invention adopts the KNN (K-Nearest Neighbor) algorithm to judge the server fault type based on the above log vector. The principle of the KNN algorithm is that if most of the K nearest neighboring samples of a sample in the feature space belong to a certain category, then the sample also belongs to this category. For the present invention, if most of the K nearest neighboring samples of a log vector in the feature space belong to a certain category, then the log vector also belongs to this category.

[0093] In a possible embodiment, the server failure type can be determined by using the KNN algorithm through the following steps:

[0094] S201, calculating the similarity between the target log vector and the sample log vectors of each sample faulty machine;

[0095] The above-mentioned sample faulty machine refers to a server that has already failed and the fault type has been determined before the fault of the current server is determined. The server that has failed and the fault type has been determined can be a server belonging to the same cluster as the target server, or a server belonging to a different cluster as the target server. The present invention does not make specific limitations on this.

[0096] In a possible embodiment, after determining the server fault type, the target log data used in the server determination process, the log vector corresponding to the target log data, and the fault type can be stored accordingly. Exemplarily, the above historical fault records can be stored in a database in a Key-Value manner, where the Key is the target log data identifier or the server identifier, and the log vector and the fault type are the values ​​corresponding to the Key.

[0097] As a possible implementation method, the log vectors corresponding to each historical fault record log can be traversed in the database, and for each log vector, the similarity between it and the target log vector is calculated. The similarity can be measured by a similarity distance, such as Euclidean distance, Manhattan distance, Minkowski distance, etc.

[0098] S202: Determine a preset number of sample log vectors with the highest similarity to the target log vector as comparison logs.

[0099] S203: Determine that among the fault types corresponding to the comparison logs, the most common fault type is the fault type corresponding to the target log vector.

[0100] As a possible implementation, each historical fault record log may be sorted according to the similarity with the target log vector, for example, each historical fault record may be sorted in descending order of similarity, and the top K historical fault record logs are obtained, where K is the above-mentioned preset number. Alternatively, each historical fault record may be sorted in descending order of similarity, and the bottom K historical fault records are obtained.

[0101] The preset number is the K value in the KNN algorithm. The KNN algorithm needs to select a suitable K value. A K value that is too small will affect the prediction performance and generalization ability of the model, causing the model to be sensitive to noise and prone to overfitting. A K value that is too large will cause the model to be unable to learn data details and prone to underfitting. In order to select a suitable K value, the present invention can determine the K value by cross-validation.

[0102] In a possible implementation manner, the preset number, i.e., the K value, may be determined by the following steps:

[0103] S31, dividing a preset fault data set into a plurality of subsets, wherein the preset fault data set includes logs of a machine having a fault and a fault type label thereof;

[0104] S32. For each preset K value, use a KNN algorithm based on the preset K value to predict the predicted fault type of each log in the training set to evaluate the performance index of the preset K value in the training set; wherein the training set is selected from the subset for the preset K value, and the performance index includes an evaluation accuracy rate;

[0105] S33, when the evaluation accuracy is higher than a preset accuracy threshold, cross-validating the preset K value based on a validation set preset for the preset K value; wherein the validation set is the remaining subsets in each of the subsets except the training set;

[0106] S34: When the preset K value passes cross validation, determine that the preset K value is the preset number.

[0107] In a possible embodiment, a data set may be obtained, and the data set may be the above-mentioned historical fault log records, or may be a data set of other record logs and their corresponding fault types, which is not specifically limited in the present invention. The data set may then be divided into a plurality of subsets, and a plurality of K values ​​may be preset, and the plurality of K values ​​may be flexibly determined according to the actual application scenario. For each preset K value, a subset may be randomly determined as a training set, and the remaining subsets may be used as validation sets to cross-validate the corresponding K values, and the remaining subsets refer to other subsets in each subset except the training set.

[0108] For the log data in the training set, its log vector can be obtained according to the above method, and S201-S203 can be executed on the log vector according to the preset K value to obtain its fault type, and the judgment effect of the preset K value can be determined based on the acquired fault type and the fault type label of the log data. Specifically, the loss function value can be calculated by the probability of the fault type output by the KNN algorithm and the fault type label. When the average value of the loss function value of each data in the training set is less than the preset loss threshold, the K value can be retained for further verification. Otherwise, the K value is discarded.

[0109] In a possible embodiment, the proportion of correct judgment results in all data included in the training set can also be determined. If the proportion exceeds a preset value, the K value can be retained for further verification; otherwise, the K value is discarded.

[0110] The reserved K value can be verified using the validation set. As a possible implementation method, the fault judgment result output for each data in the validation set based on the reserved K value can be obtained, and the mean square error is calculated based on the judgment result output by the KNN algorithm and the fault label of each data in the validation set, and the K value whose mean square error does not exceed the error threshold is selected as the target K value. In the case where the mean square errors corresponding to multiple K values ​​do not exceed the error threshold, the K value with the smallest mean square error can be determined as the target K value.

[0111] In a possible embodiment, the failure type of the server to be predicted can be voted out by the failure types of the K nearest comparison logs of the target log vector. For example, when the k value is 4, the failure types of the four neighboring sample servers of the target server are CPU overload, memory failure, memory failure, and host failure, respectively. Then, the server failure type to be predicted selects the type with the most failures in the neighboring samples, and is determined to be memory failure.

[0112] The KNN algorithm does not require a training process before application. It only needs to store training data and does not require a large amount of parameter adjustment. Therefore, it is very lightweight and easy to implement and deploy.

[0113] In a possible embodiment, the above-mentioned KNN algorithm can be encapsulated as a detection tool, which can provide an API. By inputting the target log vector into the detection tool through the API, the fault type output by the detection tool can be obtained, which greatly saves manpower consumption.

[0114] like Figure 2 As shown, Figure 2 Another flow chart of a fault diagnosis method provided by an embodiment of the present invention is provided. The fault diagnosis method may include two parts: data preprocessing and KNN prediction. The data preprocessing may include the following process:

[0115] Select a sample faulty machine, which refers to a server that has experienced a fault. Intercept the operating system log of the sample faulty machine within 6 hours before the fault occurs, and use NTLK to segment the log to obtain multiple segmentation results. Filter the preset keywords that are helpful for fault prediction, and build a TF-IDF vocabulary based on the keywords contained in the log to annotate the faulty machine dataset.

[0116] The KNN prediction process may include: determining the target K value by cross-validation, that is, dividing the training set into multiple sets for cross-validation, and selecting a suitable target K value according to the performance of the corresponding K value in each set, and the training set is the above-mentioned faulty machine data set.

[0117] For the machine to be predicted, the TF-IDF value of each keyword contained in the log of the predicted machine can be calculated, and a log vector can be constructed based on the TF-IDF value. Based on the log vector, the Euclidean distance between the server to be predicted and the sample faulty machine is calculated, the fault type of the K nearest neighbor data is obtained, and the fault type of the server to be predicted is determined by voting.

[0118] By applying the embodiment of the present invention, the operating system log generated before the server failure is fully utilized. After a series of cleaning, keyword TF-IDF value calculation, log vector encoding, etc., the server log is converted into vector data that can be used for further modeling, and the information contained in the log data is fully utilized. In addition, the log vector data processed in the first step is modeled by the KNN clustering algorithm, which takes advantage of the lightweight and easy-to-deploy features of the KNN algorithm and completes the second-level response to the target server failure prediction. Compared with the traditional fault handling method, this prediction algorithm is a more intelligent and automated fault prediction solution.

[0119] Based on the same inventive concept, the embodiment of the present invention also provides a fault judgment device, such as Figure 3 As shown, the fault judgment device 300 may include:

[0120] The determination module 301 is used to determine the fault information of the target server to be determined, wherein the fault information includes the time when the fault occurs;

[0121] An acquisition module 302 is used to acquire target log data within a preset time period based on the fault occurrence time;

[0122] An extraction module 303 is used to extract each target keyword in the target log data based on preset keywords, and determine a target log vector corresponding to the target log data based on a word frequency-inverse text frequency index value of each target keyword;

[0123] The judgment module 304 is used to predict the fault type of the target server based on the target log vector and the sample faulty machine data set by using the KNN algorithm.

[0124] In a possible embodiment, the sample fault machine data set is pre-tagged with the TF-IDF vocabulary, and the device further includes:

[0125] A calculation module, used to calculate the occurrence frequency TF of each target keyword and calculate the inverse document frequency IDF of each target keyword based on the TF-IDF vocabulary; and use the product of the occurrence frequency and the inverse document frequency as the TF-IDF value of the keyword;

[0126] The determining the target log vector corresponding to the target log data based on the TF-IDF value of each target keyword includes:

[0127] Based on the order of appearance of each of the target keywords in the target log data, the TF-IDF value of the target keyword is used as one-dimensional data in a target log vector to obtain a target log vector, wherein the dimension of the target log vector is the sum of the target keywords;

[0128] The TF-IDF vocabulary of the sample fault machine dataset is pre-obtained by the following steps:

[0129] Selecting sample faulty servers, and obtaining log data of each of the sample faulty servers within a preset time period;

[0130] Segmenting each of the log data, and obtaining a target segmentation result from each segmentation result based on a preset filtering rule, wherein the preset filtering rule includes a preset keyword;

[0131] Building a TF-IDF vocabulary based on each of the target word segmentation results, wherein the TF-IDF vocabulary includes the occurrence frequency of each of the target word segmentation results and the log identifier containing the target word segmentation result;

[0132] The TF-IDF vocabulary includes the log vector of each sample faulty machine in the sample faulty machine data set, and the log vector of the sample faulty machine is determined based on the TF-IDF value of the keyword contained in the log of the sample faulty machine;

[0133] The using of the KNN algorithm to predict the fault type of the target server based on the target log vector and the sample fault machine data set includes:

[0134] Calculating the similarity between the target log vector and the sample log vectors of each of the sample faulty machines;

[0135] Determine a preset number of sample log vectors with the highest similarity to the target log vector as comparison logs;

[0136] Determine that the fault type that occurs most frequently among the fault types corresponding to the comparison logs is the fault type corresponding to the target log vector;

[0137] The preset number is determined by the following steps:

[0138] Dividing a preset fault data set into a plurality of subsets, wherein the preset fault data set includes logs of a machine having a fault and a fault type label thereof;

[0139] For each preset K value, a KNN algorithm is used to predict the predicted fault type of each log in a training set based on the preset K value, so as to evaluate the performance index of the preset K value in the training set; wherein the training set is selected from the subset for the preset K value, and the performance index includes the evaluation accuracy;

[0140] In the case where the evaluation accuracy is higher than the preset accuracy threshold, cross-validating the preset K value based on a validation set preset for the preset K value; wherein the validation set is the remaining subsets in each of the subsets except the training set;

[0141] In a case where the preset K value passes the cross validation, determining the preset K value to be the preset number;

[0142] The device also includes a packaging module, which is used to package the KNN algorithm into a detection tool, and the detection tool is used to determine the fault type corresponding to the log vector based on the input log vector.

[0143] Among them, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in the present invention are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0144] The exemplary embodiment of the present invention further provides an electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication. The memory stores a computer program that can be executed by the at least one processor, and the computer program is used to enable the electronic device to perform a method according to an embodiment of the present invention when executed by the at least one processor.

[0145] Exemplary embodiments of the present invention also provide a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to perform a method according to an embodiment of the present invention.

[0146] An exemplary embodiment of the present invention further provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor of a computer, the computer is used to enable the computer to perform a method according to an embodiment of the present invention.

[0147] refer to Figure 4 , a block diagram of an electronic device 400 that can be used as a server or client of the present invention will now be described, which is an example of a hardware device that can be applied to various aspects of the present invention. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0148] like Figure 4 As shown, the electronic device 400 includes a computing unit 401, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 402 or a computer program loaded from a storage unit 408 into a random access memory (RAM) 403. In the RAM 403, various programs and data required for the operation of the electronic device 400 can also be stored. The computing unit 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0149] A plurality of components in the electronic device 400 are connected to the I / O interface 405, including: an input unit 406, an output unit 407, a storage unit 408, and a communication unit 409. The input unit 406 may be any type of device capable of inputting information to the electronic device 400, and the input unit 406 may receive input digital or character information, and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 407 may be any type of device capable of presenting information, and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 408 may include, but is not limited to, a disk, an optical disk. The communication unit 409 allows the electronic device 400 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks, and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0150] The computing unit 401 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 401 performs the various methods and processes described above. For example, in some embodiments, the above-mentioned fault judgment method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 408. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 400 via the ROM 402 and / or the communication unit 409. In some embodiments, the computing unit 401 may be configured to perform the above-mentioned fault judgment method in any other appropriate manner (e.g., by means of firmware).

[0151] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0152] In the context of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0153] As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0154] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0155] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0156] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

Claims

1. A fault judgment method, characterized in that: The method comprises: Determine the fault information of the target server to be judged, wherein the fault information includes the time when the fault occurs; Acquire target log data within a preset time period based on the fault occurrence time; Based on preset keywords, extract each target keyword in the target log data, and determine a target log vector corresponding to the target log data based on a word frequency-inverse text frequency index value of each target keyword; The KNN algorithm is used to predict the fault type of the target server based on the target log vector and the sample fault machine data set.

2. The method according to claim 1, characterized in that The sample fault machine data set is pre-tagged with the TF-IDF vocabulary, and the method further includes: Calculating the occurrence frequency TF of each of the target keywords and calculating the inverse document frequency IDF of each of the target keywords based on the TF-IDF vocabulary; The product of the occurrence frequency and the inverse document frequency is used as the TF-IDF value of the keyword; The determining the target log vector corresponding to the target log data based on the TF-IDF value of each target keyword includes: Based on the order of appearance of each target keyword in the target log data, the TF-IDF value of the target keyword is used as one-dimensional data in a target log vector to obtain a target log vector, wherein the dimension of the target log vector is the sum of the target keywords.

3. The method according to claim 2, characterized in that The TF-IDF vocabulary of the sample fault machine dataset is pre-obtained by the following steps: Selecting sample faulty servers, and obtaining log data of each of the sample faulty servers within a preset time period; Segmenting each of the log data, and obtaining a target segmentation result from each segmentation result based on a preset filtering rule, wherein the preset filtering rule includes a preset keyword; A TF-IDF vocabulary is constructed based on each of the target word segmentation results. The TF-IDF vocabulary includes the occurrence frequency of each of the target word segmentation results and a log identifier including the target word segmentation result.

4. The method according to claim 2, characterized in that: The TF-IDF vocabulary includes the log vector of each sample faulty machine in the sample faulty machine data set, and the log vector of the sample faulty machine is determined based on the TF-IDF value of the keyword contained in the log of the sample faulty machine; The using of the KNN algorithm to predict the fault type of the target server based on the target log vector and the sample fault machine data set includes: Calculating the similarity between the target log vector and the sample log vectors of each of the sample faulty machines; Determine a preset number of sample log vectors with the highest similarity to the target log vector as comparison logs; It is determined that the fault type that occurs most frequently among the fault types corresponding to the comparison logs is the fault type corresponding to the target log vector.

5. The method according to claim 4, characterized in that The preset number is determined by the following steps: Dividing a preset fault data set into a plurality of subsets, wherein the preset fault data set includes logs of a machine having a fault and a fault type label thereof; For each preset K value, a KNN algorithm is used to predict the predicted fault type of each log in a training set based on the preset K value, so as to evaluate the performance index of the preset K value in the training set; wherein the training set is selected from the subset for the preset K value, and the performance index includes the evaluation accuracy; In the case where the evaluation accuracy is higher than the preset accuracy threshold, cross-validating the preset K value based on a validation set preset for the preset K value; wherein the validation set is the remaining subsets in each of the subsets except the training set; In a case where the preset K value passes the cross validation, the preset K value is determined to be the preset number.

6. The method according to claim 1, characterized in that The method further includes: encapsulating the KNN algorithm into a detection tool, wherein the detection tool is used to determine the fault type corresponding to the log vector based on the input log vector.

7. A fault judgment device, characterized in that: The device comprises: A determination module, used to determine the fault information of the target server to be determined, wherein the fault information includes the time when the fault occurs; An acquisition module, used to acquire target log data within a preset time period based on the fault occurrence time; An extraction module, configured to extract each target keyword in the target log data based on preset keywords, and determine a target log vector corresponding to the target log data based on a word frequency-inverse text frequency index value of each target keyword; The judgment module is used to predict the fault type of the target server based on the target log vector and the sample fault machine data set by using the KNN algorithm.

8. The device according to claim 7, characterized in that The sample fault machine data set is pre-tagged with the TF-IDF vocabulary, and the device further includes: A calculation module, used to calculate the occurrence frequency TF of each target keyword and calculate the inverse document frequency IDF of each target keyword based on the TF-IDF vocabulary; and use the product of the occurrence frequency and the inverse document frequency as the TF-IDF value of the keyword; The determining the target log vector corresponding to the target log data based on the TF-IDF value of each target keyword includes: Based on the order of appearance of each of the target keywords in the target log data, the TF-IDF value of the target keyword is used as one-dimensional data in a target log vector to obtain a target log vector, wherein the dimension of the target log vector is the sum of the target keywords; The TF-IDF vocabulary of the sample fault machine dataset is pre-obtained by the following steps: Selecting sample faulty servers, and obtaining log data of each of the sample faulty servers within a preset time period; Segmenting each of the log data, and obtaining a target segmentation result from each segmentation result based on a preset filtering rule, wherein the preset filtering rule includes a preset keyword; Building a TF-IDF vocabulary based on each of the target word segmentation results, wherein the TF-IDF vocabulary includes the occurrence frequency of each of the target word segmentation results and the log identifier containing the target word segmentation result; The TF-IDF vocabulary includes the log vector of each sample faulty machine in the sample faulty machine data set, and the log vector of the sample faulty machine is determined based on the TF-IDF value of the keyword contained in the log of the sample faulty machine; The using of the KNN algorithm to predict the fault type of the target server based on the target log vector and the sample fault machine data set includes: Calculating the similarity between the target log vector and the sample log vectors of each of the sample faulty machines; Determine a preset number of sample log vectors with the highest similarity to the target log vector as comparison logs; Determine that the fault type that occurs most frequently among the fault types corresponding to the comparison logs is the fault type corresponding to the target log vector; The preset number is determined by the following steps: Dividing a preset fault data set into a plurality of subsets, wherein the preset fault data set includes logs of a machine having a fault and a fault type label thereof; For each preset K value, a KNN algorithm is used to predict the predicted fault type of each log in a training set based on the preset K value, so as to evaluate the performance index of the preset K value in the training set; wherein the training set is selected from the subset for the preset K value, and the performance index includes the evaluation accuracy; In the case where the evaluation accuracy is higher than the preset accuracy threshold, cross-validating the preset K value based on a validation set preset for the preset K value; wherein the validation set is the remaining subsets in each of the subsets except the training set; In a case where the preset K value passes the cross validation, determining the preset K value to be the preset number; The device also includes a packaging module, which is used to package the KNN algorithm into a detection tool, and the detection tool is used to determine the fault type corresponding to the log vector based on the input log vector.

9. An electronic device, comprising: processor; as well as Memory for storing programs, The program includes instructions, which, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 6.

10. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to make a computer execute the method according to any one of claims 1-6.

Images