Disk failure prediction method, device, equipment, and storage medium

Abstract

The present application discloses a disk failure prediction method and apparatus, a device, and a storage medium, wherein the method comprises: obtaining current attribute data of a disk to be tested; when there is no target classifier corresponding to the disk to be tested among a plurality of pre-trained target classifiers, obtaining a corresponding training set used when training each of the target classifiers; determining the similarity between the current attribute data and each of the training sets; and processing the current attribute data using a target classifier whose similarity satisfies a first condition to obtain a failure prediction result of the disk to be tested; in this way, the generalization capability of the disk failure prediction function can be improved.

Description

Technical Field

[0001] The present application relates to computer technology, and is related to but not limited to disk failure prediction methods and devices, equipment, and storage media. Background Art

[0002] As the volume of business increases, most data centers are facing the problem of aging servers. The longer the server is in service, the more likely it is to fail. In order to ensure support for existing network services, it is necessary to repair or replace failed servers in a timely manner, which consumes a lot of manpower and time costs. Among these failures, disk failures account for the largest proportion. Currently, there are a variety of backup strategies to reduce the risk of data loss, which can be repaired after the failure occurs. However, if the risk of failure can be predicted in advance, the operation and maintenance personnel can find high-risk failure points in advance during daily inspections, so that data migration can be carried out in advance, reducing the pressure caused by sudden failures.

[0003] Normally, various types of disks can periodically detect the attribute values ​​of various attributes of the disk through Self-Monitoring Analysis and Reporting Technology (SMART); based on these attribute values, the health status of the disk is reflected. When the detected attribute value exceeds the preset threshold, an alarm is issued. However, in order to reduce the false alarm rate, the SMART threshold is usually set high by the manufacturer. Therefore, the detection rate of directly using a single SMART detection value to predict disk failure is very low, which has become a consensus before various disk failure prediction methods are proposed.

[0004] Most of the existing disk failure prediction methods are based on SMART data. Through machine learning methods, SMART data related to failures are screened out; then, based on these data, classifiers are trained, and disk failures are predicted using the trained classifiers.

[0005] However, related methods are all trained for one type of disk, and the evaluation results are also obtained based on this sample. In actual production environments, there are many different types of disks. However, there is a problem with SMART data. Due to the use of different algorithms, the distribution of the detection values ​​of the same attribute of different types of disks produced by the same manufacturer is different. In addition, the attribute categories and distributions of the detection values ​​of different manufacturers are also different. This leads to the need to build a corresponding target classifier for each type of disk, and the requirements for the training set are relatively high.

[0006] Currently, there is no method for processing SMART data of different disks produced by different manufacturers. In actual production, the idea that can solve this problem is to build multiple target classifiers, and use the classifiers to which the same type of disk belongs for fault prediction. However, this method relies on a large amount of training data. Taking the SMART data of disks provided by a certain platform as an example, the data distribution is relatively unbalanced, the proportion of faulty disks is relatively small, and the proportion of different types of disks also varies greatly. For manufacturers whose businesses are built on super-large clusters, classifiers trained based on their own clusters are more adaptive, but their scalability is limited. Because the SMART data provided by disks produced by different manufacturers is not uniform, the data distribution of the same attribute ID in the SMART reports of different types of disks is also different. These factors will affect the prediction accuracy of the classifier.

[0007] For vendors that provide storage resource monitoring and management services, disk failure prediction is a relatively effective service that facilitates operation and maintenance management. However, generally speaking, the entire set of monitoring and management services is usually deployed on different types of clusters, which means that disk failure prediction, as one of the functions, is best to have strong generalization. Therefore, considering the diversity of actual application scenarios, it is necessary to study a method that can achieve as much coverage as possible by building fewer classifiers to predict failures of multiple types of disks based on a limited training set. Summary of the invention

[0008] In view of this, the present application provides a disk failure prediction method and apparatus, device, and storage medium, which can achieve failure prediction for multiple types of disks based on a limited training set by constructing fewer classifiers, thereby achieving as much coverage as possible.

[0009] The disk failure prediction method, device, equipment, and storage medium provided in this application are implemented as follows:

[0010] In a first aspect, the present application provides a disk failure prediction method, comprising: obtaining current attribute data of the disk to be tested; in the case that there is no target classifier corresponding to the disk to be tested among multiple target classifiers obtained in pre-training, obtaining a corresponding training set used when training each of the target classifiers; determining the similarity between the current attribute data and each of the training sets; and using the target classifier whose similarity meets the first condition, processing the current attribute data to obtain a failure prediction result of the disk to be tested.

[0011] In a second aspect, the present application provides a disk failure prediction device, comprising: an acquisition module, used to obtain current attribute data of the disk to be tested; the acquisition module is also used to obtain a corresponding training set used when training each target classifier when there is no target classifier corresponding to the disk to be tested among multiple target classifiers obtained in pre-training; a determination module is used to determine the similarity between the current attribute data and each of the training sets; a prediction module is used to process the current attribute data using a target classifier whose similarity meets a first condition to obtain a failure prediction result of the disk to be tested.

[0012] In a third aspect, the present application provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, and when the processor executes the program, the steps in any disk failure prediction method described in the present application are implemented.

[0013] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in any disk failure prediction method described in the present application.

[0014] In the present application, when there is no corresponding target classifier for the disk to be tested, the target classifier whose similarity meets the first condition can be used to predict the fault of the disk to be tested; in this way, the fault prediction of the disk to be tested of unknown disk type can be realized, and the generalization ability and flexibility of the electronic device in performing the fault prediction function can be improved. In other words, in the face of the diversity of actual application scenarios, through the above method, the electronic device can predict the data of multiple disk types based on a limited training set by constructing fewer target classifiers, thereby achieving as much coverage as possible. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is a schematic diagram of an application scenario of the disk failure prediction method according to an embodiment of the present application;

[0016] Figure 2 This is a schematic diagram of the implementation process of the disk failure prediction method according to an embodiment of the present application;

[0017] Figure 3 This is a schematic diagram of the implementation flow of another disk failure prediction method according to an embodiment of the present application;

[0018] Figure 4 This is a schematic diagram of the training process corresponding to the first target classifier in the embodiment of the present application;

[0019] Figure 5A This is a schematic diagram of the training process corresponding to the second target classifier in the embodiment of the present application;

[0020] Figure 5B This is another schematic diagram of a training process corresponding to the second target classifier in an embodiment of the present application;

[0021] Figure 6 A schematic diagram of the implementation flow of another disk failure prediction method according to an embodiment of the present application;

[0022] Fig. 7A This is a schematic diagram of the structure of a disk failure prediction device according to an embodiment of the present application;

[0023] Figure 7B This is a schematic diagram of the composition structure of another disk failure prediction device according to an embodiment of the present application;

[0024] Figure 8 A hardware entity schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the present application clearer, the specific technical solution of the present application will be further described in detail below in conjunction with the drawings in the present application. The following embodiments are used to illustrate the present application, but are not used to limit the scope of the present application.

[0026] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing this application and are not intended to limit this application.

[0027] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0028] It should be pointed out that the terms "first\second\third" involved in the embodiments of the present application are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0029] First, the present application embodiment provides an application scenario of a disk failure prediction method, as shown in the schematic diagram of the scenario. Figure 1As shown, cloud servers 101 to 10M provide services for user terminal cluster 12 through switch 111, where M is an integer greater than 1; user terminal cluster 12 may include: user terminal 121, user terminal 122, user terminal 123, ..., user terminal 12K, where K is an integer greater than 3. Cloud server 101 includes disk array 1011 and disk array 1012 for storing data; cloud server 10M includes disk array 10M1 and disk array 10M2 for storing data, and each disk array includes one or more types of disks.

[0030] The server can use SMART technology to collect multiple SMART data from each disk according to the disk's time series. Each piece of data may include attribute values ​​of the disk's head unit, disk temperature, disk surface medium material, motor and its drive system, disk internal circuit, underlying data read error rate and other attributes.

[0031] It should be noted that the application scenarios described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. A person of ordinary skill in the art will know that with the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0032] The present application embodiment provides a disk failure prediction method, which can be applied to an electronic device, which can be a device with computing capabilities such as a laptop computer and a desktop computer. The electronic device can be Figure 1 Any server shown may also be a management platform independent of each server. The function implemented by the disk failure prediction method can be implemented by the processor in the electronic device calling the program code, and of course the program code can be stored in a computer storage medium. It can be seen that the electronic device at least includes a processor and a storage medium.

[0033] Figure 2 FIG. 1 is a schematic diagram of the implementation flow of the disk failure prediction method according to an embodiment of the present application. Figure 2 As shown, the method may include the following steps 201 to 204:

[0034] Step 201, obtaining current attribute data of the disk to be tested.

[0035] During implementation, the electronic device can extract the attribute values ​​and time information of key attributes from the SMART data of the disk to be tested, and use the extracted data as the current attribute data of the disk to be tested. The current attribute data can be data at a moment, such as the attribute value of at least one key attribute at the current moment and the corresponding collection time. It can also be data at multiple moments, such as the current moment and data collected at at least one historical moment within a specific time from the current moment.

[0036] It can be understood that the so-called key attributes refer to SMART attributes for which the attribute value distribution of the same attribute of the faulty disk and the healthy disk is significantly different, and SMART attributes for which the attribute value changes of the same attribute of the faulty disk and the healthy disk are highly distinguishable. When implementing, the electronic device can select key attributes from the SMART attribute set by using mutation point detection. Mutation points can be those moments when irreversible permanent mutations occur.

[0037] Step 202 : when there is no target classifier corresponding to the disk to be tested among the plurality of pre-trained target classifiers, a training set corresponding to the training of each target classifier is obtained.

[0038] In an embodiment of the present application, each target classifier is used to predict the failure of a disk of a corresponding type. Each target classifier corresponds to a disk type. For example, each target classifier carries a type identifier for identifying the type of the prediction object of the target classifier. Therefore, the electronic device can obtain the type identifier of the disk to be tested, and then determine whether the type identifier is the same as the type identifier carried by the target classifier; if the same type identifier is not found, it is determined that the disk to be tested does not have a corresponding target classifier.

[0039] Of course, if the same type identifier is found, the current attribute data is processed using the target classifier with the same type identifier to achieve fault prediction for the disk to be tested.

[0040] Step 203, determining the similarity between the current attribute data and each of the training sets;

[0041] Step 204: Process the current attribute data using the target classifier whose similarity satisfies the first condition to obtain a fault prediction result of the disk to be tested.

[0042] The first condition can be various. For example, the first condition is that the similarity is greater than a specific threshold; for another example, the first condition is the maximum similarity. It can be understood that the current attribute data is most similar to which training set, indicating that the distribution of the attribute data of the disk to be tested is very different from the distribution of the attribute data of the disk corresponding to the training set.

[0043] In the embodiment of the present application, when there is no corresponding target classifier for the disk to be tested, the target classifier whose similarity meets the first condition can be used to predict the fault of the disk to be tested; in this way, the fault prediction of the disk to be tested of unknown disk type can be realized, and the generalization ability and flexibility of the electronic device in performing the fault prediction function can be improved. In other words, in the face of the diversity of actual application scenarios, through the above method, the electronic device can predict the data of multiple disk types based on a limited training set by constructing fewer target classifiers, thereby achieving as much coverage as possible.

[0044] The present application embodiment further provides a disk failure prediction method. Figure 3 FIG. 1 is a schematic diagram of the implementation flow of the disk failure prediction method according to an embodiment of the present application. Figure 3 As shown, the method may include the following steps 301 to 308:

[0045] Step 301: Obtain current attribute data and a first type identifier of the disk to be tested.

[0046] It can be understood that the first type identifier is used to uniquely identify the disk type of the disk to be tested. In some embodiments, the first type identifier can be the disk model of the disk to be tested.

[0047] Step 302, obtaining a second type identifier of a disk type corresponding to each pre-trained target classifier; wherein each of the target classifiers is used to perform fault prediction on a disk of the corresponding disk type.

[0048] There are two main types of target classifiers. One is the first target classifier, which is trained by sample data of the same type of disks, so it is used to predict faults for this type of disks. The other is the second target classifier, which is obtained by migration training with sample data of a type of disks with a smaller number of disks and a training set in the first target classifier with a similar distribution to the sample data, so it is used to predict faults for this type of disks. In other words, each target classifier corresponds to a corresponding disk type. For example, each target classifier carries a second type identifier. Different target classifiers carry different second type identifiers. The second type identifier is used to uniquely identify a type of disk that the corresponding target classifier can predict. The second type identifier can be a disk model.

[0049] Step 303, determining whether the first type identifier is different from each of the second type identifiers; if so, executing step 305; otherwise, executing step 304;

[0050] It can be understood that when the first type identifier is different from each of the second type identifiers, it is determined that there is no target classifier corresponding to the disk to be tested, and then the process goes to step 305 .

[0051] Step 304: input the current attribute data into a target classifier corresponding to a second type identifier that is the same as the first type identifier, so as to achieve fault prediction for the disk to be tested.

[0052] Step 305, obtaining a corresponding training set used when training each of the target classifiers;

[0053] Step 306, determining the distribution characteristic value of the attribute mean sequence of each key attribute in each of the training sets; using the distribution characteristic value of each key attribute in each of the training sets as the distribution characteristic vector of the corresponding training set.

[0054] It should be noted that the attribute mean sequence here includes the attribute mean of a key attribute of each disk in the training set. For example, as shown in Table 1, a training set includes the attribute mean of key attribute 1 to key attribute 5 of disk 1, disk 2 to disk N, where N is an integer greater than 2. Except for the first column, each column of data is the attribute mean sequence of the corresponding key attribute.

[0055] Table 1

[0056] Key attribute 1 Key attribute 2 Key attribute 3 Key attribute 4 Key Attribute 5 Disk 1 Attribute mean 1_1 Attribute mean 2_1 Attribute mean 3_1 Attribute mean 4_1 Attribute mean 5_1 Disk 2 Attribute mean 1_2 Attribute mean 2_2 Attribute mean 3_2 Attribute mean 4_2 Attribute mean 5_2 …… …… …… …… …… …… Disk N Attribute mean 1_n Attribute mean 2_n Attribute mean3_n Attribute mean 4_n Attribute mean 5_n

[0057] In the embodiment of the present application, the distribution characteristic value of the attribute mean sequence can be various, for example, it can be the mean, variance, range, dispersion, moment, skewness or kurtosis, etc.

[0058] Still taking Table 1 as an example, after obtaining the distribution eigenvalue of each column, as shown in Table 2, the obtained distribution eigenvector is (distribution eigenvalue 1, distribution eigenvalue 2, distribution eigenvalue 3, distribution eigenvalue 4, distribution eigenvalue 5), that is, the second row of data in Table 2.

[0059] Table 2

[0060] Key attribute 1 Key attribute 2 Key attribute 3 Key attribute 4 Key Attribute 5 Distribution eigenvalue 1 Distribution eigenvalue 2 Distribution eigenvalue 3 Distribution eigenvalue 4 Distribution eigenvalue 5

[0061] Step 307: determine the distance between the current attribute data and each of the distribution feature vectors to obtain the similarity with the corresponding training set.

[0062] In implementation, when the current attribute data is data at one moment, the distance between the current attribute data and each distribution feature vector can be directly determined. When the current attribute data is data at multiple moments, the attribute value sequence of each key attribute in the current attribute data can be weighted averaged (such as exponential smoothing) to obtain the corresponding attribute mean vector; then, the distance between the attribute mean vector and each distribution feature vector can be determined, and each distance represents the similarity between the corresponding training set and the current attribute data.

[0063] In the embodiment of the present application, there is no limitation on the type of the distance. For example, the distance is a cosine value or a Euclidean distance.

[0064] It should be noted that, due to the differences between disks, a key attribute corresponding to the distribution feature vector has an attribute value of NULL or infinity in the current attribute data. In order to avoid the problem that the electronic device cannot calculate the distance due to this situation, in some embodiments, the electronic device can determine the target attribute whose attribute value does not meet certain conditions from the current attribute data, such as an attribute value of NULL or infinity; the electronic device can process the attribute value of the target attribute in the current attribute data according to the distribution feature value corresponding to the target attribute in the distribution feature vector. For example, the attribute value of the target attribute in the current attribute data is updated to the distribution feature value corresponding to the target attribute in the distribution feature vector; then, the electronic device determines the distance between the distribution feature vector and the processed current attribute data.

[0065] Step 308: Process the current attribute data using the target classifier whose similarity satisfies the first condition to obtain a fault prediction result of the disk to be tested.

[0066] Before starting the disk failure prediction function, the electronic device needs to obtain multiple target classifiers through training, and then use these target classifiers to implement the disk failure prediction function. It should be noted that the training process corresponding to the target classifier can also be implemented by other electronic devices.

[0067] The multiple target classifiers include at least one first target classifier and at least one second target classifier. The corresponding training strategies can refer to steps 401 to 403 and steps 501 to 503 of the following embodiment. Among them, the training process corresponding to the first target classifier is as follows: Figure 4 As shown, the following steps 401 to 403 may be included:

[0068] Step 401, performing weighted average processing on the historical attribute value sequence of each key attribute of each first disk to obtain a first attribute mean value of the corresponding key attribute; taking the first attribute mean value of each key attribute of each first disk as the first attribute mean value vector of the corresponding disk, and adding it to the first data set;

[0069] Each of the first disks belongs to a first disk type, the total number of the first disks is greater than a first threshold, and the number of failed disks in all of the first disks is greater than a second threshold.

[0070] It is understandable that the historical attribute value sequence described here refers to the attribute value of a key attribute of the first disk in the time series. For example, the attribute value sequence corresponding to a key attribute collected within a year. The historical attribute value sequence of a key attribute is weighted averaged to obtain the attribute mean of the key attribute. It is understandable that if a disk has multiple key attributes, the historical attribute value sequence of each key attribute of the disk is weighted averaged to obtain the attribute mean vector of the disk.

[0071] In the embodiment of the present application, the weighted average processing method can be various, such as exponential smoothing. The purpose of weighted average processing is to make the obtained attribute mean value able to characterize the data change characteristics of the corresponding key attribute.

[0072] It can be understood that if the total number of the first disks is greater than the first threshold and the number of failed disks among all the first disks is greater than the second threshold, the data of the obtained training set will be richer, thereby obtaining a target classifier with better classification performance.

[0073] Step 402: Determine a first training set corresponding to the first disk type based on the first data set.

[0074] Generally speaking, the number of healthy disks in the first data set is much larger than the number of faulty disks. Therefore, it is necessary to select the attribute mean vectors of representative healthy disks from the first data set as elements in the training set. In some embodiments, the electronic device can select elements in the training set from the first data set by clustering the first data set.

[0075] For example, the first data set includes data of 100 faulty disks and data of 10,000 healthy disks. Obviously, the number of the two types of disks in this set is too different. Therefore, 100 disks need to be selected from the 10,000 healthy disks. The selection method, for example, first determines the mean of the attribute mean sequence corresponding to the 10,000 healthy disks, and then finds the first attribute mean vector of the 100 healthy disks closest to the mean, as the element in the first training set.

[0076] Step 403: Use the first training set to train a first initial classifier to obtain the first target classifier.

[0077] During implementation, the electronic device may implement step 403 by using a Regularized Greedy Forest (RGF) algorithm. Of course, it may also be implemented by using other classification algorithms, such as a decision tree, a random forest, and the like.

[0078] Understandably, in actual application scenarios, it is not possible to obtain sample data that meets the training conditions for each type of disk. For some types of disks, the number of faulty disks is insufficient, or the total number of disks is insufficient. In this way, directly using the attribute data of these disks to train the classifier will result in poor classification performance of the target classifier. Therefore, in order to make up for this deficiency, the first training set with better data can be combined to implement the training of the initial classifier.

[0079] For example, in some embodiments, after obtaining each of the first target classifiers, the second initial classifier is trained according to the first training set corresponding to the first target classifier to obtain the second target classifier; in implementation, for example Figure 5A As shown, the following steps 501 to 503 may be included:

[0080] Step 501, performing weighted average processing on the historical attribute value sequence of each key attribute of each second disk to obtain a second attribute mean of the corresponding key attribute; taking the second attribute mean of each key attribute of each second disk as the second attribute mean vector of the corresponding disk, and adding it to the second data set; wherein each second disk belongs to the second disk type, and the number of failed disks in all the second disks is less than or equal to the first threshold;

[0081] Step 502: Determine a target training set from the determined first training set, whose data distribution satisfies a second condition with that of the second data set.

[0082] For example, the second condition is that the distribution is most similar. During implementation, the electronic device may use the ks_2samp method to check whether the first training set and the second data set obey the same or similar distribution.

[0083] Step 503: Use the second data set and the target training set as a second training set corresponding to the second disk type to train a second initial classifier to obtain the second target classifier.

[0084] In the embodiment of the present application, a classifier may also be provided for disk types with a small amount of data. In order to compensate for the low accuracy of the classifier finally trained due to the small amount of data, data with similar distribution can be selected from representative disk types, and this part of data can be used to train the classifier corresponding to the disk type with a small amount of data; in this way, the classification accuracy of the second target classifier can be improved.

[0085] When implementing step 503, a transfer learning algorithm may be used to train the second initial classifier using the second training set, thereby obtaining a second target classifier. In the embodiment of the present application, the type of transfer learning algorithm is not limited. For example, the transfer learning algorithm is an adaptive boosting (Adaptive Boosting, AdaBoost) algorithm or a TrAdaBoost algorithm. Taking the TrAdaBoost algorithm as an example, Figure 5B As shown, step 503 can be implemented by following steps 5031 to 5035:

[0086] Step 5031, assigning an initial weight to the attribute mean vector corresponding to each disk in the second training set;

[0087] Step 5032: training the second initial classifier according to the second training set, the weight of each attribute mean vector and the weight distribution of the second training set to obtain an intermediate classifier;

[0088] Step 5033, counting the error rate of the intermediate classifier on the second attribute mean vector of the second training set;

[0089] Step 5034, adjusting the weights of the misclassified attribute mean vectors in the second training set according to the error rate, so that the weight of the misclassified first attribute mean vector is reduced and the weight of the misclassified second attribute mean vector is increased;

[0090] Step 5035: Return to train the intermediate classifier according to the second training set after adjusting the weights, until the number of returns reaches a set value, and use the intermediate classifier obtained from the last training as the second target classifier.

[0091] It should be noted that returning to train the intermediate classifier, i.e. returning to step 5032, the only difference is that what is inputted at this time is the second training set after adjusting the weights, the adjusted weights of each attribute mean vector, and the weight distribution of the second training set determined based on the adjusted weights.

[0092] It should be noted that for a known disk type with an existing target classifier, a better prediction effect can be obtained directly through the corresponding target classifier. Taking ST4000DM000 as an example, its fault prediction accuracy can reach more than 90%. However, the SMART attribute value distribution of the disk with the same ID produced by the same manufacturer is also different. Taking the SMART attribute SMART1: underlying data read error rate (Raw_Read_Error_Rate) as an example, as shown in Tables 3 and 4, the data distribution of the same attribute of different types of disks is different, and different types of disks produced by other manufacturers also have similar phenomena; among them, Table 3 shows the variance and mean corresponding to each type of disk based on the original SMART data of the disk; Table 4 shows the mean and variance after normalization.

[0093] Table 3

[0094] Disk Model Mean variance ST4000DM000 122109678.179 4.9678 ST4000DM002 121967071.283 5.0242 ST8000NM0055 120862664.365 5.3390 ST6000DX000 119434977.04 5.1223 ST4000DX000 121486491.335 4.9441

[0095] Table 4

[0096] Disk Model Mean variance ST4000DM000 115.3013 14.9689 ST4000DM002 80.0266 20.8823 ST8000NM0055 80.1360 25.2788 ST6000DX000 114.6478 25.0776 ST4000DX000 115.2888 14.9036

[0097] Training a classifier on the same type of disks and then using the classifier to classify the disks is based on the fact that the data of the disks follow the same distribution. However, the classification effect of the classifier on another set of data with different distributions, such as another type of disk data, is obviously poor. However, for disks produced by the same manufacturer, these data with different distributions are actually of reference value. How to use these data with different distributions to train the classifier so that it can be more effective on new data can be solved by using the idea of ​​the TrAdaBoost algorithm. The basic principle is to put the SMART data of two types of disks, A and B, in the training set, where the distribution of B is the same as the distribution of the new disk type, and the distribution of A is different from the distribution of the new disk type. When the classifier is trained with the training set to classify the test set, when a sample in B is misclassified, a larger weight is given to this sample. When a sample in A is misclassified, the weight of the sample is reduced and the next iteration is entered.

[0098] Based on the above analysis, the following will describe an exemplary application of the present application embodiment in a practical application scenario. Figure 6 As shown, a disk failure prediction method is provided, which mainly includes a training phase and a prediction phase; wherein the training phase includes the following steps 601 to 610:

[0099] Step 601 , select a relatively representative disk type, that is, a disk type with relatively more historical SMART data of failed disks in the corresponding data set.

[0100] Step 602, for each selected disk type, filter out key attributes and obtain corresponding historical SMART data;

[0101] Step 603, preprocessing the historical SMART data corresponding to each selected key attribute by using a time series exponential smoothing algorithm to obtain a first data set;

[0102] Step 604, extracting data corresponding to each failed disk from the first data set, that is, obtaining a positive sample set;

[0103] Step 605, clustering the first data set according to the size of the positive sample set to obtain a negative sample set of a size equivalent to that of the positive sample set; that is, the negative sample set includes sample data of healthy disks;

[0104] Step 606, training the first initial classifier according to the positive sample set, the negative sample set and the RGF classification algorithm to obtain a first target classifier;

[0105] Step 607, counting the distribution characteristic values ​​of the attribute mean sequence of each key attribute of the disk type, such as the mean, variance, range, dispersion, moment, skewness, and kurtosis;

[0106] Step 608, for a disk type with a small amount of data, statistics are collected on data distribution characteristics of the disk type;

[0107] Step 609, selecting the most similar distribution from the training set corresponding to the representative disk type by the ks_2samp method;

[0108] Step 610: Based on the training set and the SMART data corresponding to the disk type with a smaller amount of data, the classifier x (ie, the second initial classifier) ​​is trained by the TrAdaBoost method to obtain the classifier x_1, ie, the second target classifier.

[0109] The prediction stage may include the following steps 621 to 623:

[0110] Step 621, determine whether the disk type of the disk to be tested exists in the training set; if yes, execute step 622; otherwise, execute step 623;

[0111] Step 622, determining that the disk to be tested is a disk of a known type, preprocessing the current attribute data of the disk by using a time series exponential smoothing algorithm, and then using the first target classifier corresponding to the disk to predict the failure of the disk;

[0112] Step 623, determine that the disk to be tested is a disk of unknown type, calculate the cosine value between its current attribute data and the training set corresponding to each target classifier, and select the target classifier with the closest distance to perform fault prediction on the disk to be tested.

[0113] It should be noted that the disk SMART data during cluster operation can be recorded in the sqlite database file. In order to cover as many disk types as possible, the TrAdaBoost algorithm is required. During operation, new target classifiers are continuously built based on cluster disk training. The training process consumes some computing resources, so when computing resources are tight, only data collection is done, which can be uploaded to the cloud for separate calculations at regular intervals, and then the target classifier trained on the cloud is synchronized.

[0114] The advantage of pre-trained classifiers is that there is a sufficient amount of public data available, but the disadvantage is that they are difficult to customize. For example, the disk SMART data provided by Backblaze generally only has normalized values ​​and raw data values, and the worst value, which has a high degree of reference, is not provided. The disk I / O of some clusters is relatively intensive, or values ​​such as memory usage can actually be added to the classifier training process as customized requirements.

[0115] In an embodiment of the present application, a method for processing disk SMART monitoring information is provided, which can realize predictive analysis of monitoring data of more disk types when the types of training set samples are relatively small. The method is: select a target classifier with appropriate distribution through ks_2samp, and build fewer models to cover as many disk types as possible through the TrAdaBoost algorithm, providing a disk prediction function with high generalization. For new types of disk data, the most approximate distribution model is selected by calculating the cosine value, the data is supplemented by missing values, and the classifier corresponding to the selected distribution model is used for prediction.

[0116] Compared with the related art, the embodiment of the present application increases the compatible processing capability of SMART monitoring information of multiple disk types, and can realize fault prediction for multiple disk types when the training set is limited. The related disk fault prediction technology solves the prediction problem of a single disk model, has a high degree of dependence on the training set, and does not have a method for processing SMART data of multiple disk types. It is more practical for large clusters that have a preference for disk types.

[0117] If you need to provide disk failure prediction function for different clusters, it will be a big challenge to collect data during modeling. It is equivalent to first building a database that covers all disk types on the market, which takes a long time. According to the statistical analysis of currently available public data, the proportion of various disk types varies greatly. If all disk types are modeled uniformly, their detection rate will be reduced. If individual disk types are modeled, their scalability will be reduced and generalization will be poor. If the disk failure prediction function does not have too high requirements for the disk type used by the cluster, it will have relatively high versatility.

[0118] The method provided in the embodiment of the present application will train some basic prediction models before deployment, and will not occupy too many cluster computing resources. At the same time, the server performance monitoring data collected by the monitoring management platform during the normal operation of the cluster will be stored in the sqlite file database with the disk monitoring data while providing monitoring and alarm services, so as to collect data, optimize and train classifiers, customize training classifiers, etc. This achieves the effect of increasing prediction accuracy as cluster data increases without affecting the normal business of the cluster.

[0119] Based on the foregoing embodiments, an embodiment of the present application provides a disk failure prediction device, which includes the modules included and the units included in the modules, and can be implemented by a processor in an electronic device; of course, it can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP) or a field programmable gate array (FPGA), etc.

[0120] Fig. 7A FIG. 1 is a schematic diagram of the structure of the disk failure prediction device according to an embodiment of the present application. Fig. 7A As shown, the device 700 includes an acquisition module 701, a determination module 702 and a prediction module 703; wherein,

[0121] The acquisition module 701 is used to acquire the current attribute data of the disk to be tested;

[0122] The acquisition module 701 is further used to acquire a training set used when training each target classifier when there is no target classifier corresponding to the disk to be tested among the multiple target classifiers obtained by pre-training;

[0123] A determination module 702, used to determine the similarity between the current attribute data and each of the training sets;

[0124] The prediction module 703 is used to process the current attribute data using the target classifier whose similarity satisfies the first condition to obtain a fault prediction result of the disk to be tested.

[0125] In some embodiments, the determination module 702 is used to: determine the distribution characteristic value of the attribute mean sequence of each key attribute in each of the training sets; use the distribution characteristic value of each key attribute in each of the training sets as the distribution characteristic vector of the corresponding training set; determine the distance between the current attribute data and each of the distribution characteristic vectors to obtain the corresponding similarity.

[0126] In some embodiments, the determination module 702 is used to: determine a target attribute whose attribute value does not satisfy a specific condition from the current attribute data; update the attribute value of the target attribute in the current attribute data to the distribution feature value corresponding to the target attribute in the distribution feature vector; and determine the distance between the distribution feature vector and the processed current attribute data.

[0127] In some embodiments, the determination module 702 is also used to: obtain a first type identifier of the disk to be tested; obtain a second type identifier of the disk type corresponding to each of the target classifiers; wherein each of the target classifiers is used to perform fault prediction on a disk of the corresponding disk type; and when the first type identifier is different from each of the second type identifiers, determine that there is no target classifier corresponding to the disk to be tested among the pre-trained multiple target classifiers.

[0128] In some embodiments, the plurality of target classifiers include at least one first target classifier and at least one second target classifier; Figure 7B As shown, the fault prediction device 700 also includes a training module 704, which is used to: perform weighted averaging processing on the historical attribute value sequence of each key attribute of each first disk obtained, corresponding to the first attribute mean of the key attribute; wherein each first disk belongs to a first disk type, the total number of the first disks is greater than a first threshold, and the number of failed disks in all the first disks is greater than a second threshold; add the first attribute mean of each key attribute of each first disk as the first attribute mean vector of the corresponding disk to the first data set; determine the first training set corresponding to the first disk type based on the first data set; use the first training set to train the first initial classifier to obtain the first target classifier.

[0129] In some embodiments, after obtaining each of the first target classifiers, the training module 704 is also used to train the second target classifier, that is: weighted averaging the historical attribute value sequence of each key attribute of each second disk obtained to obtain a second data set containing the second attribute mean vector of each second disk; wherein each of the second disks belongs to the second disk type, and the number of failed disks in all the second disks is less than or equal to the first threshold; from the determined first training set, determine a target training set that is most similar to the data distribution of the second data set; use the second data set and the target training set as the second training sets corresponding to the second disk type, train the second initial classifier, and obtain the second target classifier.

[0130] In some embodiments, the training module 704 is used to: assign initial weights to the attribute mean vectors corresponding to each disk in the second training set; train the second initial classifier according to the second training set, the weights of each attribute mean vector and the weight distribution of the second training set to obtain an intermediate classifier; count the error rate of the intermediate classifier on the second attribute mean vector of the second training set; adjust the weights of the misclassified attribute mean vectors in the second training set according to the error rate so that the weights of the misclassified first attribute mean vectors are reduced and the weights of the misclassified second attribute mean vectors are increased; return to training the intermediate classifier according to the second training set after the weight adjustment until the number of returns reaches a set value, and use the intermediate classifier obtained from the last training as the second target classifier.

[0131] The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0132] It should be noted that, in the embodiment of the present application, if the above-mentioned disk failure prediction method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application can be essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable an electronic device (which can be a laptop, a desktop computer, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile disk, a read-only memory (ROM), a magnetic disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific combination of hardware and software.

[0133] Correspondingly, an embodiment of the present application provides an electronic device, Figure 8 A hardware entity diagram of an electronic device according to an embodiment of the present application is shown in FIG. Figure 8 As shown, the hardware entity of the electronic device 800 includes: a memory 801 and a processor 802, wherein the memory 801 stores a computer program that can be run on the processor 802, and the processor 802 implements the steps in the disk failure prediction method provided in the above embodiment when executing the program.

[0134] The memory 801 is configured to store instructions and applications executable by the processor 802, and can also cache data to be processed or processed by the processor 802 and various modules in the electronic device 800 (for example, image data, audio data, voice communication data, and video communication data), which can be implemented through flash memory (FLASH) or random access memory (Random Access Memory, RAM).

[0135] Correspondingly, an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps in the disk failure prediction method provided in the above embodiment are implemented.

[0136] It should be noted here that the description of the above storage medium, chip and terminal device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium, chip and terminal device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0137] It should be understood that "one embodiment" or "an embodiment" or "some embodiments" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" or "in some embodiments" appearing throughout the specification do not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The above-mentioned sequence numbers of the embodiments of the present application are for description only and do not represent the advantages and disadvantages of the embodiments.

[0138] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0139] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The embodiments of the touch screen system described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as: multiple modules or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or modules can be electrical, mechanical or other forms.

[0140] The modules described above as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules; they may be located in one place or distributed on multiple network units; some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0141] In addition, all functional modules in the embodiments of the present application may be integrated into one processing unit, or each module may be a separate unit, or two or more modules may be integrated into one unit; the above-mentioned integrated modules may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0142] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, etc., various media that can store program codes.

[0143] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can be essentially or partly reflected in the form of a software product that contributes to the relevant technology. The computer software product is stored in a storage medium, including several instructions to enable an electronic device (which can be a laptop, desktop computer, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0144] The methods disclosed in several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0145] The features disclosed in several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0146] The features disclosed in several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0147] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A disk failure prediction method, characterized in that: The method comprises: Get the current attribute data of the disk to be tested; When there is no target classifier corresponding to the disk to be tested among the multiple target classifiers obtained by pre-training, obtaining a corresponding training set used when training each of the target classifiers; Determining the similarity between the current attribute data and each of the training sets; Using the target classifier whose similarity satisfies the first condition, the current attribute data is processed to obtain a fault prediction result of the disk to be tested; Wherein, determining the similarity between the current attribute data and each of the training sets includes: Determining a distribution characteristic value of the attribute mean sequence of each key attribute in each of the training sets; Using the distribution characteristic value of each key attribute in each of the training sets as the distribution characteristic vector of the corresponding training set; The distance between the current attribute data and each of the distribution feature vectors is determined to obtain a corresponding similarity.

2. The method according to claim 1, characterized in that The determining of the distance between the current attribute data and each of the distribution feature vectors comprises: Determine, from the current attribute data, a target attribute whose attribute value does not satisfy a specific condition; Updating the attribute value of the target attribute in the current attribute data to the distribution feature value corresponding to the target attribute in the distribution feature vector; The distance between the distribution feature vector and the updated current attribute data is determined.

3. The method according to claim 1, characterized in that The method further comprises: Obtaining a first type identifier of the disk to be tested; Obtaining a second type identifier of a disk type corresponding to each of the target classifiers; wherein each of the target classifiers is used to perform fault prediction on a disk of the corresponding disk type; In a case where the first type identifier is different from each of the second type identifiers, it is determined that there is no target classifier corresponding to the disk to be tested among the plurality of pre-trained target classifiers.

4. The method according to any one of claims 1 to 3, characterized in that: The multiple target classifiers include at least one first target classifier; wherein the training process corresponding to the first target classifier includes: Performing weighted average processing on the acquired historical attribute value sequence of each key attribute of each first disk to obtain a first attribute mean value of the corresponding key attribute; wherein each first disk belongs to a first disk type, the total number of the first disks is greater than a first threshold, and the number of failed disks in all the first disks is greater than a second threshold; Adding the first attribute mean of each key attribute of each of the first disks as the first attribute mean vector of the corresponding disk to the first data set; Determine, according to the first data set, a first training set corresponding to the first disk type; The first initial classifier is trained using the first training set to obtain the first target classifier.

5. The method according to claim 4, characterized in that The multiple target classifiers also include at least one second target classifier; after obtaining each of the first target classifiers, training a second initial classifier according to a first training set corresponding to the first target classifier to obtain the second target classifier, including: Performing weighted average processing on the historical attribute value sequence of each key attribute of each second disk, to obtain a second attribute mean value of the corresponding key attribute; wherein each second disk belongs to the second disk type, and the number of failed disks in all the second disks is less than or equal to the first threshold; taking the second attribute mean of each key attribute of each second disk as the second attribute mean vector of the corresponding disk, and adding it to the second data set; Determine, from the determined first training set, a target training set whose data distribution satisfies a second condition with that of the second data set; The second data set and the target training set are used as the second training set corresponding to the second disk type, and the second initial classifier is trained to obtain the second target classifier.

6. The method according to claim 5, characterized in that Using the second training set to train the second initial classifier to obtain the second target classifier, comprising: Assigning an initial weight to the attribute mean vector corresponding to each disk in the second training set; Training the second initial classifier according to the second training set, the weight of each of the attribute mean vectors and the weight distribution of the second training set to obtain an intermediate classifier; Counting the error rate of the intermediate classifier on the second attribute mean vector of the second training set; According to the error rate, adjusting the weights of the misclassified attribute mean vectors in the second training set so that the weight of the misclassified first attribute mean vector is reduced and the weight of the misclassified second attribute mean vector is increased; According to the second training set after adjusting the weight, return to train the intermediate classifier until the number of returns reaches a set value, and use the intermediate classifier obtained from the last training as the second target classifier.

7. A disk failure prediction device, characterized in that: include: The acquisition module is used to obtain the current attribute data of the disk to be tested; The acquisition module is further used to acquire a corresponding training set used when training each target classifier when there is no target classifier corresponding to the disk to be tested among the multiple target classifiers obtained by pre-training; A determination module, used to determine the similarity between the current attribute data and each of the training sets; wherein the determination of the similarity between the current attribute data and each of the training sets includes: determining the distribution characteristic value of the attribute mean sequence of each key attribute in each of the training sets; using the distribution characteristic value of each key attribute in each of the training sets as the distribution characteristic vector of the corresponding training set; determining the distance between the current attribute data and each of the distribution characteristic vectors to obtain the corresponding similarity; The prediction module is used to process the current attribute data using a target classifier whose similarity satisfies a first condition, so as to obtain a fault prediction result of the disk to be tested.

8. An electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the program, the steps in the disk failure prediction method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps in the disk failure prediction method according to any one of claims 1 to 6 are implemented.

Images