Disk failure prediction method and device

Abstract

The present invention discloses a disk failure prediction method and device, which obtains information of disks of different models, and constructs a training data set based on the disk information; constructs a deep neural network model including a feature network structure and a classification network structure; wherein the feature network structure is used to extract feature information of the disk based on the disk information; the classification network structure is used to determine whether the disk is faulty based on the feature information of the disk; the deep neural network model is trained based on the training data set to obtain a trained disk failure prediction model; obtains information of a target disk to be predicted for failure, and inputs the target disk information into the disk failure prediction model to obtain a failure prediction result of the target disk. It can be seen that the disk failure prediction model of the present application can simultaneously predict failures of disks of different models, and has good versatility.

Description

Technical Field

[0001] The present invention relates to the field of storage, and in particular to a disk failure prediction method and device. Background Art

[0002] With the development of emerging technologies such as cloud computing and blockchain, the demand for storage systems is increasing. At present, in storage systems, disks are still the mainstream storage devices. In existing solutions, SMART (Self-Monitoring Analysis and Reporting Technology) technology is usually used to predict whether the disk in the storage system is faulty. The fault prediction principle is: set normal threshold intervals for each type of index data of the disk. When the disk has an index value outside the normal threshold interval, it is considered that the disk is pre-failed. However, there may be more than one disk model in the same storage system, and the disk models of different storage systems may also be different. The distribution of various index data of disks of different models is not the same, so it is often necessary to set different normal threshold intervals for various index data of disks of different models, resulting in poor versatility of this disk failure prediction method.

[0003] Therefore, how to provide a solution to the above technical problems is a problem that technicians in this field need to solve at present. Summary of the invention

[0004] The purpose of the present invention is to provide a disk failure prediction method and device. The disk failure prediction model can simultaneously perform failure prediction on disks of different models and has good versatility.

[0005] In order to solve the above technical problems, the present invention provides a disk failure prediction method, comprising:

[0006] Obtaining information of disks of different models, and constructing a training data set based on the information of the disks;

[0007] Constructing a deep neural network model including a feature network structure and a classification network structure; wherein the feature network structure is used to extract feature information of the disk based on the information of the disk; and the classification network structure is used to determine whether the disk is faulty based on the feature information of the disk;

[0008] Training the deep neural network model based on the training data set to obtain a trained disk failure prediction model;

[0009] The information of the target disk to be predicted for failure is obtained, and the information of the target disk is input into the disk failure prediction model to obtain a failure prediction result of the target disk.

[0010] Optionally, obtaining information of disks of different models and constructing a training data set based on the information of the disks includes:

[0011] Obtain various indicator data of disks of different models at different times, and obtain fault information of the disks;

[0012] According to the fault information of the disk and the preset fault prediction advance time, each type of index data of the disk is labeled with a label indicating whether the index data is fault data;

[0013] Various index data of the disk and their corresponding labels are combined to form the training data set.

[0014] Optionally, obtaining information of disks of different models and constructing a training data set based on the information of the disks further includes:

[0015] After obtaining various indicator data of disks of different models at different times, before combining the various indicator data of the disks and their corresponding labels, invalid data in the various indicator data of the disks are erased, and the various indicator data of the disks after erasure are normalized, so as to combine the various indicator data of the disks after normalization and their corresponding labels to form the training data set.

[0016] Optionally, a deep neural network model including a feature network structure and a classification network structure is constructed; and the deep neural network model is trained based on the training data set to obtain a trained disk failure prediction model, including:

[0017] Constructing a feature network structure composed of a first fully connected layer and a plurality of residual layers in sequence; wherein the feature network structure is used to extract high-dimensional implicit features corresponding to the index data based on various index data of the disk;

[0018] Constructing a classification network structure composed of a second fully connected layer; wherein the output dimension of the classification network structure is 2, corresponding to fault and normal respectively;

[0019] Combining the constructed feature network structure and the classification network structure to obtain a deep neural network model;

[0020] Training the deep neural network model based on the training data set to obtain a trained deep neural network model;

[0021] The high-dimensional implicit features of the disk and their corresponding labels are combined to form a new training data set, and an XGBoost classifier used to replace the second fully connected layer is trained based on the new training data set to obtain a trained XGBoost classifier;

[0022] The trained XGBoost classifier replaces the second fully connected layer in the trained deep neural network model, and the replaced deep neural network model is used as the disk failure prediction model.

[0023] Optionally, a feature network structure consisting of a first fully connected layer and a plurality of residual layers is constructed, including:

[0024] Based on F out =X*W+b constructs the first fully connected layer; where F out is the output vector of the first fully connected layer; X is the input vector of the first fully connected layer; W is the network weight of the first fully connected layer; b is the bias of the first fully connected layer;

[0025] Based on x a+1 =x a +F(x a ,W a )、F(x a ,W a )=Relu(x a *W a ) constructs L residual layers; where x a+1 is the output of the a-th residual layer; x a is the input of the a-th residual layer; F(x a ,W a ) is the residual learning function of the a-th layer; Relu is the activation function, Relu(x)=max(0,x); 1≤a≤L and a is an integer;

[0026] The constructed first fully connected layer and the L residual layers are combined to obtain the feature network structure.

[0027] Optionally, training the deep neural network model based on the training data set to obtain a trained deep neural network model includes:

[0028] Inputting various index data of the disk in the training data set into the deep neural network model to obtain a fault prediction result of the disk;

[0029] Substituting the fault prediction result of the disk and the labels corresponding to various index data of the disk into a preset loss calculation function to perform loss calculation to obtain a first loss;

[0030] With reducing the first loss to 0 as the optimization goal, the adjustable parameters of the neural network model are optimized and adjusted using a preset back propagation algorithm until all the training data sets are trained on the neural network model to obtain a trained deep neural network model.

[0031] Optionally, training an XGBoost classifier for replacing the second fully connected layer based on the new training data set to obtain a trained XGBoost classifier includes:

[0032] Inputting each high-dimensional implicit feature of the disk in the new training data set into the XGBoost classifier to obtain a fault classification result of the disk;

[0033] Substituting the fault classification result of the disk and the labels corresponding to each high-dimensional implicit feature of the disk into a preset loss calculation function to perform loss calculation to obtain a second loss;

[0034] With reducing the second loss to 0 as the optimization goal, the adjustable parameters of the XGBoost classifier are optimized and adjusted using a preset back propagation algorithm until all of the new training data sets are trained on the XGBoost classifier to obtain a trained XGBoost classifier.

[0035] Optionally, before inputting the information of the target disk into the disk failure prediction model, the disk failure prediction method further includes:

[0036] Learning the optimal structural parameters of the disk failure prediction model according to a preset heuristic genetic algorithm;

[0037] The structural parameters of the disk failure prediction model are adjusted according to the optimal structural parameters, so as to input the information of the target disk into the disk failure prediction model of the optimal structure.

[0038] Optionally, learning the optimal structural parameters of the disk failure prediction model according to a preset heuristic genetic algorithm includes:

[0039] The structural parameters of the disk failure prediction model are combined into a parameter vector, and the parameter vector is randomly initialized, and the initialized parameter vector is added to a preset vector priority queue; wherein the vector priority queue is a large root heap structure, and the sorting key of the large root heap structure is the false alarm rate score value corresponding to each parameter vector;

[0040] Adjusting the structural parameters of the disk failure prediction model according to the initialized parameter vector to obtain the false alarm rate score value of the disk failure prediction model under the current structural parameters, and returning to the step of randomly initializing the parameter vector until the number of initializations reaches a preset number threshold;

[0041] A preset first number of parameter vector pairs are randomly selected from the vector priority queue, and each of the parameter vector pairs is sorted according to param new=(param1+param2) / 2 to perform crossover operation to obtain the parameter vector param of the disk failure prediction model. new The new false alarm rate score value under the condition; wherein param1 and param2 are two parameter vectors in each of the parameter vector pairs;

[0042] sorting the new false alarm rate score value and the false alarm rate score value currently corresponding to the vector priority queue from small to large, and retaining only a preset second number of parameter vectors that are sorted first in the vector priority queue;

[0043] Traverse all parameter vectors in the vector priority queue and based on the mutation probability p=e iteration / 3 Each of the parameter vectors param is divided into var = param + rand * step to mutate, so as to obtain the disk failure prediction model in the parameter vector param var The mutation false alarm rate score value under this condition; where iteration is the number of iterations; step is the basic unit of mutation of each parameter vector; rand is the random number generated in each mutation process;

[0044] Sorting the mutation false alarm rate score value and the false alarm rate score value currently corresponding to the vector priority queue from small to large, retaining only the preset second number of parameter vectors sorted in front in the vector priority queue, and returning to execute the step of randomly selecting a preset first number of parameter vector pairs from the vector priority queue to enter the next iteration until the number of iterations reaches a preset iteration number threshold;

[0045] After all iterations are completed, a parameter vector with the smallest false alarm rate score is selected from the finally obtained vector priority queue as the optimal structural parameter.

[0046] In order to solve the above technical problems, the present invention further provides a disk failure prediction device, comprising:

[0047] Memory for storing computer programs;

[0048] A processor is used to implement the steps of any one of the above disk failure prediction methods when executing the computer program.

[0049] The present invention provides a disk failure prediction method, which obtains information of disks of different models, and constructs a training data set based on the disk information; constructs a deep neural network model including a feature network structure and a classification network structure; wherein the feature network structure is used to extract feature information of the disk based on the disk information; the classification network structure is used to determine whether the disk is faulty based on the feature information of the disk; the deep neural network model is trained based on the training data set to obtain a trained disk failure prediction model; obtains information of a target disk to be predicted for failure, and inputs the target disk information into the disk failure prediction model to obtain a failure prediction result of the target disk. It can be seen that the disk failure prediction model of the present application can simultaneously predict failures of disks of different models, and has good versatility.

[0050] The present invention also provides a disk fault prediction device, which has the same beneficial effects as the above fault prediction method. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the prior art and the drawings required for use in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0052] Figure 1 A flowchart of a disk failure prediction method provided by an embodiment of the present invention;

[0053] Figure 2 A schematic diagram of a characteristic network structure provided by an embodiment of the present invention;

[0054] Figure 3 A schematic diagram of a disk failure prediction model provided by an embodiment of the present invention;

[0055] Figure 4 A schematic diagram of a residual layer provided by an embodiment of the present invention;

[0056] Figure 5 A graph showing the variation of mutation probability with the number of iterations provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0057] The core of the present invention is to provide a disk failure prediction method and device. The disk failure prediction model can simultaneously perform failure prediction on disks of different models and has good versatility.

[0058] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] Please refer to Figure 1 , Figure 1 A flowchart of a disk failure prediction method provided by an embodiment of the present invention.

[0060] The disk failure prediction method includes:

[0061] Step S1: Obtain information of disks of different models, and construct a training data set based on the disk information.

[0062] Specifically, the present application obtains information about disks of different models to construct a training data set DataSet1 based on the information about disks of different models (used for training a deep neural network model for predicting disk failures)

[0063] Step S2: construct a deep neural network model including a feature network structure and a classification network structure, and train the deep neural network model based on a training data set to obtain a trained disk failure prediction model.

[0064] Specifically, the present application constructs a deep neural network model including a feature network structure and a classification network structure, wherein the feature network structure is used to extract feature information of the disk based on the information of the disk; the classification network structure is used to determine whether the disk is faulty based on the feature information of the disk, that is, the information of a disk is input into the deep neural network model, and the deep neural network model can output the fault prediction result of this disk.

[0065] This application trains a deep neural network model based on the training data set DataSet1. The purpose is to improve the accuracy of model prediction by training the deep neural network model, and finally obtain a disk failure prediction model with higher accuracy.

[0066] Step S3: Obtain information of the target disk to be predicted for failure, and input the information of the target disk into a disk failure prediction model to obtain a failure prediction result of the target disk.

[0067] Specifically, after obtaining a disk failure prediction model with high accuracy, the present application can obtain information of the target disk to be predicted for failure, and then input the information of the target disk to be predicted for failure into the disk failure prediction model to obtain a failure prediction result of the target disk.

[0068] It can be seen that the disk failure prediction model of the present application can simultaneously predict failures of disks of different models and has good versatility.

[0069] Based on the above embodiments:

[0070] As an optional embodiment, obtaining information of disks of different models and constructing a training data set based on the information of the disks includes:

[0071] Obtain various indicator data of disks of different models at different times, and obtain disk failure information;

[0072] According to the fault information of the disk and the preset fault prediction advance time, each type of index data of the disk is labeled with a label indicating whether the index data is fault data;

[0073] Combine various index data of the disk and their corresponding labels to form a training data set.

[0074] Specifically, the construction process of the training data set of the present application includes: 1) obtaining various indicator data of disks of different models at different times (i.e., SMART data information of disks of different models at different times, and indicator data can also be obtained from the Backblaze (cloud storage) open source disk data set), and obtaining fault information of disks of different models (the date on which the disk failure occurred can be known); 2) according to the fault information of any model of disk and the preset fault prediction advance time, various indicator data of the disk of this model are labeled one by one to indicate whether the indicator data is fault data. For example, if the fault prediction advance time is set to 14 days, various indicator data of the disk for a total of 14 days on and before the date of the failure are labeled one by one to indicate that the indicator data is fault data (such as 1), and the indicator data of the disk at the rest of the time are labeled one by one to indicate that the indicator data is normal data (such as 0); 3) various indicator data of disks of different models at different times and their corresponding labels are combined to form a training data set DataSet1.

[0075] As an optional embodiment, obtaining information of disks of different models and constructing a training data set based on the information of the disks further includes:

[0076] After obtaining various indicator data of disks of different models at different times, before combining the various indicator data of the disks and their corresponding labels, invalid data in the various indicator data of the disks are erased, and the various indicator data of the erased disks are normalized, so as to combine the various indicator data of the normalized disks and their corresponding labels to form a training data set.

[0077] Furthermore, after obtaining various indicator data of disks of different models at different times, the present application can also erase invalid data in various indicator data of disks of different models to avoid invalid data affecting model training; then, various indicator data of disks of different models after erasure are processed according to x=(x i -x min ) / (x max -x min ) is normalized to facilitate subsequent data processing, where x i is the index data to be normalized; x max is the maximum index value of the index data to be normalized; x min is the minimum index value of the index data to be normalized; x is the normalized result of the index data to be normalized; finally, the normalized index data of different models of disks at different times and their corresponding labels are combined to form the training data set DataSet1.

[0078] As an optional embodiment, a deep neural network model including a feature network structure and a classification network structure is constructed; the deep neural network model is trained based on a training data set to obtain a trained disk failure prediction model, including:

[0079] Constructing a feature network structure composed of a first fully connected layer and a plurality of residual layers in sequence; wherein the feature network structure is used to extract high-dimensional implicit features corresponding to the index data one by one based on various index data on the disk;

[0080] Construct a classification network structure consisting of the second fully connected layer; the output dimension of the classification network structure is 2, corresponding to fault and normal respectively;

[0081] Combine the constructed feature network structure and classification network structure to obtain a deep neural network model;

[0082] The deep neural network model is trained based on the training data set to obtain a trained deep neural network model;

[0083] The high-dimensional implicit features of the disk and their corresponding labels are combined to form a new training data set, and the XGBoost classifier used to replace the second fully connected layer is trained based on the new training data set to obtain a trained XGBoost classifier;

[0084] The trained XGBoost classifier is used to replace the second fully connected layer in the trained deep neural network model, and the replaced deep neural network model is used as the disk failure prediction model.

[0085] Specifically, the construction process of the disk failure prediction model of the present application includes: 1) Figure 2 As shown in FIG. 1 , a feature network structure FeatureNet is constructed which is composed of a first FC (Fully Connected) layer and multiple RESNET (Residual Network) layers in sequence; wherein the feature network structure is used to extract high-dimensional implicit features feature corresponding to various index data x of the disk based on various index data x of the disk. h , that is, feature h = x·FeatureNet; 2) Construct a classification network structure ClassifierNet composed of the second FC layer, that is, the output of the classification network structure out = feature h ·ClassifierNet; It should be noted that the output out of the classification network structure needs to perform softmax (a function used for multi-classification) processing on the output of ClassifierNet. The dimension of the output out is 2, corresponding to fault (1) and normal (0) respectively; 3) Combine the constructed feature network structure (first FC layer + multiple RESNET layers) and the classification network structure (second FC layer) to obtain a deep neural network model; 4) Train the deep neural network model based on the training data set DataSet1 to obtain a trained deep neural network model; 5) Different types of disks have corresponding labels for various indicator data at different times, and different types of disks have corresponding high-dimensional implicit features for various indicator data at different times. Then, the high-dimensional implicit features of different types of disks at different times also correspond to labels one by one. Combine the high-dimensional implicit features of different types of disks at different times and their corresponding labels to form a new training data set DataSet2; 6) Based on the new training data set DataSet2, XGBoost (ExtremeGradient Boosting, extreme gradient boosting) classifier is trained to obtain a trained XGBoost classifier; 7) The trained XGBoost classifier replaces the second FC layer in the trained deep neural network model, and the replaced deep neural network model is used as the disk failure prediction model, that is, the final disk failure prediction model is a feature network structure FeatureNet+XGBoost classifier (such as Figure 3 It is used for disk failure prediction.

[0086] As an optional embodiment, a feature network structure consisting of a first fully connected layer and a plurality of residual layers in sequence is constructed, including:

[0087] Based on F out=X*W+b constructs the first fully connected layer; where F out is the output vector of the first fully connected layer; X is the input vector of the first fully connected layer; W is the network weight of the first fully connected layer; b is the bias of the first fully connected layer;

[0088] Based on x a+1 =x a +F(x a ,W a )、F(x a ,W a )=Relu(x a *W a ) constructs L residual layers; where x a+1 is the output of the a-th residual layer; x a is the input of the a-th residual layer; F(x a ,W a ) is the residual learning function of the a-th layer; Relu is the activation function, Relu(x)=max(0,x); 1≤a≤L and a is an integer;

[0089] The constructed first fully connected layer and L residual layers are combined to obtain the feature network structure.

[0090] Specifically, the construction process of the feature network structure of the present application includes: 1) based on F out =X*W+b constructs the first fully connected layer. The function of the fully connected layer is to map the vector to another space, increase the complexity of the model, and improve the fit. Among them, F out is the output vector of the first fully connected layer, i.e., the input vector of the first residual layer; X is the input vector of the first fully connected layer. When the model is trained, the first fully connected layer inputs the training data set DataSet1. When the model is officially used, the first fully connected layer inputs the information of the target disk to be predicted for faults; W is the network weight of the first fully connected layer; b is the bias of the first fully connected layer; 2) Figure 4 As shown, based on x a+1 =x a +F(x a ,W a )、F(x a ,W a )=Relu(x a *W a ) constructs L residual layers; where x a+1 is the output of the a-th residual layer, i.e., the input of the a+1-th residual layer; x a is the input of the a-th residual layer; F(x a ,W a) is the residual learning function of the ath layer; Relu is the activation function, Relu(x) = max(0,x); 3) Combine the constructed first fully connected layer and L residual layers to obtain the feature network structure (the last residual layer Resnet L The output is the high-dimensional implicit feature representation of various index data of the input disk, denoted as feature h ).

[0091] As an optional embodiment, the deep neural network model is trained based on the training data set to obtain a trained deep neural network model, including:

[0092] Input various index data of the disk in the training data set into the deep neural network model to obtain the failure prediction results of the disk;

[0093] Substitute the disk failure prediction result and the labels corresponding to various index data of the disk into the preset loss calculation function to calculate the loss and obtain the first loss;

[0094] With the optimization goal of reducing the first loss to 0, the preset back-propagation algorithm is used to optimize and adjust the adjustable parameters of the neural network model until all training data sets are trained on the neural network model to obtain a trained deep neural network model.

[0095] Specifically, the training process of the deep neural network model of the present application includes: 1) inputting various index data of the disk in the training data set DataSet1 into the deep neural network model to obtain the fault prediction result of the disk; 2) substituting the fault prediction result of the disk and the labels corresponding to the various index data of the disk into the preset loss calculation function (such as the CrossEntropy function) to perform loss calculation to obtain the first loss; 3) with reducing the first loss to 0 as the optimization goal (that is, taking the disk failure prediction result of the deep neural network model and the actual failure condition of the disk as consistent as possible as the optimization goal), the preset back propagation algorithm is used to optimize and adjust the adjustable parameters of the neural network model until the training data set DataSet1 is fully trained on the neural network model to obtain a trained deep neural network model.

[0096] As an optional embodiment, the XGBoost classifier used to replace the second fully connected layer is trained based on the new training data set to obtain a trained XGBoost classifier, including:

[0097] Input each high-dimensional implicit feature of the disk in the new training data set into the XGBoost classifier to obtain the fault classification result of the disk;

[0098] Substituting the fault classification result of the disk and the labels corresponding to each high-dimensional implicit feature of the disk into the preset loss calculation function to perform loss calculation to obtain the second loss;

[0099] With the optimization goal of reducing the second loss to 0, the preset back propagation algorithm is used to optimize the adjustable parameters of the XGBoost classifier until all new training data sets are trained on the XGBoost classifier to obtain a trained XGBoost classifier.

[0100] Specifically, the training process of the XGBoost classifier of the present application includes: 1) inputting each high-dimensional implicit feature of the disk in the new training data set DataSet2 into the XGBoost classifier to obtain the fault classification result of the disk; 2) substituting the fault classification result of the disk and the labels corresponding to each high-dimensional implicit feature of the disk into a preset loss calculation function (such as the CrossEntropy function) for loss calculation to obtain a second loss; 3) with reducing the second loss to 0 as the optimization goal (that is, taking the disk fault classification result of the XGBoost classifier as consistent as possible with the actual fault condition of the disk as the optimization goal), the preset back propagation algorithm is used to optimize and adjust the adjustable parameters of the XGBoost classifier until the new training data set DataSet2 is fully trained on the XGBoost classifier to obtain a trained XGBoost classifier.

[0101] It can be seen that this application uses a deep neural network to extract high-dimensional implicit features of various index data of the disk, and uses the XGBoost classifier to replace the fully connected classification layer of the deep neural network. While retaining the excellent feature extraction capability of the deep neural network, it uses better function approximation tools to construct a better classifier, which greatly reduces the false alarm rate of faults while ensuring a relatively high prediction accuracy.

[0102] As an optional embodiment, before inputting the information of the target disk into the disk failure prediction model, the disk failure prediction method further includes:

[0103] The optimal structural parameters of the disk failure prediction model are learned according to the preset heuristic genetic algorithm;

[0104] The structural parameters of the disk failure prediction model are adjusted according to the optimal structural parameters, so as to input the information of the target disk into the disk failure prediction model with the optimal structure.

[0105] Furthermore, before inputting the information of the target disk into the disk failure prediction model, the present application can also learn the optimal structural parameters of the disk failure prediction model according to a preset heuristic genetic algorithm (such as learning the optimal parameter values ​​of the number of neurons m in the first fully connected layer, the number of layers L of the residual network, the batch size batch_size for model training, the number n of XGboost decision trees, and the maximum depth k of each decision tree), and adjust the structural parameters of the disk failure prediction model according to the optimal structural parameters, so as to input the information of the target disk into the disk failure prediction model with the optimal structure for fault prediction.

[0106] As an optional embodiment, the optimal structural parameters of the disk failure prediction model are learned according to a preset heuristic genetic algorithm, including:

[0107] The structural parameters of the disk failure prediction model are combined into a parameter vector, and the parameter vector is randomly initialized, and the initialized parameter vector is added to a preset vector priority queue; wherein the vector priority queue is a large root heap structure, and the sorting key of the large root heap structure is the false alarm rate score value corresponding to each parameter vector;

[0108] Adjust the structural parameters of the disk failure prediction model according to the initialized parameter vector to obtain the false alarm rate score value of the disk failure prediction model under the current structural parameters, and return to the step of executing the random initialization parameter vector until the number of initializations reaches a preset number threshold;

[0109] Randomly select a preset first number of parameter vector pairs from the vector priority queue, and place each parameter vector pair according to param new =(param1+param2) / 2 to perform crossover operation to obtain the disk failure prediction model in the parameter vector param new The new false alarm rate score value under param1 and param2 are the two parameter vectors in each parameter vector pair;

[0110] The new false alarm rate score value and the false alarm rate score value currently corresponding to the vector priority queue are sorted from small to large, and only the preset second number of parameter vectors sorted in front are retained in the vector priority queue;

[0111] Traverse all parameter vectors in the vector priority queue and based on the mutation probability p = e iteration / 3 Each parameter vector param is divided into var = param+rand*step to mutate the disk failure prediction model in the parameter vector param varThe mutation false alarm rate score value under this condition; where iteration is the number of iterations; step is the basic unit of mutation of each parameter vector; rand is the random number generated in each mutation process;

[0112] Sort the mutation false alarm rate score value and the false alarm rate score value currently corresponding to the vector priority queue from small to large, retain only the preset second number of parameter vectors sorted in front in the vector priority queue, and return to execute the step of randomly selecting the preset first number of parameter vector pairs from the vector priority queue to enter the next iteration until the number of iterations reaches the preset iteration number threshold;

[0113] After all iterations are completed, the parameter vector with the smallest false alarm rate score is selected from the final vector priority queue as the optimal structural parameter.

[0114] Specifically, the learning process of the optimal structural parameters of the disk failure prediction model of the present application includes: 1) combining the structural parameters of the disk failure prediction model into a parameter vector, such as the number of neurons m in the first fully connected layer, the number of layers L of the residual network, the batch size batch_size of the model training, the number n of XGboost decision trees and the maximum depth k of each decision tree into a 5-dimensional parameter vector [m, L, batch_size, n, k], denoted as param, and randomly initializing the parameter vector param, and adding the initialized parameter vector param to the preset vector priority queue queue; wherein the vector priority queue queue is a big root heap structure (the big root heap is a complete binary tree), and the big root The sorting key (node ​​key value) of the heap structure is the false alarm rate score value corresponding to each parameter vector; 2) the structural parameters (m, L, batch_size, n, k) of the disk failure prediction model are adjusted according to the initialized parameter vector param to obtain the false alarm rate score value of the disk failure prediction model under the current structural parameters, and return to the step of randomly initializing the parameter vector param until the number of initializations reaches a preset threshold (such as 10 times); 3) (2*preset first number) parameter vectors are randomly selected from the vector priority queue queue, and (2*preset first number) parameter vectors are teamed in pairs to obtain a preset first number (such as 20) of parameter vector pairs, and each parameter vector pair is sorted according to param new =(param1+param2) / 2 to perform crossover operation to obtain the disk failure prediction model in each parameter vector param new4) Sort the first number of new false alarm rate scores and the false alarm rate scores currently corresponding to the vector priority queue together from small to large, and only retain the first preset second number (such as 20) of parameter vectors param in the vector priority queue, that is, only retain the first 20 parameter vectors param with the smallest false alarm rate scores, and discard the other parameter vectors param by natural selection; 5) traverse all parameter vectors param in the vector priority queue, and based on the mutation probability p=e iteration / 3 Each parameter vector param is divided into var = param+rand*step to mutate to obtain the disk failure prediction model in each parameter vector param var The mutation false alarm rate score value under ; where iteration is the number of iterations, p is the mutation probability, and the relationship between the two is as follows Figure 5 As shown, the more iterations there are, the smaller the mutation probability is; step is the basic unit of mutation of each parameter vector, such as step = [10, 1, 16, 5, 1], the basic unit of mutation of the number of neurons m in the first fully connected layer is 10, and the same is true for other variables; rand is a random number generated in each mutation process, which can be generated from [-10, 10]; 6) Sort the false alarm rate score values ​​of each mutation and the false alarm rate score values ​​currently corresponding to the vector priority queue from small to large, and only retain the preset second number of parameter vectors in the front order in the vector priority queue, that is, only retain the first 20 parameter vectors param with the smallest false alarm rate score values, and discard the other parameter vectors param by natural selection, and return to execute step 3) to enter the next iteration until the number of iterations reaches the preset iteration number threshold; 7) After all iterations are completed, select the parameter vector with the smallest false alarm rate score value from the final vector priority queue as the optimal structural parameter.

[0115] It can be seen that this strategy solves the problem of searching for optimal model parameters in disk failure prediction. Compared with the classical genetic algorithm, the heuristic genetic algorithm in this application optimizes the mutation and selection strategies. The mutation probability decreases with the increase of the number of iterations, and it is combined with the heuristic search method of the priority queue, which can quickly search for solutions in various dimensions and approach the optimal solution as quickly as possible.

[0116] In summary, this application uses a deep neural network to learn a large number of samples of various index data features of the disk, explores the characteristics of each feature and the implicit correlation between different feature combinations, extracts higher-dimensional features, and then uses an integrated tree model to predict faults based on these features, and uses an automatic learning strategy based on a genetic algorithm to learn the optimal network structure, which greatly reduces the false alarm rate while maintaining a high recognition accuracy. The deep learning-based disk failure prediction method of this application is suitable for operation and maintenance scenarios with large disk usage and large amounts of data, such as cloud platforms, storage service providers, and government data centers. It can perform fault analysis and prediction on complex disk data information, detect problematic disks in advance, and make corresponding preparations and maintenance, thereby improving the efficiency of operation and maintenance personnel, reducing operation and maintenance costs, and thus improving product competitiveness.

[0117] The present application also provides a disk failure prediction device, comprising:

[0118] Memory for storing computer programs;

[0119] A processor is used to implement the steps of any one of the above disk failure prediction methods when executing a computer program.

[0120] For an introduction to the disk failure prediction device provided in the present application, please refer to the above-mentioned embodiment of the disk failure prediction method, and the present application will not go into details here.

[0121] It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0122] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A disk failure prediction method, characterized in that: include: Obtaining information of disks of different models, and constructing a training data set based on the information of the disks; Constructing a deep neural network model including a feature network structure and a classification network structure; wherein the feature network structure is used to extract feature information of the disk based on the information of the disk; and the classification network structure is used to determine whether the disk is faulty based on the feature information of the disk; Training the deep neural network model based on the training data set to obtain a trained disk failure prediction model; Acquire information of a target disk to be predicted for failure, and input the information of the target disk into the disk failure prediction model to obtain a failure prediction result of the target disk; Before inputting the information of the target disk into the disk failure prediction model, the disk failure prediction method further includes: Learning the optimal structural parameters of the disk failure prediction model according to a preset heuristic genetic algorithm; adjusting the structural parameters of the disk failure prediction model according to the optimal structural parameters, so as to input the information of the target disk into the disk failure prediction model of the optimal structure; The optimal structural parameters of the disk failure prediction model are learned according to a preset heuristic genetic algorithm, including: The structural parameters of the disk failure prediction model are combined into a parameter vector, and the parameter vector is randomly initialized, and the initialized parameter vector is added to a preset vector priority queue; wherein the vector priority queue is a large root heap structure, and the sorting key of the large root heap structure is the false alarm rate score value corresponding to each parameter vector; Adjusting the structural parameters of the disk failure prediction model according to the initialized parameter vector to obtain the false alarm rate score value of the disk failure prediction model under the current structural parameters, and returning to the step of randomly initializing the parameter vector until the number of initializations reaches a preset number threshold; A preset first number of parameter vector pairs are randomly selected from the vector priority queue, and each of the parameter vector pairs is sorted according to param new =(param1+param2) / 2 to perform crossover operation to obtain the parameter vector param of the disk failure prediction model. new The new false alarm rate score value under the condition; wherein param1 and param2 are two parameter vectors in each of the parameter vector pairs; sorting the new false alarm rate score value and the false alarm rate score value currently corresponding to the vector priority queue from small to large, and retaining only a preset second number of parameter vectors that are sorted first in the vector priority queue; Traverse all parameter vectors in the vector priority queue and based on the mutation probability p=e iteration / 3 Each of the parameter vectors param is divided into var =param+rand*step to mutate to obtain the parameter vector param of the disk failure prediction model var The mutation false alarm rate score value under this condition; where iteration is the number of iterations; step is the basic unit of mutation of each parameter vector; rand is the random number generated in each mutation process; Sorting the mutation false alarm rate score value and the false alarm rate score value currently corresponding to the vector priority queue from small to large, retaining only the preset second number of parameter vectors sorted in front in the vector priority queue, and returning to execute the step of randomly selecting a preset first number of parameter vector pairs from the vector priority queue to enter the next iteration until the number of iterations reaches a preset iteration number threshold; After all iterations are completed, a parameter vector with the smallest false alarm rate score is selected from the vector priority queue finally obtained as the optimal structural parameter.

2. The disk failure prediction method according to claim 1, characterized in that: Obtain information about disks of different models, and construct a training data set based on the information about the disks, including: Obtain various indicator data of disks of different models at different times, and obtain fault information of the disks; According to the fault information of the disk and the preset fault prediction advance time, each type of index data of the disk is labeled with a label indicating whether the index data is fault data; Various index data of the disk and their corresponding labels are combined to form the training data set.

3. The disk failure prediction method according to claim 2, characterized in that: Obtaining information of disks of different models and constructing a training data set based on the information of the disks, further comprising: After obtaining various indicator data of disks of different models at different times, before combining the various indicator data of the disks and their corresponding labels, invalid data in the various indicator data of the disks are erased, and the various indicator data of the disks after erasure are normalized, so as to combine the various indicator data of the disks after normalization and their corresponding labels to form the training data set.

4. The disk failure prediction method according to claim 2, characterized in that: Constructing a deep neural network model including a feature network structure and a classification network structure; training the deep neural network model based on the training data set to obtain a trained disk failure prediction model, including: Constructing a feature network structure composed of a first fully connected layer and a plurality of residual layers in sequence; wherein the feature network structure is used to extract high-dimensional implicit features corresponding to the index data based on various index data of the disk; Constructing a classification network structure composed of a second fully connected layer; wherein the output dimension of the classification network structure is 2, corresponding to fault and normal respectively; Combining the constructed feature network structure and the classification network structure to obtain a deep neural network model; Training the deep neural network model based on the training data set to obtain a trained deep neural network model; The high-dimensional implicit features of the disk and their corresponding labels are combined to form a new training data set, and an XGBoost classifier for replacing the second fully connected layer is trained based on the new training data set to obtain a trained XGBoost classifier; The trained XGBoost classifier replaces the second fully connected layer in the trained deep neural network model, and the replaced deep neural network model is used as the disk failure prediction model.

5. The disk failure prediction method according to claim 4, characterized in that: Construct a feature network structure consisting of a first fully connected layer and multiple residual layers in sequence, including: Based on F out =X*W+b to construct the first fully connected layer; where F out is the output vector of the first fully connected layer; X is the input vector of the first fully connected layer; W is the network weight of the first fully connected layer; b is the bias of the first fully connected layer; Based on x a+1 =x a +F(x a ,W a )、F(x a ,W a )=Relu(x a *W a ) constructs L residual layers; where x a+1 is the output of the a-th residual layer; x a is the input of the a-th residual layer; F(x a ,W a ) is the residual learning function of the a-th layer; Relu is the activation function, Relu(x)=max(0,x); 1≤a≤L and a is an integer; The constructed first fully connected layer and the L residual layers are combined to obtain the feature network structure.

6. The disk failure prediction method according to claim 4, characterized in that: The deep neural network model is trained based on the training data set to obtain a trained deep neural network model, including: Inputting various index data of the disk in the training data set into the deep neural network model to obtain a fault prediction result of the disk; Substituting the fault prediction result of the disk and the labels corresponding to various index data of the disk into a preset loss calculation function to perform loss calculation to obtain a first loss; With reducing the first loss to 0 as the optimization goal, the adjustable parameters of the neural network model are optimized and adjusted using a preset back propagation algorithm until all the training data sets are trained on the neural network model to obtain a trained deep neural network model.

7. The disk failure prediction method according to claim 4, characterized in that: The XGBoost classifier used to replace the second fully connected layer is trained based on the new training data set to obtain a trained XGBoost classifier, including: Inputting each high-dimensional implicit feature of the disk in the new training data set into the XGBoost classifier to obtain a fault classification result of the disk; Substituting the fault classification result of the disk and the labels corresponding to each high-dimensional implicit feature of the disk into a preset loss calculation function to perform loss calculation to obtain a second loss; With reducing the second loss to 0 as the optimization goal, the adjustable parameters of the XGBoost classifier are optimized and adjusted using a preset back propagation algorithm until all of the new training data sets are trained on the XGBoost classifier to obtain a trained XGBoost classifier.

8. A disk failure prediction device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the disk failure prediction method according to any one of claims 1 to 7 when executing the computer program.

Images