Hard disk failure prediction and data migration method for low-quality data sets

Abstract

The present invention provides a hard disk failure prediction and data migration method for low-quality data sets, including: obtaining the SMART information of the hard disk to obtain an information set, reconstructing the information set with positive and negative samples, and using the information set with lost data as the original data set; performing useless data cleaning operations on the original data set and performing undersampling processing; filling missing values ​​on the original data set, and converting the original data set after missing value filling into time series data; constructing and training a prediction model, inputting the time series data and the corresponding ASFD features into the prediction model to obtain the prediction result of the hard disk failure; identifying the hard disk that is about to fail according to the prediction result, and completing the repair of the hard disk data that is about to fail based on the maximum matching strategy of the bipartite graph and the repair scheduling strategy. Beneficial effects of the present invention: In the case of low-quality hard disk SMART, a higher hard disk failure prediction accuracy is achieved, and the data of the hard disk that is about to fail can be actively migrated and repaired in advance.

Description

Technical Field

[0001] The present invention belongs to the field of data storage, and in particular relates to a hard disk failure prediction and data migration method for low-quality data sets. Background Art

[0002] Today, hard disks are the most important storage devices in computers, and many data centers rely on a large number of hard disks to store important information. In some scenarios where large-scale storage systems are used, such as high-performance computing and Internet services, hard disk failures occur frequently. A survey of relevant literature shows that 78% of hardware replacements are caused by hard disk failures. The catastrophic consequences of hard disk failures are permanent and difficult to recover, thereby reducing the reliability of the data center. Therefore, predicting disk failures as early as possible can not only reduce the risk of data loss, but also reduce the cost of data recovery.

[0003] SMART (Self-Monitoring Analysis and Reporting Technology) was proposed in the 1990s. This technology can detect various working information inside the hard disk, such as the number of hard disk reads and writes, the number of head load and unload times, and the seek error rate. The traditional threshold-based hard disk failure technology compares the current SMART attribute of the hard disk with the set threshold. If the SMART attribute of the hard disk exceeds the corresponding threshold, the hard disk will send an alarm message to the operating system. However, this method has only a 3% to 10% TPR (True Positive Rate) when the FPR (False Positive Rate) is 0.1%. In the past few decades, many scholars have used machine learning and deep learning technologies to propose a variety of methods to improve the accuracy of fault prediction, including the use of Long short-term memory (LSTM), Generative Adversarial Network (GAN), regularized greedy forest (RGF), Temporal Convolutional Networks (TCNs), recurrent neural networks (RNNs), Convolutional neural networks (CNNs) and other algorithms to establish hard disk failure prediction models. Unfortunately, these related works generally train models based on high-quality datasets, such as the Backblaze open source dataset. Such prediction models have poor applicability in low-quality dataset scenarios.

[0004] In actual industrial applications, some data may be lost during the collection, transmission, and storage of hard disk SMART data due to objective or subjective reasons. For example, various errors may occur during data collection, such as sensor errors, data transmission errors, equipment failures, etc. These errors will cause inaccurate or incomplete data in the data set. For another example, a large-scale data center must turn off the SMART collection function during holidays. In addition, in order to prevent themselves from being replaced by intelligent operation and maintenance software, operation and maintenance engineers destroy sample data to reduce the accuracy of hard disk failure prediction technology. How to achieve good prediction results under low-quality data sets has become an important problem that needs to be solved in the actual engineering application of hard disk failure prediction technology.

[0005] When faced with low-quality data sets, existing technologies often use the method of filling the mode. Although it is simple to implement, the accuracy of hard disk failure prediction is insufficient. At the same time, existing technologies do not consider the timing of hard disk data. Hard disk failure does not occur suddenly at a certain moment, but is a gradual process. Summary of the invention

[0006] In view of this, the present invention aims to propose a hard disk failure prediction and data migration method for low-quality data sets, in order to solve at least one of the above-mentioned technical problems.

[0007] To achieve the above object, the technical solution of the present invention is achieved as follows:

[0008] The hard disk failure prediction and data migration method for low-quality data sets includes the following steps:

[0009] Obtaining SMART information of the hard disk to obtain an information set, marking the information set with positive and negative samples, and using the information set with missing data as the original data set;

[0010] Performing a useless data cleaning operation and under-sampling processing on the original data set;

[0011] Filling missing values ​​in the original data set, converting the original data set after missing value filling into time series data, and introducing ASFD features;

[0012] Build and train a prediction model, input time series data and corresponding ASFD features into the prediction model to obtain the prediction results of hard disk failure;

[0013] According to the prediction results, the hard disk that is about to fail is identified, and the migration and repair of the failed hard disk is completed based on the bipartite graph maximum matching strategy and repair scheduling strategy.

[0014] Furthermore, the process of labeling the information set as positive and negative samples includes:

[0015] Obtain hard disk failure information, mark the information set in the time period before the hard disk failure as a positive sample, and mark the information set in the rest of the time as a negative sample.

[0016] Furthermore, the process of performing useless data cleaning operation and undersampling processing on the original data set includes:

[0017] Obtain the fixed data in the hard disk and treat it as useless data and remove it from the original data set; select the corresponding number of negative sample information sets according to the number of information sets marked as positive samples in the original data set, and remove the remaining negative sample information sets.

[0018] Furthermore, the process of filling missing values ​​in the original data set includes:

[0019] Get the hard disk model of the current hard disk, and collect the SMART information of all hard disks with the same hard disk model as a filling set;

[0020] Get the SMART items corresponding to the lost data in the current hard disk as the missing set;

[0021] The mode of each SMART item in the intersection of the filled set and the missing set is extracted, and the mode is used as the missing data for filling.

[0022] Furthermore, the process of introducing the ASFD feature includes:

[0023] Traversing each element in the time series, and calculating the difference between the current element and the previous element, wherein the element is the hard disk SMART information converted into a time series;

[0024] The absolute value of the difference is taken, and the absolute value corresponding to each element is added to obtain an ASFD value, and the ASFD value is added to the original data set, wherein each ASFD value corresponds to a SMART item.

[0025] Furthermore, the process of building and training the prediction model includes:

[0026] The prediction model is trained using the LGB algorithm, and the optimal hyperparameters of the prediction model are obtained using the OPTUNA framework.

[0027] Furthermore, the process of completing the migration and repair of the faulty hard disk based on the bipartite graph maximum matching strategy and the repair scheduling strategy includes:

[0028] According to the bipartite graph maximum matching strategy, each data block in the hard disk is allocated to multiple reconstruction sets, and the reconstruction sets have the ability to be reconstructed in parallel, and according to the repair scheduling strategy, the number of data blocks in each migration reconstruction is maximized.

[0029] Compared with the prior art, the hard disk failure prediction and data migration method for low-quality data sets described in the present invention has the following beneficial effects:

[0030] In the case of low-quality hard disk SMART, a high hard disk failure prediction accuracy rate can be achieved, and data on the hard disk that is about to fail can be actively migrated and repaired in advance. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The accompanying drawings constituting a part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the accompanying drawings:

[0032] Figure 1 It is a schematic diagram of the workflow of the present invention;

[0033] Figure 2 Schematic diagram of the distribution of three stripes under RS ​​(4,2) coding conditions in the present invention;

[0034] Figure 3 A schematic diagram of identifying blocks to be transmitted in the present invention;

[0035] Figure 4 A schematic diagram for identifying how to store repair blocks in the present invention;

[0036] Figure 5 A schematic diagram of performing repair in the present invention;

[0037] Figure 6 Schematic diagram of performing migration and reconstruction in the repair round of the present invention. DETAILED DESCRIPTION

[0038] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0039] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and the like are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, features defined as "first", "second", and the like may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0040] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood by specific circumstances.

[0041] The present invention will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0042] The hard disk failure prediction and data migration method for low-quality data sets includes the following steps:

[0043] S1. Obtain the SMART information of the hard disk to obtain an information set, mark the information set with positive and negative samples, and use the information set with missing data as the original data set.

[0044] The process of labeling the information set with positive and negative samples in step S1 includes:

[0045] Obtain hard disk failure information, mark the information set in the time period before the hard disk failure as a positive sample, and mark the information set in the rest of the time as a negative sample.

[0046] Preferably, a specific execution process of step S1 and its explanation are as follows:

[0047] Collect SMART information from hard disks in large data centers regularly, for example, collect information from each hard disk once a day, and mark the data of the failed disk within 30 days before the failure as 1, which is called a positive sample; the data in other time periods is marked as 0, which is called a negative sample. The negative sample is the data of healthy hard disks;

[0048] Part of the data may be lost due to objective or subjective reasons during the collection, transmission, and storage of hard disk SMART data. Common loss situations include: SMART data loss of the same hard disk for several consecutive days, SMART data loss of several hard disks on the same day, and partial SMART field loss of the same hard disk on the same day. Therefore, the data set that is actually collected and has partial data loss is selected as the original data set.

[0049] S2. Cleaning up useless data and performing under-sampling processing on the original data set.

[0050] The process of performing useless data cleaning and undersampling processing on the original data set in step S2 includes:

[0051] Obtain the fixed data in the hard disk and treat it as useless data and remove it from the original data set; select the corresponding number of negative sample information sets according to the number of information sets marked as positive samples in the original data set, and remove the remaining negative sample information sets.

[0052] The purpose of filtering out useless data in the SMART dataset is to reduce the workload of subsequent analysis. At the same time, due to the large disparity between the number of positive and negative samples in the hard disk dataset, undersampling is used to artificially change the ratio of positive and negative samples.

[0053] S3. Fill missing values ​​in the original data set, convert the original data set after missing value filling into time series data, and introduce ASFD features.

[0054] By constructing time series, we can effectively make up for the low quality of the SMART data set. In order to better calculate and process the data in the original data set, we create and use time windows to convert the original data set into time series data. The time window can be used to more representatively discover the relationship between data and facilitate subsequent calculations. By constructing a small sliding window and sliding on the constructed time window, we can better discover the changes in data within the time window and enrich the calculation of high-order data. The features of multiple time series are merged, and the features that perform well after interaction are selected to strengthen the prediction model.

[0055] S31, the process of filling missing values ​​in the original data set in step S3 includes:

[0056] Get the hard disk model of the current hard disk, and collect the SMART information of all hard disks with the same hard disk model as a filling set;

[0057] Get the SMART items corresponding to the lost data in the current hard disk as the missing set;

[0058] The mode of each SMART item in the intersection of the filled set and the missing set is extracted, and the mode is used as the missing data for filling.

[0059] Preferably, a specific execution process of step S31 and its explanation are as follows:

[0060] After converting the original data set into time series data, even if a large amount of original data is lost, the change trend of the original data can still be obtained from the high-order values, making the prediction model more accurate and capable of dealing with the situation of low quality data set;

[0061] In order to obtain better prediction results, healthy disks and faulty disks are classified and processed separately. For healthy disks, the missing value of hard disk i in the jth SMART item is filled with the mode of all jth SMART items of the model to which the healthy disk belongs. For faulty disks, the missing value of hard disk i in the jth SMART item is filled with the mode of all jth SMART items of the model to which the faulty disk belongs.

[0062] S32: The process of introducing the ASFD feature in step S3 includes:

[0063] Traversing each element in the time series, and calculating the difference between the current element and the previous element, wherein the element is the hard disk SMART information converted into a time series;

[0064] The absolute value of the difference is taken, and the absolute value corresponding to each element is added to obtain an ASFD value, and the ASFD value is added to the original data set, wherein each ASFD value corresponds to a SMART item.

[0065] Preferably, a specific execution process of step S32 and its explanation are as follows:

[0066] ASFD (Absolute Sum of First Difference) is a measure of the change or variation between values ​​in a time series. It is calculated as the difference between each element in a time series and its previous element. The absolute sum is the sum of the absolute values ​​of these differences. The result value of ASFD is a non-negative number. A value of zero means that all values ​​in the time series are the same. A higher value means a greater change in the values.

[0067] The calculation formula of the ASFD is: .

[0068] Where y is the desired ASFD value; X is the desired SMART attribute value of the SMART attribute column; I is the i-th row; N is the number of rows in the desired SMART attribute column; x i is the SMART attribute value of the i-th row of the SMART attribute column; xi+1 is the SMART attribute value of the i+1th row of the desired SMART attribute column.

[0069] For each SMART item of each hard disk, the ASFD value is calculated one by one and added to the original data set as a new attribute. The number of entries in the processed original data set remains unchanged, but the number of SMART items for each entry is doubled. For example, for the hard disk numbered 4214, the attribute value of the SMART data item ID 1 on June 1, 2024 is 15, and the ASFD value of the SMART data item ID 1 is 0.9. That is, a corresponding ASFD value is added for each SMART data value.

[0070] S4. Build and train a prediction model, input the time series data and the corresponding ASFD features into the prediction model to obtain the prediction result of the hard disk failure.

[0071] S41, the process of constructing and training the prediction model in step S4 includes:

[0072] The prediction model is trained using the LGB algorithm, and the optimal hyperparameters of the prediction model are obtained using the OPTUNA framework.

[0073] Preferably, the reasons for selecting the term explanation, i.e., the technical solution, proposed in step S41 are as follows:

[0074] LGB (lightgbm) is common knowledge in this field: Gradient Boosting Decision Tree (GBDT) is a commonly used model in machine learning. Its main idea is to use weak classifiers for iterative training to obtain the optimal model. This model has the advantages of high accuracy and low overfitting. GDBT is widely used in the industry and is usually used for tasks such as classification, regression and sorting. LGB is an open source framework for GBDT implemented by the Microsoft DMTK team. As an implementation of GBDT, the LGB algorithm has made many optimizations on the traditional GBDT algorithm and has the advantages of supporting efficient parallel training, lower memory consumption, and higher accuracy.

[0075] Compared with the traditional GBDT algorithm, the LGB algorithm has the following advantages:

[0076] Histogram optimization is used. Histogram optimization simplifies the expression of data and reduces the use of memory. Histograms can avoid overfitting to a certain extent. The histogram of a leaf node can be obtained by subtracting the histogram of a brother node from the histogram of the father node. Based on this, the histogram of a leaf node with a smaller amount of data can be calculated first, and then the histogram of a leaf node with a larger amount of data can be obtained by subtracting the histogram to achieve the acceleration effect.

[0077] The leaf-wise method is used to improve the accuracy of the model. Compared with the level-wise method, the leaf-wise method is more efficient. When the number of leaf nodes is the same, the leaf-wise method can obtain smaller training errors and higher accuracy. If the leaf-wise method is used alone to grow, overfitting may occur on small data sets. Therefore, LGB adds restrictions on the depth of the tree in leaf-wise terms.

[0078] Support for category features. Traditional machine learning algorithms generally cannot directly input categories. Category features need to be discretized and converted into multi-dimensional 0 / 1 features. This approach is inefficient both in space and time. LGB directly supports category features by changing the decision rules of the decision tree algorithm without the need for additional discretization.

[0079] Supports parallelization. Feature Parallelism is suitable for scenarios with small data but more features, Data Parallelism is suitable for scenarios with large data volume but fewer features, and Voting Parallelism is suitable for scenarios with large data volume and more features. Feature Parallelism vertically splits the data so that different machines have all sample points but different features, and then finds the optimal segmentation points of different features on different machines, and then synchronizes them to obtain the global optimal segmentation point. Data Parallelism horizontally splits the data so that different machines have part of the data but all the features, and then constructs histograms on different machines, merges the histograms, and finds the optimal segmentation point on the merged global histogram. Voting Parallelism is an optimization based on data parallelism. The bottleneck of data parallelism is mainly merging histograms. Voting Parallelism merges only the histograms of some feature values ​​by voting, thereby reducing the amount of communication. First, the optimal features of TOP K are obtained through local data, and then the features that may be the global optimal segmentation points are obtained by voting. When merging histograms, only these selected features are merged, thereby reducing the amount of communication when merging histograms.

[0080] OPTUNA is common knowledge in this field: Regarding the hyperparameter tuning of the model, many scholars use the brute force search method of GridSearch. Although it is simple and easy to understand, its disadvantage is obvious and it is too time-consuming to run. In contrast, the parameter tuning method based on the Bayesian framework, such as HyperOPT and OPTUNA, is lightweight and more powerful, and the speed is also very fast. As the parameter tuning method adopted by the present invention, generally, the hyperparameters of the tree-based model can be divided into 4 categories: 1) parameters that affect the structure and learning of the decision tree; 2) parameters that affect the training speed; 3) parameters that improve the accuracy; 4) parameters that prevent overfitting.

[0081] In most cases, there is a lot of overlap between these categories. Improving the efficiency of one category may reduce the efficiency of another category. Relying entirely on manual parameter tuning is not only time-consuming and labor-intensive, but also difficult to find the optimal balance. However, if a suitable parameter grid is provided, OPTUNA can automatically search and find the most balanced parameter combination between these categories, thereby optimizing the overall performance.

[0082] The above-mentioned LGB and OPTUNA methods are an optimal combination obtained after long-term experiments and comparisons based on a large number of model algorithms and parameter tuning. Other model algorithms, such as LSTM, random forest, and SVM, cannot achieve the effect of the LGB method. Other parameter tuning methods, such as GridSearch and HyperOPT, also cannot achieve the effect of the OPTUNA method.

[0083] S42, the process of inputting the time series data and the corresponding ASFD features into the prediction model in step S4 to obtain the prediction result of the hard disk failure includes:

[0084] The collected hard disk SMART data is checked for partial data loss. If there is data loss, i.e., low quality, the missing data is filled with the mode of the SMART item of the hard disk in the last N days, where N is generally 7 or 10 days. Then, the ASFD value of each SMART item of the hard disk is calculated to obtain the latest sample of the hard disk, which is input into the model. The input result is 0 or 1, where 0 represents the prediction result that the hard disk is healthy, and 1 represents that the hard disk is about to fail.

[0085] S5. Identify the hard disk that is about to fail based on the prediction results, and complete the migration and repair of the failed hard disk based on the bipartite graph maximum matching strategy and the repair scheduling strategy.

[0086] The process of completing the repair of the faulty hard disk data based on the bipartite graph maximum matching strategy and the repair scheduling strategy in step S5 includes:

[0087] According to the bipartite graph maximum matching strategy, each data block in the hard disk is allocated to multiple reconstruction sets, and the reconstruction sets have the ability to be reconstructed in parallel, and according to the repair scheduling strategy, the number of data blocks in each migration reconstruction is maximized.

[0088] Preferably, the specific execution processes of the bipartite graph maximum matching strategy and the repair scheduling strategy in step S5 are respectively:

[0089] The maximum matching strategy for bipartite graphs is:

[0090] Construct a set C and an initial reconstruction set R, wherein the set C represents the set of all blocks of the faulty disk that need to be repaired;

[0091] For each data block in the set C, determine in turn whether the block set in the newly constructed reconstruction set R can be reconstructed in parallel if it is added to the reconstruction set R;

[0092] If parallel reconstruction is possible, the data block is added to the reconstruction set R and deleted from the set C until all blocks in the set C are organized into the reconstruction set R.

[0093] In the bipartite graph maximum matching strategy, the specific process of sequentially judging and adding data blocks to the reconstruction set includes:

[0094] For each data block C i , use the MATCH function to construct a bipartite graph containing N−1 nodes, one part of the bipartite graph consists of blocks of healthy hard disks, and the other part consists of check blocks;

[0095] Identify the maximum matching in the bipartite graph. If there are k(1+|R|) edges in the maximum matching, it means that the current data block C i It can be rebuilt in parallel using k(1+|R|) different healthy hard disks and check blocks. At this time, the MATCH function returns true and the current data block C i Add to the reconstruction set R and delete the current data block C from the set C i .

[0096] Set multiple repair rounds, and repeat the above steps for each data block C in each repair round. i The operation is performed until all data blocks in set C are added to the reconstruction set R, and the set of all reconstruction sets R is obtained. d}, where d is the total number of reconstruction sets.

[0097] In order to increase the number of blocks in the reconstruction set and thus improve the capability of parallel reconstruction, it may be checked whether the reconstruction set R can be expanded by exchanging a data block in the reconstruction set R with other data blocks not in the reconstruction set R.

[0098] Repair scheduling strategy:

[0099] Sort all reconstruction sets R in set D in monotonically decreasing order according to the number of data blocks they contain, so as to ensure that reconstruction sets containing more blocks are given priority during the repair process;

[0100] Set two indices l and u, where l represents the index of the current smallest reconstruction set index and u represents the current largest reconstruction set index, that is, the first reconstruction set is R l , the final reconstruction set is R u ;

[0101] Establish a set of blocks C to be repaired by migration m , establish a block set C that is repaired by collaborative reconstruction using healthy hard disks r ;

[0102] Calculate C m , determine the current C m Is it satisfied? (l+1) ∪…∪R u |≤ C m ;

[0103] If satisfied, then R (l+1) To R u The reconstruction set between them can be repaired by migration, R l Repair is performed through reconstruction, and the migration and reconstruction processes are performed in parallel, and the repair round is interrupted to complete the repair;

[0104] Otherwise, find the reconstruction set with the least number of data blocks in the current set D, so that , x is the index of the reconstruction set index of the current reconstruction set, and obtains the subset R of the current reconstruction set x 'Belongs to R x , so that ;

[0105] Let R x = R x -R x ', let M l = R x ' ∪R x+1 …∪R u , update the indicators l and u, let l = l +1, u = x, so that the new reconstruction set can be considered in the next repair round, and iterate this process until all reconstruction sets have been processed.

[0106] Working process:

[0107] The present invention discloses a hard disk failure prediction and data migration method for low-quality data sets. In order to better illustrate the working process of the present invention, a specific implementation process as shown below is given:

[0108] 1) Create a time window: In low-quality data sets, we use a standard time window size of 10 days to create time series features, which facilitates the subsequent addition of time series features.

[0109] 2) Set sliding window: Preset two different sliding windows of 7 days and 1 day in advance to facilitate the subsequent addition of time feature sequences. The sliding window will slide on the above time windows for calculation;

[0110] Add ASFD feature. Use the sliding window method to add the time series feature AbsoluteSum of First Difference (ASFD) to the data set to reflect the absolute fluctuation between adjacent observations of hard disk SMART data. If a time series x = [1,1,0,1,0] is given, then the first-order difference is [0,0,1,1,1], and ASFD is y = [0,0,1,2,3];

[0111] 3) CID features can also be added as a supplement. The time series feature Complexity Invariant Distance (CID) is added using the sliding window method to reflect the time series complexity of the hard disk SMART data. For time series Q and C, the CID calculation formula is: ,Through multiple experiments, adding CID features will improve the ,prediction accuracy in most cases, but it will increase the processing time, and ,engineering technicians need to decide whether to add CID based on actual ,needs.

[0112] 4) Overview of the training dataset: After preprocessing the original dataset, each SMART attribute is added with ASFD features and CID features. The training dataset has a total of 121 dimensions of features.

[0113] 5) Model training: Use LGB (lightgbm) developed by Microsoft combined with the preprocessed training set to train the hard disk failure prediction model.

[0114] 6) Model hyperparameter tuning: Use the OPTUNA method to find the optimal hyperparameters of the LGB algorithm and tune the model.

[0115] 7) Hard disk failure prediction: After obtaining the failure prediction model with the optimal hyperparameters, the test set data is input into the failure prediction model to obtain the disk-level prediction result, that is, whether the hard disk will fail.

[0116] 8) Data migration and reconstruction:

[0117] like Figure 2 , 3 , 4, 5: Construct a bipartite graph with two sets of left and right vertices, {N1, N2, N3, N4, N5}, {N4, N5, N6, N7, N8} whose node vertices are connected to the k = 5 block vertices of strips S1 and S2 respectively, find the maximum matching, and conclude that blocks can be retrieved from N1, N2, N3, N4 to reconstruct the block of S1. Construct another bipartite graph, {N6, N7, N8}, {N1, N2, N3} and {N3, N4, N5} whose node vertices are connected to the strip vertices of strips S1, S2 and S3 respectively, and the maximum matching shows that the repair block of S1 can be stored in N6.

[0118] like Figure 6 As shown: Another embodiment of data migration and reconstruction is given, taking a reconstruction set of d=7, Cm=4. For the first round of repair, it is found that the largest x is 5, so that |R5|+|R6|+|R7|>Cm = 4, and then further find a subset R5'⊂R5, R5' contains a block in R5, so that M1 = R5'∪R6∪R7. Through migration, the number of remaining data blocks in R5 is reduced to 2 in parallel with the reconstruction of R1 in the first repair round, and finally all blocks are repaired in the third round of repair.

[0119] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention, and they should all be included in the scope of the claims and specification of the present invention.

[0120] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A hard disk failure prediction and data migration method for low-quality data sets, characterized in that: The steps include: S1, obtaining the SMART information of the hard disk to obtain an information set, reconstructing the positive and negative samples of the information set, and using the information set with lost data as the original data set; S2, performing a useless data cleaning operation and under-sampling processing on the original data set; S3, filling missing values ​​in the original data set, converting the original data set after the missing values ​​are filled into time series data, and introducing ASFD features; wherein, creating and using a time window to convert the original data set into time series data; The process of filling missing values ​​in the original data set includes: obtaining the hard disk model of the current hard disk, collecting SMART information of all hard disks with the same hard disk model as a filling set; obtaining SMART items corresponding to the missing data in the current hard disk as a missing set; extracting the mode of each SMART item in the intersection of the filling set and the missing set, and filling the missing data with the mode; The process of introducing the ASFD feature includes: traversing each element in the time series, calculating the difference between the current element and the previous element, wherein the element is the hard disk SMART information converted into a time series; taking the absolute value of the difference, and adding the absolute value corresponding to each element to obtain the ASFD value, and adding the ASFD value to the original data set, wherein each ASFD value corresponds to a SMART item; S4, building and training a prediction model, inputting the time series data and the corresponding ASFD features into the prediction model to obtain the prediction result of the hard disk failure; Identify hard disks that are about to fail based on the prediction results, and complete the migration and repair of the failed hard disks based on the bipartite graph maximum matching strategy and repair scheduling strategy; The process of migrating and repairing the faulty hard disk is specifically as follows: according to the bipartite graph maximum matching strategy, each data block in the hard disk is allocated to multiple reconstruction sets, and the reconstruction sets have the ability of parallel reconstruction, and according to the repair scheduling strategy, the number of data blocks in each migration and reconstruction is maximized; The repair scheduling strategy specifically includes: Sort all reconstruction sets R in monotonically decreasing order according to the number of data blocks they contain, and set two indices l and u, where l represents the index of the current smallest reconstruction set index and u represents the current largest reconstruction set index; Establish a set of blocks C to be repaired by migration m , and establish a block set C that is repaired by collaborative reconstruction using healthy hard disks r ; Calculate C m , determine the current C m Is it satisfied? (l+1) ∪…∪R u |≤C m ; If satisfied, then R (l+1) To R u The reconstruction set between them is repaired by migration, R l Repair by rebuilding, with migration and rebuilding processes performed in parallel; Otherwise, obtain the reconstruction set with the least number of data blocks among all reconstruction sets R, so that x is the index of the reconstruction set index of the current reconstruction set, and obtains the subset of the current reconstruction set Make Let R x =R x -R x ', let M l =R x '∪R x+1 …∪R u , update the indicators l and u, let l = l + 1, u = x, and iterate until all reconstruction sets are processed.

2. The hard disk failure prediction and data migration method for low-quality data sets according to claim 1, characterized in that: The process of reconstructing positive and negative samples of the information set includes: Obtain the information set of the failed hard disk in the time period before the failure, and mark it as a positive sample; according to the length of the time period, obtain the information set of the healthy hard disk in the same time period, and mark it as a negative sample; The SMART information of the hard disk records the fault status of the hard disk.

3. The hard disk failure prediction and data migration method for low-quality data sets according to claim 1, characterized in that: The process of performing useless data cleaning operation and undersampling processing on the original data set includes: Obtain the fixed data in the hard disk and treat it as useless data and remove it from the original data set; select the corresponding number of negative sample information sets according to the number of information sets marked as positive samples in the original data set, and remove the remaining negative sample information sets.

4. The hard disk failure prediction and data migration method for low-quality data sets according to claim 1, characterized in that: The process of filling missing values ​​in the original data set includes: Get the hard disk model of the current hard disk, and collect the SMART information of all hard disks with the same hard disk model as a filling set; Get the SMART items corresponding to the lost data in the current hard disk as the missing set; The mode of each SMART item in the intersection of the filled set and the missing set is extracted, and the mode is used as the missing data for filling.

5. The hard disk failure prediction and data migration method for low-quality data sets according to claim 1, characterized in that: The process of introducing the ASFD feature includes: Traversing each element in the time series, and calculating the difference between the current element and the previous element, wherein the element is the hard disk SMART information converted into a time series; The absolute value of the difference is taken, and the absolute value corresponding to each element is added to obtain an ASFD value, and the ASFD value is added to the original data set, wherein each SMART item corresponds to an ASFD value.

6. The hard disk failure prediction and data migration method for low-quality data sets according to claim 1, characterized in that: The process of building and training a prediction model includes: The prediction model is trained using the LGB algorithm, and the optimal hyperparameters of the prediction model are obtained using the OPTUNA framework.

Images