A hard disk failure prediction method and system based on multi-model fusion
Through multi-model fusion and feature selection technology, the problem of insufficient accuracy of existing hard disk fault prediction methods on different hard disk models is solved, and higher fault prediction accuracy and early warning capabilities are achieved.
Patent Information
- Application Number
- CN202411841531.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-13
- Publication Date
- 2025-04-08
AI Technical Summary
The existing hard disk failure prediction methods are not accurate on hard disks of different manufacturers and models, and are greatly affected by factors such as sample ratio, data volume and model adaptability.
The hard disk failure prediction method based on multi-model fusion is adopted. By obtaining hard disk SMART data, feature extraction and selection is performed using progressive window sampling and extreme random tree models, machine learning models such as LightGBM, XGBoost and CatBoost are trained in parallel, and the final prediction model is generated through voting fusion and Pearson correlation coefficient analysis.
It improves the accuracy of hard disk failure prediction, can sense the health status of hard disk in advance, conduct early warnings, and reduces operation and maintenance pressure and data repair investment.
Smart Images
Figure CN119782945A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of storage technology, and in particular relates to a hard disk failure prediction method and system based on multi-model fusion. Background Art
[0002] At present, the methods for hard disk failure prediction generally use basic machine learning models, such as decision trees and random forests, which are simple to train but have poor generalization and are difficult to accurately predict a single hard disk model with a small number of samples; or use deep learning models, such as BP neural networks and recurrent neural networks.
[0003] Although the above hard disk failure prediction methods have certain effects, there is still no universal model that is suitable for hard disks of different models from different manufacturers. In actual scenarios, they are easily affected by various factors such as the ratio of positive and negative samples, the amount of data, and the adaptability of the model, resulting in low accuracy in hard disk failure prediction. Summary of the invention
[0004] The present invention provides a hard disk fault prediction method and system based on multi-model fusion, aiming to make full use of the feature extraction ability and time series modeling ability of the machine learning integrated model, capture the change process of each SMART attribute of hard disks of different models, and improve the accuracy of hard disk fault prediction.
[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is:
[0006] In a first aspect, a hard disk failure prediction method based on multi-model fusion is provided, comprising:
[0007] Get the hard disk SMART data of different models of hard disks;
[0008] Based on the progressive window sampling method, the hard disk SMART data is subjected to feature extraction to obtain the initial feature set;
[0009] Use the extreme random tree model to select features from the initial feature set and obtain the optimal feature set;
[0010] By training multiple machine learning models with the optimal feature set, a hard disk failure prediction model based on multi-model fusion is obtained;
[0011] When it is necessary to predict the failure of the target hard disk, the current SMART data of the target hard disk is obtained and the current feature set is extracted;
[0012] The current feature set is input into the hard disk failure prediction model based on multi-model fusion, and the failure prediction result of the target model hard disk is output.
[0013] Furthermore, based on the progressive window sampling method, the hard disk SMART data is subjected to feature extraction to obtain the initial feature set, including:
[0014] Get the degree of SMART attribute change based on the hard disk SMART data;
[0015] Analyze the relationship between the degree of change in SMART attributes and the fault labels of the corresponding hard disks;
[0016] According to the corresponding relationship, the historical SMART data of the hyperparameter window length value L before the failure occurs is used as input;
[0017] Based on the progressive window sampling method, feature sampling is performed on the historical SMART data within a sampling window of size L to obtain the initial feature set.
[0018] Furthermore, the extreme random tree model is used to select features from the initial feature set to obtain the optimal feature set, including:
[0019] Input all the initial features in the initial feature set into the extreme random tree model to obtain the initial model accuracy and the initial feature weight value of each initial feature;
[0020] The initial features whose initial feature weight values are lower than a preset weight threshold are removed from the initial feature set to obtain a first feature set;
[0021] Inputting the first feature set into the extreme random tree model for training, obtaining a first model accuracy and a first feature weight value of each first feature in the first feature set;
[0022] Determine whether the number of features in the first feature set is higher than a preset number threshold;
[0023] If it is not higher, the first feature set stops feature elimination;
[0024] If it is higher, determining whether there is a first feature whose first feature weight value is lower than a preset weight threshold;
[0025] If it does not exist, the first feature set stops feature elimination;
[0026] If so, the first feature whose weight value is lower than the preset weight threshold is removed from the first feature set to obtain a second feature set, and the second model accuracy is obtained;
[0027] Repeat the above steps, input the second feature set into the extreme random tree model to continue training, until the Kth feature set stops feature elimination, and the Kth model accuracy is obtained, where K is a positive integer greater than or equal to 2;
[0028] After stopping feature elimination, compare the model accuracy corresponding to each feature set, and select the feature set with the highest model accuracy as the optimal feature set.
[0029] Furthermore, multiple machine learning models are trained through the optimal feature set to obtain a hard disk failure prediction model based on multi-model fusion, including:
[0030] Select LightGBM algorithm model, XGBoost algorithm model and CatBoost algorithm model from multiple machine learning models as the base classifier of the first layer;
[0031] The base classifiers of the LightGBM algorithm model, XGBoost algorithm model, and CatBoost algorithm model are trained in parallel using the optimal feature set to obtain the training prediction results and feature importance ranking of each model.
[0032] The training prediction results of each model are voted and integrated, and the Pearson correlation coefficient is used to analyze the correlation between the top N features in the feature importance ranking of each model to obtain the correlation data;
[0033] Select the features with greater correlation in the correlation data as the second-layer input features;
[0034] The second-layer input features are input into the meta-model of the second-layer Bagging algorithm random forest model for training to obtain the final prediction results;
[0035] According to the trained first-layer base classifier and the meta-model of the Bagging algorithm random forest model in the second layer, a hard disk failure prediction model based on multi-model fusion is constructed.
[0036] In the second aspect, a hard disk failure prediction system based on multi-model fusion is provided, including:
[0037] A data acquisition module is used to obtain hard disk SMART data of hard disks of different models;
[0038] A feature extraction module is used to extract features from hard disk SMART data based on a progressive window sampling method to obtain an initial feature set;
[0039] The feature selection module is used to select the initial feature set using the extreme random tree model to obtain the optimal feature set;
[0040] Model training module, used to train multiple machine learning models through the optimal feature set to obtain a hard disk failure prediction model based on multi-model fusion;
[0041] The data acquisition module is also used to obtain the current SMART data of the target model hard disk and extract the current feature set when it is necessary to predict the failure of the target model hard disk;
[0042] The fault prediction module is used to input the current feature set into the hard disk fault prediction model based on multi-model fusion and output the fault prediction result of the target model hard disk.
[0043] Furthermore, the feature extraction module is specifically used to obtain the degree of change of SMART attributes based on the hard disk SMART data; analyze the correspondence between the degree of change of SMART attributes and the fault label of the corresponding hard disk; take the historical SMART data of the hyperparameter window length value L before the failure occurs as input based on the correspondence; perform feature sampling on the historical SMART data in the sampling window of size L based on the progressive window sampling method to obtain an initial feature set.
[0044] Furthermore, the feature selection module is specifically used to input all the initial features in the initial feature set into the extreme random tree model to obtain the initial model accuracy and the initial feature weight value of each initial feature; remove the initial features whose initial feature weight values are lower than the preset weight threshold from the initial feature set to obtain the first feature set; input the first feature set into the extreme random tree model for training to obtain the first model accuracy and the first feature weight value of each first feature in the first feature set; determine whether the number of features in the first feature set is higher than the preset number threshold; if not, stop feature removal in the first feature set; if higher, determine whether there is The first feature whose weight value is lower than the preset weight threshold is selected; if it does not exist, the first feature set stops feature elimination; if it exists, the first feature whose weight value is lower than the preset weight threshold is eliminated from the first feature set to obtain the second feature set, and the second model accuracy is obtained; repeat the above steps, input the second feature set into the extreme random tree model to continue training, until the Kth feature set stops feature elimination, and the Kth model accuracy is obtained, K is a positive integer greater than or equal to 2; when feature elimination is stopped, the model accuracy corresponding to each feature set is compared, and the feature set corresponding to the highest model accuracy is selected as the optimal feature set.
[0045] Furthermore, the model training module is specifically used to select LightGBM algorithm model, XGBoost algorithm model and CatBoost algorithm model from multiple machine learning models as the base classifiers of the first layer; the base classifiers of the LightGBM algorithm model, XGBoost algorithm model and CatBoost algorithm model are trained in parallel through the optimal feature set to obtain the training prediction results and feature importance rankings of each model; the training prediction results of each model are voted and fused, and the Pearson correlation coefficient is used to analyze the correlation between the top N features in the feature importance ranking of each model to obtain correlation data; features with greater correlation in the correlation data are selected as second-layer input features; the second-layer input features are input into the meta-model of the second-layer Bagging algorithm random forest model for training to obtain the final prediction results; according to the trained first-layer base classifiers and the second-layer Bagging algorithm random forest model meta-model, a hard disk failure prediction model based on multi-model fusion is constructed.
[0046] The beneficial effects achieved by the present invention are:
[0047] Obtain hard disk SMART data of hard disks of different models; extract features from hard disk SMART data based on progressive window sampling to obtain an initial feature set; use extreme random tree model to select features from the initial feature set to obtain the optimal feature set; train multiple machine learning models with the optimal feature set to obtain a hard disk failure prediction model based on multi-model fusion; when failure prediction of the target model hard disk is required, obtain the current SMART data of the target model hard disk and extract the current feature set; input the current feature set into the hard disk failure prediction model based on multi-model fusion to output the failure prediction result of the target model hard disk. Make full use of the feature extraction and time series modeling capabilities of the machine learning integrated model to capture the changing process of each SMART attribute of different models of hard disks and improve the accuracy of hard disk failure prediction;
[0048] By sensing the health status of the hard disk in advance and providing early warning before the hard disk fails, it can help operation and maintenance personnel replace or perform other repair operations on the disk that is about to fail in advance based on the model prediction results, thereby effectively avoiding the impact of equipment instability when failure is imminent on the business, alleviating the pressure on operation and maintenance personnel and reducing the investment in data repair. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] Figure 1 It is a flow chart of the hard disk failure prediction method based on multi-model fusion of the present invention;
[0050] Figure 2 A flow chart of performing feature selection to obtain an optimal feature set for the present invention;
[0051] Figure 3 This is a structural diagram of the hard disk failure prediction system based on multi-model fusion of the present invention. DETAILED DESCRIPTION
[0052] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.
[0053] like Figure 1 As shown, an embodiment of the present invention provides a hard disk failure prediction method based on multi-model fusion, including:
[0054] 101, obtain the hard disk SMART data of different models of hard disks;
[0055] In this embodiment, the hard disk SMART data, which is the full name of the hard disk self-monitoring, analysis and reporting technology (Self-MonitoringAnalysis and Reporting Technology) data, is a data tool used to record the operating status of a hard disk drive (HDD) or a solid state disk (SSD). The hard disk SMART data can provide the health status, performance indicators and potential fault warnings of the hard disk, helping users or system administrators take measures before the hard disk has problems. Since the present invention aims to train a fault prediction model for different types of hard disks from different manufacturers, it is necessary to first obtain the hard disk SMART data of different models of hard disks.
[0056] 102, extracting features from the hard disk SMART data based on a progressive window sampling method to obtain an initial feature set;
[0057] In this embodiment, the hard disk SMART data is when or before the hard disk fails, so the hard disk SMART data can be analyzed to know the degree of SMART attribute change, and the corresponding relationship between the degree of SMART attribute change and the fault label of the corresponding hard disk is analyzed;
[0058] According to the corresponding relationship, the historical SMART data of the hyperparameter window length value L before the failure occurs is used as input; for example, the data L days before the actual failure date is also marked as a failure, which fundamentally increases the positive sample ratio. Each failure prediction requires L days of historical SMART data as input;
[0059] According to the progressive window sampling method based on hard disk failure, the 1×C positive sample is expanded to W×C, and then the sample size of W×C is obtained through window sampling. At this time, the number of samples increases from 1 to L. This sampling method not only enriches the data volume, but also can capture more failure modes and record the entire process of the hard disk gradually failing.
[0060] In order to further leverage the advantages of sliding window features, statistics were performed on the attributes in each window, and the statistical value features Min (minimum value), Max (maximum value), Mean (mean) and Var (variance) were used to replace the original features. At this time, the feature dimension of each sample changed from W×C to 4×C. While completing feature generation, feature dimensionality reduction was also completed to obtain the initial feature set.
[0061] 103, using the extreme random tree model to select features from the initial feature set to obtain the optimal feature set;
[0062] In this embodiment, the specific process of performing feature selection to obtain the optimal feature set is as follows: Figure 2 As shown, the steps are as follows:
[0063] 201, input all the initial features in the initial feature set into the extreme random tree model to obtain the initial model accuracy and the initial feature weight value of each initial feature;
[0064] 202, removing initial features whose initial feature weight values are lower than a preset weight threshold from the initial feature set to obtain a first feature set;
[0065] 203, inputting the first feature set into the extreme random tree model for training, and obtaining a first model accuracy and a first feature weight value of each first feature in the first feature set;
[0066] 204, determining whether the number of features in the first feature set is higher than a preset number threshold;
[0067] If it is not higher, the first feature set stops feature elimination and executes step 207; if it is higher, executes step 205;
[0068] 205, determining whether there is a first feature whose first feature weight value is lower than a preset weight threshold;
[0069] If it does not exist, the first feature set stops feature elimination and executes step 207; if it does exist, execute step 206;
[0070] 206, removing the first feature whose weight value is lower than the preset weight threshold from the first feature set to obtain a second feature set, and obtaining a second model accuracy;
[0071] Repeat the above step 203, input the second feature set into the extreme random tree model to continue training, until the Kth feature set stops feature elimination, and the Kth model accuracy is obtained, where K is a positive integer greater than or equal to 2;
[0072] 207, after stopping feature elimination, compare the model accuracy corresponding to each feature set, and select the feature set corresponding to the highest model accuracy as the optimal feature set.
[0073] 104, training multiple machine learning models through the optimal feature set to obtain a hard disk failure prediction model based on multi-model fusion;
[0074] In this embodiment, there are many kinds of machine learning models. When selecting the base classifier of the first layer, the LightGBM algorithm model, XGBoost algorithm model and CatBoost algorithm model based on the Boosting algorithm are selected as the base classifier of the first layer; LightGBM (Light Gradient Boosting Machine) is a machine learning algorithm based on the gradient boosting framework; XGBoost (eXtreme Gradient Boosting) is an ensemble learning algorithm based on the gradient boosting tree; CatBoost is a machine learning method based on the gradient boosting decision tree. These three Boosting algorithms are among the top-ranked algorithms;
[0075] The base classifiers of the LightGBM algorithm model, XGBoost algorithm model, and CatBoost algorithm model are trained in parallel using the optimal feature set to obtain the training prediction results and feature importance ranking of each model.
[0076] The training prediction results of each model are voted and integrated, and the Pearson correlation coefficient is used to analyze the correlation between the top N features in the feature importance ranking of each model to obtain the correlation data; the top N features are selected to screen out features with strong correlation, and the value range of N can be 1≤N≤n / 3, where n is the total number of features;
[0077] Select the features with greater correlation in the correlation data as the second-layer input features;
[0078] Input the second-layer input features into the meta-model of the second-layer Bagging algorithm random forest model for training to obtain the final prediction results; Bagging (Bootstrap aggregating), also known as the bagging algorithm, is an integrated learning method that aims to improve the accuracy and stability of the overall model by combining the prediction results of multiple models, while reducing the variance of the results and avoiding overfitting;
[0079] According to the trained first-layer base classifier and the meta-model of the Bagging algorithm random forest model in the second layer, a hard disk failure prediction model based on multi-model fusion is constructed.
[0080] 105, when it is necessary to predict the failure of the target model hard disk, obtain the current SMART data of the target model hard disk and extract the current feature set;
[0081] In this embodiment, after constructing a hard disk failure prediction model based on multi-model fusion using the hard disk SMART data of hard disks of different models, when it is necessary to predict failure of the target model hard disk, the current SMART data of the target model hard disk is obtained, and the current feature set is extracted according to the content of step 102.
[0082] 106, input the current feature set into the hard disk failure prediction model based on multi-model fusion, and output the failure prediction result of the target model hard disk.
[0083] In this embodiment, after the current feature set is input into the hard disk fault prediction model based on multi-model fusion, it is possible to predict whether the target model hard disk will fail according to the output fault prediction result, so that an early warning can be issued.
[0084] The beneficial effects of the embodiments of the present invention are:
[0085] Obtain hard disk SMART data of hard disks of different models; extract features from hard disk SMART data based on progressive window sampling to obtain an initial feature set; use extreme random tree model to select features from the initial feature set to obtain the optimal feature set; train multiple machine learning models with the optimal feature set to obtain a hard disk failure prediction model based on multi-model fusion; when failure prediction of the target model hard disk is required, obtain the current SMART data of the target model hard disk and extract the current feature set; input the current feature set into the hard disk failure prediction model based on multi-model fusion to output the failure prediction result of the target model hard disk. Make full use of the feature extraction and time series modeling capabilities of the machine learning integrated model to capture the changing process of each SMART attribute of different models of hard disks and improve the accuracy of hard disk failure prediction;
[0086] By sensing the health status of the hard disk in advance and providing early warning before the hard disk fails, it can help operation and maintenance personnel replace or perform other repair operations on the disk that is about to fail in advance based on the model prediction results, thereby effectively avoiding the impact of equipment instability when failure is imminent on the business, alleviating the pressure on operation and maintenance personnel and reducing the investment in data repair.
[0087] In combination with the hard disk failure prediction method based on multi-model fusion described in the above embodiment, the hard disk failure prediction system based on multi-model fusion is described below through an embodiment.
[0088] like Figure 3 As shown, an embodiment of the present invention provides a hard disk failure prediction system based on multi-model fusion, including:
[0089] The data acquisition module 301 is used to acquire the hard disk SMART data of hard disks of different models;
[0090] A feature extraction module 302 is used to extract features from the hard disk SMART data based on a progressive window sampling method to obtain an initial feature set;
[0091] A feature selection module 303 is used to select features from the initial feature set using an extreme random tree model to obtain an optimal feature set;
[0092] A model training module 304 is used to train multiple machine learning models through an optimal feature set to obtain a hard disk failure prediction model based on multi-model fusion;
[0093] The data acquisition module 301 is also used to acquire the current SMART data of the target model hard disk and extract the current feature set when it is necessary to predict the failure of the target model hard disk;
[0094] The fault prediction module 305 is used to input the current feature set into the hard disk fault prediction model based on multi-model fusion, and output the fault prediction result of the target model hard disk.
[0095] Combination Figure 3 The embodiments shown are preferably some embodiments of the present invention.
[0096] The feature extraction module 302 is specifically used to obtain the degree of change of SMART attributes based on the hard disk SMART data; analyze the correspondence between the degree of change of SMART attributes and the fault label of the corresponding hard disk; take the historical SMART data of the hyperparameter window length value L before the failure occurs as input based on the correspondence; perform feature sampling on the historical SMART data in the sampling window of size L based on the progressive window sampling method to obtain an initial feature set.
[0097] The feature selection module 303 is specifically used to input all the initial features in the initial feature set into the extreme random tree model to obtain the initial model accuracy and the initial feature weight value of each initial feature; remove the initial features whose initial feature weight values are lower than the preset weight threshold from the initial feature set to obtain the first feature set; input the first feature set into the extreme random tree model for training to obtain the first model accuracy and the first feature weight value of each first feature in the first feature set; determine whether the number of features in the first feature set is higher than the preset number threshold; if not, stop feature removal in the first feature set; if higher, determine whether there is a first feature set. A first feature with a feature weight value lower than a preset weight threshold; if it does not exist, the first feature set stops feature elimination; if it does exist, the first feature with a first feature weight value lower than the preset weight threshold is eliminated from the first feature set to obtain a second feature set, and a second model accuracy is obtained; repeat the above steps, input the second feature set into the extreme random tree model to continue training, until the Kth feature set stops feature elimination, and the Kth model accuracy is obtained, K is a positive integer greater than or equal to 2; when feature elimination is stopped, the model accuracy corresponding to each feature set is compared, and the feature set corresponding to the highest model accuracy is selected as the optimal feature set.
[0098] The model training module 304 is specifically used to select the LightGBM algorithm model, the XGBoost algorithm model and the CatBoost algorithm model from multiple machine learning models as the base classifiers of the first layer; the base classifiers of the LightGBM algorithm model, the XGBoost algorithm model and the CatBoost algorithm model are trained in parallel through the optimal feature set to obtain the training prediction results and feature importance rankings of each model; the training prediction results of each model are voted and fused, and the Pearson correlation coefficient is used to analyze the correlation between the top N features in the feature importance ranking of each model to obtain correlation data; the features with greater correlation in the correlation data are selected as the second-layer input features; the second-layer input features are input into the meta-model of the second-layer Bagging algorithm random forest model for training to obtain the final prediction results; according to the trained first-layer base classifiers and the second-layer Bagging algorithm random forest model meta-model, a hard disk failure prediction model based on multi-model fusion is constructed.
[0099] The beneficial effects of the embodiments of the present invention are:
[0100] The data acquisition module 301 acquires the hard disk SMART data of hard disks of different models; the feature extraction module 302 extracts features from the hard disk SMART data based on the progressive window sampling method to obtain the initial feature set; the feature selection module 303 uses the extreme random tree model to select features from the initial feature set to obtain the optimal feature set; the model training module 304 trains multiple machine learning models through the optimal feature set to obtain a hard disk fault prediction model based on multi-model fusion; when it is necessary to predict the fault of the target model hard disk, the data acquisition module 301 acquires the current SMART data of the target model hard disk and extracts the current feature set; the fault prediction module 305 inputs the current feature set into the hard disk fault prediction model based on multi-model fusion and outputs the fault prediction result of the target model hard disk. Make full use of the feature extraction capability and time series modeling capability of the machine learning integrated model to capture the changing process of each SMART attribute of different models of hard disks and improve the accuracy of hard disk fault prediction;
[0101] By sensing the health status of the hard disk in advance and providing early warning before the hard disk fails, it can help operation and maintenance personnel replace or perform other repair operations on the disk that is about to fail in advance based on the model prediction results, thereby effectively avoiding the impact of equipment instability when failure is imminent on the business, alleviating the pressure on operation and maintenance personnel and reducing the investment in data repair.
[0102] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0103] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0104] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0105] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0106] The above are merely embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention are included in the scope of the claims of the present invention to be approved.
Claims
1. A hard disk failure prediction method based on multi-model fusion, characterized in that: include: Get the hard disk SMART data of different models of hard disks; Extracting features from the hard disk SMART data based on a progressive window sampling method to obtain an initial feature set; Using an extreme random tree model to perform feature selection on the initial feature set to obtain an optimal feature set; Training multiple machine learning models using the optimal feature set to obtain a hard disk failure prediction model based on multi-model fusion; When a fault prediction is required for a target hard disk, the current SMART data of the target hard disk is obtained to extract a current feature set; The current feature set is input into the hard disk fault prediction model based on multi-model fusion, and the fault prediction result of the target model hard disk is output.
2. The hard disk failure prediction method based on multi-model fusion according to claim 1 is characterized in that: The feature extraction of the hard disk SMART data based on the progressive window sampling method to obtain an initial feature set includes: Obtaining a degree of change in SMART attributes according to the hard disk SMART data; Analyze the corresponding relationship between the degree of change of the SMART attribute and the fault label of the corresponding hard disk; According to the corresponding relationship, historical SMART data of the hyperparameter window length value L before the failure occurs is used as input; Based on the progressive window sampling method, feature sampling is performed on the historical SMART data within the sampling window of size L to obtain an initial feature set.
3. The hard disk failure prediction method based on multi-model fusion according to claim 1 is characterized in that: The method of using the extreme random tree model to perform feature selection on the initial feature set to obtain the optimal feature set includes: Inputting all the initial features in the initial feature set into the extreme random tree model to obtain the initial model accuracy and the initial feature weight value of each initial feature; Eliminate the initial features whose initial feature weight values are lower than a preset weight threshold from the initial feature set to obtain a first feature set; Inputting the first feature set into the extreme random tree model for training to obtain a first model accuracy and a first feature weight value of each first feature in the first feature set; Determine whether the number of features in the first feature set is higher than a preset number threshold; If it is not higher, the first feature set stops feature elimination; If it is higher, determining whether there is a first feature whose weight value of the first feature is lower than the preset weight threshold; If it does not exist, the first feature set stops feature elimination; If so, the first feature whose weight value is lower than the preset weight threshold is removed from the first feature set to obtain a second feature set, and a second model accuracy is obtained; Repeat the above steps, input the second feature set into the extreme random tree model to continue training, until the Kth feature set stops feature elimination, and the Kth model accuracy is obtained, where K is a positive integer greater than or equal to 2; After stopping feature elimination, compare the model accuracy corresponding to each feature set, and select the feature set with the highest model accuracy as the optimal feature set.
4. The hard disk failure prediction method based on multi-model fusion according to claim 1 is characterized in that: The method of training multiple machine learning models by using the optimal feature set to obtain a hard disk failure prediction model based on multi-model fusion includes: Select LightGBM algorithm model, XGBoost algorithm model and CatBoost algorithm model from multiple machine learning models as the base classifier of the first layer; The base classifiers of the LightGBM algorithm model, the XGBoost algorithm model and the CatBoost algorithm model are trained in parallel by using the optimal feature set to obtain the training prediction results and feature importance ranking of each model; Voting and fusing the training prediction results of each model, and using the Pearson correlation coefficient to analyze the correlation between the top N features in the feature importance ranking of each model to obtain correlation data; Selecting features with greater correlation in the correlation data as second-layer input features; Input the second layer input features into the meta-model of the second layer Bagging algorithm random forest model for training to obtain the final prediction result; According to the trained base classifier of the first layer and the meta-model of the Bagging algorithm random forest model of the second layer, a hard disk failure prediction model based on multi-model fusion is constructed.
5. A hard disk failure prediction system based on multi-model fusion, characterized in that: include: A data acquisition module is used to obtain hard disk SMART data of hard disks of different models; A feature extraction module, used for extracting features from the hard disk SMART data based on a progressive window sampling method to obtain an initial feature set; A feature selection module, used for performing feature selection on the initial feature set using an extreme random tree model to obtain an optimal feature set; A model training module, used to train multiple machine learning models using the optimal feature set to obtain a hard disk failure prediction model based on multi-model fusion; The data acquisition module is also used to acquire the current SMART data of the target model hard disk and extract the current feature set when it is necessary to predict the failure of the target model hard disk; The fault prediction module is used to input the current feature set into the hard disk fault prediction model based on multi-model fusion, and output the fault prediction result of the target model hard disk.
6. The hard disk failure prediction system based on multi-model fusion according to claim 5 is characterized in that: The feature extraction module is specifically used to obtain the degree of change of SMART attributes according to the hard disk SMART data; analyze the correspondence between the degree of change of the SMART attributes and the fault label of the corresponding hard disk; take the historical SMART data of the hyperparameter window length value L before the failure occurs as input according to the correspondence; perform feature sampling on the historical SMART data in the sampling window of size L based on the progressive window sampling method to obtain an initial feature set.
7. The hard disk failure prediction system based on multi-model fusion according to claim 5 is characterized in that: The feature selection module is specifically used to input all the initial features in the initial feature set into the extreme random tree model to obtain the initial model accuracy and the initial feature weight value of each initial feature; remove the initial features whose initial feature weight value is lower than the preset weight threshold from the initial feature set to obtain the first feature set; input the first feature set into the extreme random tree model for training to obtain the first model accuracy and the first feature weight value of each first feature in the first feature set; determine whether the number of features in the first feature set is higher than the preset number threshold; if not, stop feature removal from the first feature set; if higher, determine whether there is a first feature whose first feature weight value is lower than the preset weight threshold; if not, stop feature removal from the first feature set; If it exists, the first feature whose weight value is lower than the preset weight threshold is removed from the first feature set to obtain a second feature set, and a second model accuracy is obtained; repeat the above steps, input the second feature set into the extreme random tree model to continue training, until the Kth feature set stops feature removal, and the Kth model accuracy is obtained, where K is a positive integer greater than or equal to 2; after stopping feature removal, compare the model accuracy corresponding to each feature set, and select the feature set corresponding to the highest model accuracy as the optimal feature set.
8. The hard disk failure prediction system based on multi-model fusion according to claim 5 is characterized in that: The model training module is specifically used to select a LightGBM algorithm model, an XGBoost algorithm model and a CatBoost algorithm model from a plurality of machine learning models as the base classifiers of the first layer; the base classifiers of the LightGBM algorithm model, the XGBoost algorithm model and the CatBoost algorithm model are respectively trained in parallel through the optimal feature set to obtain the training prediction results of each model and the feature importance ranking; Voting and fusing the training prediction results of each model, and using the Pearson correlation coefficient to analyze the correlation between the top N features in the feature importance ranking of each model to obtain correlation data; selecting features with greater correlation in the correlation data as second-layer input features; The second-layer input features are input into the meta-model of the Bagging algorithm random forest model of the second layer for training to obtain the final prediction result; according to the trained base classifier of the first layer and the meta-model of the Bagging algorithm random forest model of the second layer, a hard disk failure prediction model based on multi-model fusion is constructed.
Citation Information
Cited By
Disk fault early warning method, device and equipment based on multi-feature fusion and medium
CN120929338A