A hard disk failure prediction method, device, storage medium and electronic device
By collecting hard drive SMART data and performance data, and combining pre-trained models and expert systems for secondary judgment, the problem of low accuracy in hard drive fault prediction has been solved, the prediction accuracy has been improved, the interpretability has been enhanced, and the number of fault-free repairs has been reduced.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ZTE INTELLIGENT TECH NANJING CO LTD
- Filing Date
- 2022-06-09
- Publication Date
- 2026-07-07
AI Technical Summary
Current technologies using artificial intelligence to predict hard drive failures are not very accurate, resulting in a high rate of repairs even when no faults are found, and the inability to explain risk conditions leads to user skepticism and distrust.
The system collects hard drive SMART data and performance data, concatenates them into target data, and inputs it into a pre-trained model. It then combines this data with an expert system for secondary judgment, further determining the risk of failure through SMART and performance data, thereby improving prediction accuracy and interpretability.
It improved the accuracy of hard drive failure prediction, reduced the proportion of repairs without faults, enhanced the interpretability of prediction results, and reduced user doubts about the system.
Smart Images
Figure CN117271229B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of communications, and more specifically, to a hard disk fault prediction method, apparatus, storage medium, and electronic device. Background Technology
[0002] In large-scale data centers, the decline in IT infrastructure stability and reliability caused by hard drive failures is a frequent problem. Since the beginning of the 21st century, academia has conducted extensive research on hard drive failure prediction and diagnosis. These studies combine big data and artificial intelligence methods, achieving high accuracy rates. However, for hard drive failure prediction in specific industrial scenarios, there are many problems such as complex environmental operations, high noise levels, and sudden device failures. The accuracy and generalization ability of hard drive failure prediction algorithms still do not achieve satisfactory results in industrial applications.
[0003] Both academia and industry have conducted extensive research on predicting hard drive failures. Academia typically focuses on single hard drive models, achieving an accuracy rate of around 90% for failure prediction. Industry, however, needs to consider complex environments, multiple hard drive manufacturers, and various models, resulting in an overall accuracy rate of only around 80%. Incorrect predictions increase the rate of no-trouble-found (NTF) repairs, impacting data center quality metrics. Furthermore, incorrect predictions can lead to incorrect hardware replacements, causing financial losses for data centers. Additionally, when using artificial intelligence to predict hard drive failures, it often fails to explain to users why a drive is in a "risky" state, leading to skepticism and distrust of the hard drive failure prediction system.
[0004] There is currently no solution to the problem that artificial intelligence in related technologies is not accurate enough in predicting hard drive failures, resulting in high NTF values and the inability to explain to users why a drive is in a risky state, which leads to user doubts and distrust of the hard drive failure prediction system. Summary of the Invention
[0005] This application provides a hard disk failure prediction method, apparatus, storage medium, and electronic device to at least address the issues in related technologies where artificial intelligence-based hard disk failure prediction is inaccurate, resulting in high NTF values and the inability to explain to users why a disk is in a risky state, leading to user skepticism and distrust of the hard disk failure prediction system.
[0006] According to one embodiment of this application, a hard disk failure prediction method is provided, the method comprising:
[0007] Collect hard disk data to be processed, wherein the hard disk data includes self-monitoring analysis and reporting technology (SMART) data and performance data;
[0008] The SMART data and the performance data are concatenated to obtain the target hard drive data;
[0009] The target hard disk data is input into a pre-trained fault prediction model to obtain the first fault prediction result output by the fault prediction model.
[0010] If the first fault prediction result indicates that there is a fault risk, the second fault prediction result is obtained by further determining whether there is a fault risk in the hard disk data to be processed based on the SMART data and performance data of the hard disk data to be processed.
[0011] The hard drive failure risk level is determined based on the second failure prediction result.
[0012] According to another embodiment of this application, a hard disk failure prediction device is also provided, the device comprising:
[0013] The first acquisition module is used to acquire hard disk data to be processed, wherein the hard disk data includes SMART data and performance data;
[0014] The first splicing module is used to splice the SMART data and the performance data to obtain the target hard disk data;
[0015] The input module is used to input the target hard disk data into a pre-trained fault prediction model to obtain the first fault prediction result output by the fault prediction model.
[0016] The secondary prediction module is used to determine whether the hard disk data to be processed has a risk of failure based on the SMART data and performance data of the hard disk data to be processed when the first fault prediction result indicates that there is a risk of failure, and to obtain a second fault prediction result.
[0017] The first determining module is used to determine the hard drive failure risk level based on the second failure prediction result.
[0018] According to yet another embodiment of this application, a computer-readable storage medium is also provided, wherein a computer program is stored therein, wherein the computer program is configured to perform the steps in any of the above method embodiments when it is run.
[0019] According to yet another embodiment of this application, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.
[0020] In this embodiment, hard disk data to be processed is collected, including SMART data and performance data. The SMART data and performance data are concatenated to obtain target hard disk data. The target hard disk data is input into a pre-trained fault prediction model to obtain a first fault prediction result output by the fault prediction model. If the first fault prediction result indicates a fault risk, a second fault prediction result is obtained by further determining whether the hard disk data to be processed has a fault risk based on the SMART data and performance data of the hard disk data to be processed. This addresses the issues of high NTF (Network Time To Failure) in related technologies where artificial intelligence-based hard disk fault prediction is inaccurate and often fails to explain to users why a faulty disk is in a risky state, leading to user skepticism and distrust of the hard disk fault prediction system. By combining expert systems and fault prediction models to predict hard disk faults, the accuracy of prediction is improved, and the NTF is reduced when the fault prediction model is inaccurate, thus improving the interpretability of the prediction results. Attached Figure Description
[0021] Figure 1 This is a hardware structure block diagram of a mobile terminal for the hard disk failure prediction method according to an embodiment of this application.
[0022] Figure 2 This is a flowchart of a hard disk failure prediction method according to an embodiment of this application;
[0023] Figure 3 This is a flowchart of a hard disk failure prediction method according to an optional embodiment of this application;
[0024] Figure 4 This is a flowchart of hard disk data preprocessing according to this embodiment;
[0025] Figure 5 This is a flowchart of hard disk data tagging according to this embodiment;
[0026] Figure 6 This is a flowchart of data cleaning according to this embodiment;
[0027] Figure 7 This is a flowchart of hard disk failure prediction and risky hard disk handling according to this embodiment;
[0028] Figure 8 This embodiment is a flowchart of how an expert system determines a risky hard drive;
[0029] Figure 9 This is a flowchart of the faulty hard drive handling process according to this embodiment;
[0030] Figure 10 This is a flowchart illustrating the determination of the automatic hard disk backup cycle according to this embodiment;
[0031] Figure 11 This is a flowchart of hard disk data backup according to this embodiment;
[0032] Figure 12 This is a block diagram of a hard disk failure prediction device according to an embodiment of this application;
[0033] Figure 13 This is a block diagram of a hard disk failure prediction device according to an optional embodiment of this application. Detailed Implementation
[0034] The embodiments of this application will be described in detail below with reference to the accompanying drawings and examples.
[0035] It should be noted that the terms "first," "second," etc., in the specification, claims, and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.
[0036] The methods and embodiments provided in this application can be executed on a mobile terminal, a computer terminal, or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 This is a hardware structure block diagram of a mobile terminal for the hard disk fault prediction method according to an embodiment of this application, as shown below. Figure 1 As shown, a mobile terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. The mobile terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the mobile terminal described above. For example, the mobile terminal may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.
[0037] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the hard disk failure prediction method in this embodiment. The processor 102 executes various functional applications and business chain address pool slicing processing by running the computer program stored in the memory 104, thus implementing the above-described method. The memory 104 may include high-speed random access memory and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.
[0038] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the mobile terminal's communication provider. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.
[0039] This embodiment provides a hard disk failure prediction method that runs on the aforementioned mobile terminal or network architecture. Figure 2 This is a flowchart of a hard disk failure prediction method according to an embodiment of this application, such as... Figure 2 As shown, the process includes the following steps:
[0040] Step S202: Collect hard disk data to be processed, wherein the hard disk data includes hard disk SMART data and performance data;
[0041] Step S204: Combine the SMART data and performance data to obtain the target hard drive data;
[0042] Step S206: Input the target hard disk data into the pre-trained fault prediction model to obtain the fault prediction result output by the fault prediction model.
[0043] Step S208: If the fault prediction result indicates that there is a risk of fault, perform a second fault prediction on the hard disk data to be processed to obtain a second fault prediction result.
[0044] In this embodiment, in step S208 above, it is determined whether the second fault prediction result indicates the presence of a hard disk risk based on the SMART data of the hard disk data to be processed; if the determination result is yes, it is determined that the second fault prediction result indicates the presence of a hard disk risk; if the determination result is no, it is determined whether the second fault prediction result indicates the presence of a hard disk risk based on the performance data; if the determination result is yes, it is determined that the second fault prediction result indicates the presence of a hard disk risk; if the determination result is no, it is determined that the second fault prediction result indicates the absence of a hard disk risk. Furthermore, the aforementioned SMART data includes at least: SMART5 (remapped sector count), SMART187 (count of uncorrectable errors), SMART188 (command timeout count), SMART197 (count of currently unmapped sectors), and SMART198 (count of offline uncorrectable sectors). If the original value of SMART5 is greater than a first preset value (e.g., 500), the second fault prediction result is determined to be a hard drive risk; if the original value of SMART187 is greater than a second preset value (e.g., 100), the second fault prediction result is determined to be a hard drive risk; if the original value of SMART188 is greater than a third preset value (e.g., 100), the second fault prediction result is determined to be a hard drive risk; if SMART197 or... If the original value of SMART198 is greater than the fourth preset value (e.g., 10), the second fault prediction result is determined to be that there is a hard drive risk. If the performance data shows that the average number of files successfully read per second is greater than the fifth preset value (e.g., 50), the second fault prediction result is determined to be that there is a hard drive risk. If none of the above conditions are met, that is, if the original value of SMART5 is not greater than the first preset value, the original value of SMART187 is not greater than the second preset value, the original value of SMART188 is not greater than the third preset value, the original value of SMART197 is not greater than the fourth preset value, the original value of SMART198 is not greater than the fourth preset value, and the performance data shows that the average number of files successfully read per second is not greater than the fifth preset value, the second fault prediction result is determined to be that there is no hard drive risk. Of course, other SMART data can also be used to determine whether the second fault prediction result indicates the presence of a hard drive risk. The determination method is similar to the above and will not be elaborated here.Additionally, it should be noted that the order in which SMART5 (remapped sector count), SMART187 (count of uncorrectable errors), SMART188 (command timeout count), SMART197 (count of currently unmapped sectors), and SMART198 (count of offline uncorrectable sectors) are judged in the above-mentioned judgment process is not limited by the embodiment. One SMART data can be judged first. If the judgment result is yes, the judgment ends. If the judgment result is no, one of the remaining SMART data is selected to continue the judgment. If the judgment result is yes, the judgment ends. Otherwise, one of the remaining SMART data is selected for judgment, and so on, until all SMART data has been judged and the final result is obtained. This will not be elaborated on here.
[0045] Step S210: Determine the hard drive failure risk level based on the second failure prediction result.
[0046] In this embodiment, step S208 may specifically include: if the fault prediction result is abnormal, determining for the second time whether the hard disk data to be processed has a fault risk based on the SMART data and performance data of the hard disk data to be processed, and obtaining a second fault prediction result; if the second fault prediction result indicates that there is a fault risk, determining the hard disk fault risk level as the first level; if the second fault prediction result indicates that there is no fault risk, determining the hard disk fault risk level as the second level, wherein the risk level of the first level is higher than the risk level of the second level.
[0047] Through the above steps S202 to S208, the problems of high NTF when artificial intelligence predicts hard drive failures are not accurate enough in related technologies, and the inability to explain to users why a risky drive is in a risky state, which leads to users questioning and distrusting the hard drive failure prediction system, can be solved. By combining expert systems and fault pre-models to predict hard drive failures, the prediction accuracy is improved. At the same time, when the fault pre-model prediction is not accurate enough, the NTF is reduced, and the interpretability of the prediction results is improved.
[0048] In one embodiment, before step S204 above, feature extraction is performed on the hard disk data to be processed to obtain feature data to be processed; the feature data to be processed is filtered to obtain target feature data to be processed; it is determined that the target feature data to be processed meets the preset requirements, specifically, it is determined whether the data time length of the target feature data to be processed is greater than or equal to the preset time length and whether the number of sampling points is greater than or equal to the preset value; if the determination result is yes, it is determined that the target feature data to be processed meets the preset requirements.
[0049] In another embodiment, after step S208, if the hard drive failure risk level is level one, a prompt is made to replace the hard drive corresponding to the data to be processed; if the hard drive failure risk level is level two, the hard drive data to be processed is backed up. Specifically, the RAID configuration of the hard drive corresponding to the data to be processed is obtained; if the RAID configuration is RAID0, the hard drive data is backed up daily; if the RAID configuration is RAID5, the hard drive data is backed up weekly; if the RAID configuration is RAID1 or a RIAID with a level greater than RAID1, the hard drive data is backed up according to a preset time period; if the RAID configuration is other types of RAID, the hard drive data is backed up weekly, where other types of RAID are RAID excluding RAID0, RAID5, RAID1, and RIAIDs with a level greater than RAID1. For hard drives predicted as risky by AI, they are divided into two categories according to the expert system. If the expert system determines that the hard drive is in a risky state, the hard drive is directly replaced; if the expert system does not determine that the hard drive is risky, the data of the risky hard drive is automatically backed up. For high-risk hard drives, the hard drive is directly replaced; for medium- and low-risk hard drives, the data is automatically backed up.
[0050] Figure 3 This is a flowchart of a hard disk failure prediction method according to an optional embodiment of this application, such as... Figure 3 As shown, prior to step S202 above, the process includes the following steps:
[0051] Step S302: Collect a preset amount of hard disk data, wherein the hard disk data includes SMART data and performance data;
[0052] Step S304: Combine a preset number of SMART data and performance data to form a training dataset;
[0053] Step S306: Train the fault prediction model based on the training dataset to obtain the trained fault prediction model.
[0054] Through steps S302 to S306 above, the fault prediction model can be trained to predict faults in the hard drive data to be processed. Big data analytics methods are used to analyze hard drive SMART data and performance data, machine learning algorithms are used to train the AI on the data, and the trained model is used to predict hard drive faults.
[0055] In one embodiment, prior to step S304 above, invalid and noisy data in the training dataset are cleaned, and missing data is filled in. Negative samples of faulty disks in the training data are deleted to reduce data noise.
[0056] In another embodiment, prior to step S304 above, feature extraction is performed on a preset number of hard disk data to obtain feature data corresponding to the preset number of hard disk data; the feature data corresponding to the preset number of hard disk data is filtered to obtain target feature data corresponding to the preset number of hard disk data; it is determined that the target feature data corresponding to the preset number of hard disk data meets preset requirements. Specifically, it is determined whether the data time length of the target feature data corresponding to the preset number of hard disk data is greater than or equal to a preset time length and whether the number of sampling points is greater than or equal to a preset value; if the determination result is yes, it is determined that the target feature data corresponding to the preset number of hard disk data meets the preset requirements.
[0057] In this embodiment, step S306 may specifically include:
[0058] S1, Set the label for each data item in the training dataset according to the hard drive failure time;
[0059] Furthermore, in S1 above, the label can be set in the following way: comparing the hard drive failure time and the data acquisition time; if the interval between the data acquisition time and the hard drive failure time is less than N days, the label is set to 1; if the interval between the data acquisition time and the hard drive failure time is greater than N days and less than M days, and the original value of one of the multiple attribute fields of the SMART data is greater than 0, the label is set to 1; if the interval between the data acquisition time and the hard drive failure time is greater than N days and less than M days, and the original value of multiple attribute fields of the SMART data is equal to 0, the label is set to 0; if the interval between the data acquisition time and the hard drive failure time is greater than M days, the label is set to 0, where 1 represents risk and 0 represents no risk. When labeling the training data, expert experience is combined to segment the data, making the data labels more consistent with the actual state of the hard drive and have better interpretability.
[0060] S2, The fault prediction model is trained based on the training dataset, and the trained fault prediction model is obtained when the loss function meets the preset conditions.
[0061] The loss function L used in this embodiment or fl for: Where α is the balance factor, γ is the modulation parameter, y′ is the predicted value, and y is the actual sample value. Using the Focal Loss function as the loss function can solve the problem of severe imbalance between positive and negative samples in hard disk data.
[0062] Figure 4 This is a flowchart of hard disk data preprocessing according to this embodiment, such as... Figure 4 As shown, the data processing method includes the following steps:
[0063] Step S401: Collect hard disk data;
[0064] Two types of data are collected: 1. SMART data: The SmartCtrl tool is used to collect SMART data from the hard drive, once a day at 3 AM. 2. Hard drive performance data: In-band tools are used to collect hard drive performance data, once an hour.
[0065] Step S402: Extract features from the hard disk data;
[0066] The two types of data were analyzed using the Pearson correlation coefficient, and feature columns with high correlation and no change in data were removed.
[0067] Step S403: Verify the hard disk data;
[0068] Verify whether the data duration and number of sampling points meet the minimum data requirements for fault detection and prediction: data should be collected once a day for three consecutive days, and at least one of the two types of data should be collected.
[0069] Step S404: Combine SMART data and performance data to form a training dataset.
[0070] Step S405: Set labels for the hard disk data in the training dataset;
[0071] The label for each data point in the training dataset is calculated based on the hard drive failure time. For normal hard drives, the label is uniformly set to 0. Figure 5 This is a flowchart of hard disk data tagging according to this embodiment, such as... Figure 5 As shown, it includes the following steps:
[0072] Step S501: Record the time of failure of the faulty hard drive;
[0073] For faulty hard drives, record the date of the failure.
[0074] Step S502: Determine if the hard drive has failed;
[0075] For hard drives that have not failed, each data entry is marked as 0; otherwise, proceed to step S503.
[0076] Step S503: For the hard drive that has failed, determine the difference in the number of days between the data acquisition time and the failure time.
[0077] If the number of days is less than 2 days, mark it as 1. If the number of days is greater than 5 days, mark it as 0. Otherwise, proceed to step S504.
[0078] Step S504: For data with a difference of 2 to 5 days, check the SMART attribute of the data and set labels for the hard disk data according to the SMART attribute;
[0079] Specifically, if any of the five attributes SMART5, SMART187, SMART188, SMART197, and SMART198 has a value greater than 0, it is marked as 1; otherwise, it is marked as 0.
[0080] The selection of SMART attributes and corresponding thresholds in this embodiment are only for illustrating the steps of the scheme. In actual implementation, the attribute selection and thresholds can be adjusted according to the specific circumstances.
[0081] Step S406: Clean the hard disk data in the training dataset;
[0082] Fill missing attribute values with 0 to reduce noise in the data. Figure 6 This is a flowchart of data cleaning according to this embodiment, such as... Figure 6 As shown, it includes the following steps:
[0083] Step S601: View the label of each data entry for the faulty disk in the dataset. Specifically, iterate through the data of the faulty disk in the dataset for each day and view the labels.
[0084] Step S602: Determine if the label is 0;
[0085] Determine if the label of the faulty disk is 0. Faulty disk samples with a label of 0 are considered noise data.
[0086] Step S603: Delete noisy data, that is, delete data with label 0.
[0087] Figure 7 This is a flowchart of hard disk failure prediction and risky hard disk handling according to this embodiment, such as... Figure 7 As shown, the data processing method includes the following steps:
[0088] Step S701: Collect hard disk data;
[0089] This includes training data collection and test data collection. The training data needs to record the hard drive failure time and hard drive serial number for labeling.
[0090] Step S702, data processing;
[0091] For training data, the above method can be used for data processing; for test data, the above method can be used for data processing, but the data labeling and data noise reduction steps are omitted.
[0092] Step S703: Train the fault prediction model;
[0093] The LightGBM binary classification algorithm was used to train the dataset. The objective function of the algorithm used the FocalLoss loss function, and the formula for the FocalLoss function is as follows:
[0094] Where α is the balance factor, γ is the modulation parameter, y′ is the predicted value, and y is the actual value of the sample.
[0095] Step S704: Evaluate the trained fault prediction model;
[0096] 30% of the data in the training set is randomly selected as the validation set, and the remaining data is used as the training set. For the validation set data, the F1-Score is used as the evaluation metric. The relevant terminology and detailed metrics are defined as follows, where precision is the accuracy rate and recall is the recall rate: n pp This refers to the number of hard drives predicted to fail within the next 30 days within the assessment window, n. tpp This refers to the number of faulty memory instances detected 30 days in advance within the assessment window, n. tr This refers to the total number of hard drive failures within the evaluation window, n. tpr This refers to the number of faulty hard drives that are detected 30 days in advance within the assessment window.
[0097]
[0098]
[0099]
[0100] Step S705: Infer the hard disk data based on the trained fault prediction model;
[0101] The trained model is used to infer the test data, and the result of the inference should be a value between 0 and 1.
[0102] Step S706: Perform secondary prediction on the hard disk data using an expert system;
[0103] For hard drives identified as risky by AI models, the data is input into an expert system. Figure 8 This embodiment is a flowchart of how an expert system determines risky hard drives, as shown below. Figure 8 As shown, the expert system consists of two sets of rules: one is the SMART rule set, and the other is the performance data rule set, including the following steps:
[0104] Step S801: View the hard drive SMART data and performance data;
[0105] Input the hard drive data that the AI model identifies as risky into the expert system to view the raw values and performance data of these hard drives' SMART data.
[0106] Step S802: Determine whether the original value of SMART5 is greater than 500 (an example of the first preset value mentioned above). If the determination result is no, proceed to step S803. If the determination result is yes, proceed to step S807.
[0107] Step S803: Determine whether the original value of SMART187 is greater than 100 (an example of the second preset value mentioned above). If the determination result is no, proceed to step S804. If the determination result is yes, proceed to step S807.
[0108] Step S804: Determine whether the original value of SMART188 is greater than 100 (an example of the third preset value mentioned above). If the determination result is no, proceed to step S805. If the determination result is yes, proceed to step S807.
[0109] Step S805: Determine whether the original values of SMART197 and SMART198 are greater than 10 (an example of the fourth preset value mentioned above). If the determination result is no, proceed to step S806. If the determination result is yes, proceed to step S807.
[0110] Step S806: Determine whether the average number of files successfully read per second on the hard disk is greater than 50 (an example of the fifth preset value mentioned above). If the determination result is no, proceed to step S807. If the determination result is yes, proceed to step S808.
[0111] If the hard drive successfully reads more than 50 files per second on average, it is considered to be at risk; otherwise, it is considered to be normal.
[0112] Step S807: Confirm that the hard drive is working properly.
[0113] Step S808, Risk Hard Drive Handling: Specifically, for hard drives that are simultaneously identified as risky by both the expert system and AI, it is recommended that users replace the hard drive.
[0114] The selection of SMART attributes and performance attributes and the corresponding thresholds in this embodiment are only for illustrating the steps of the scheme. In actual implementation, the attribute selection and thresholds can be adjusted according to the specific situation.
[0115] Figure 9 This is a flowchart of the faulty hard drive handling process according to this embodiment, such as... Figure 9 As shown, it includes the following steps:
[0116] Step S901: Obtain the first fault prediction result obtained by the fault prediction model through reasoning on the hard disk data;
[0117] After preprocessing the hard drive data to be processed, the trained fault prediction model is used to infer the hard drive data.
[0118] Step S902: Determine whether the first fault prediction result indicates that the hard drive is normal. If the result is negative, proceed to step S903.
[0119] If the reasoning result is normal, the hard drive is considered to be in a healthy state; otherwise, proceed to step S903.
[0120] Step S903: Obtain the second fault prediction result obtained from the expert system's secondary prediction;
[0121] Step S904: Determine the hard drive risk level based on the second fault prediction result, and process the risky hard drive according to the hard drive risk level.
[0122] If the expert system's second fault prediction result indicates that the hard drive is at risk and has a high risk level (corresponding to the first level), then the hard drive will be replaced. If the expert system's second fault prediction result indicates that the hard drive is not at risk and has a low risk level (corresponding to the second level), then the hard drive data will be automatically backed up at regular intervals.
[0123] For hard drives that AI identifies as risky but the expert system deems risk-free, the system considers the hard drive to be in a low-risk state and automatically backs up the hard drive data. Figure 10 This is a flowchart illustrating the determination of the automatic hard disk backup cycle according to this embodiment, such as... Figure 10 As shown, it includes the following steps:
[0124] Step S1001: Check the configuration of the Redundant Arrays of Independent Disks (RAID).
[0125] Step S1002: If the RAID is configured as RAID0, the data will be automatically backed up every morning at midnight; otherwise, proceed to step S1003.
[0126] Step S1003: If the RAID configuration is RAID5, automatically back up data weekly; otherwise, proceed to step S1004.
[0127] Step S1004: If the RAID is configured as RAID1 or a higher level RAID, no backup is performed by default, but the user can configure the automatic backup cycle via mobile phone; otherwise, proceed to step S1005.
[0128] Step S1005: If the RAID configuration is other types, automatically back up the data weekly.
[0129] The backup cycle in this embodiment is only for illustrating the steps of the scheme. In actual implementation, the backup cycle can be adjusted according to the specific circumstances.
[0130] Figure 11 This is a flowchart of hard disk data backup according to this embodiment, such as... Figure 11 As shown, it includes the following steps:
[0131] Step S1101: Compress the hard drive data that needs to be backed up;
[0132] Step S1102: Determine if the local machine has a spare hard drive. If the result is yes, proceed to step S1103. If the result is no, proceed to step S1104.
[0133] Step S1103: Back up the compressed data to other spare hard drives in the data center;
[0134] Step S1104: Back up the compressed data to the spare hard drive.
[0135] According to another aspect of the embodiments of this application, a hard disk failure prediction device is also provided. Figure 12 This is a block diagram of a hard disk failure prediction device according to an embodiment of this application, such as... Figure 12 As shown, the device includes:
[0136] The first acquisition module 122 is used to acquire hard disk data to be processed, wherein the hard disk data includes SMART data and performance data;
[0137] The first splicing module 124 is used to splice the SMART data and the performance data to obtain the target hard disk data;
[0138] Input module 126 is used to input the target hard disk data into a pre-trained fault prediction model to obtain the first fault prediction result output by the fault prediction model;
[0139] The secondary prediction module 128 is used to determine whether the hard disk data to be processed has a risk of failure based on the SMART data and performance data of the hard disk data to be processed when the first fault prediction result indicates that there is a risk of failure, and to obtain a second fault prediction result.
[0140] The first determining module 1210 is used to determine the hard disk failure risk level based on the second failure prediction result.
[0141] In one embodiment, the device further includes:
[0142] The first feature extraction module is used to extract features from the hard disk data to be processed, and obtain the feature data to be processed.
[0143] The first filtering module is used to filter the feature data to be processed to obtain the target feature data to be processed.
[0144] The second determining module is used to determine whether the target feature data to be processed meets preset requirements.
[0145] In one embodiment, the first determining module is further configured to determine the hard disk failure risk level as a first level if the second fault prediction result indicates that there is a fault risk; and to determine the hard disk failure risk level as a second level if the second fault prediction result indicates that there is no fault risk, wherein the risk level of the first level is higher than that of the second level.
[0146] In one embodiment, the device further includes:
[0147] The prompting module is used to prompt for replacement of the hard drive corresponding to the hard drive data to be processed when the hard drive failure risk level is the first level;
[0148] The backup module is used to back up the hard disk data to be processed when the hard disk failure risk level is the second level.
[0149] In one embodiment, the backup module is further configured to obtain the RAID configuration of the hard drive corresponding to the hard drive data to be processed; if the RAID configuration is RAID0, the hard drive data is backed up daily; if the RAID configuration is RAID5, the hard drive data is backed up weekly; if the RAID configuration is RAID1 or a RIAID with a level greater than RAID1, the hard drive data is backed up according to a preset time period; if the RAID configuration is other types of RAID, the hard drive data is backed up weekly, wherein the other types of RAID are RAIDs other than RAID0, RAID5, RAID1 and RIAIDs with a level greater than RAID1.
[0150] In one embodiment, the secondary prediction module 128 is further configured to determine whether the second fault prediction result indicates the presence of a hard disk risk based on the SMART data of the hard disk data to be processed; if the determination result is yes, determine that the second fault prediction result indicates the presence of a hard disk risk; if the determination result is no, determine whether the second fault prediction result indicates the presence of a hard disk risk based on the performance data; if the determination result is yes, determine that the second fault prediction result indicates the presence of a hard disk risk; if the determination result is no, determine that the second fault prediction result indicates the absence of a hard disk risk.
[0151] In one embodiment, the secondary prediction module 128 is further configured to: determine whether the original value of SMART5 is greater than a first preset value if the SMART data includes SMART5, SMART187, SMART188, SMART197, and SMART198; and if the determination result is yes, determine that the second fault prediction result indicates a hard drive risk; if the determination result is no, determine whether the original value of SMART187 is greater than a second preset value; if the determination result is yes, determine that the second fault prediction result indicates a hard drive risk; and if the determination result is no, determine whether the original value of SMART187 is greater than a second preset value. If the value is greater than a third preset value, and the result is yes, then the second fault prediction result is determined to indicate that there is a hard drive risk. If the result is no, then the original value of SMART197 or SMART198 is determined to be greater than a fourth preset value. If the result is yes, then the second fault prediction result is determined to indicate that there is a hard drive risk. If the result is no, then the average number of files successfully read per second by the hard drive in the performance data is determined to be greater than a fifth preset value. If the result is yes, then the second fault prediction result is determined to indicate that there is a hard drive risk. If the result is no, then the second fault prediction result is determined to indicate that there is no hard drive risk.
[0152] In one embodiment, the device further includes:
[0153] The second acquisition module is used to acquire a preset amount of hard disk data, wherein the hard disk data includes SMART data and performance data;
[0154] The second splicing module is used to splice a preset number of SMART data and the performance data together to form a training dataset.
[0155] The training module is used to train the fault prediction model based on the training dataset to obtain the trained fault prediction model.
[0156] In one embodiment, the device further includes:
[0157] The cleaning module is used to clean invalid and noisy data in the training dataset and fill in missing data.
[0158] In one embodiment, the device further includes:
[0159] The second feature extraction module is used to extract features from the preset number of hard disk data to obtain feature data corresponding to the preset number of hard disk data.
[0160] The second filtering module is used to filter the feature data corresponding to the preset number of hard disk data to obtain the target feature data corresponding to the preset number of hard disk data.
[0161] The third determining module is used to determine whether the target feature data corresponding to the preset number of hard disk data meets the preset requirements.
[0162] In one embodiment, the training module includes:
[0163] The settings submodule is used to set the label for each data item in the training dataset based on the hard drive failure time.
[0164] The training submodule is used to train the fault prediction model based on the training dataset, and obtain the trained fault prediction model when the loss function meets the preset conditions.
[0165] In one embodiment, the loss function L fl for: Where α is the balance factor, γ is the modulation parameter, y′ is the predicted value, and y is the actual value of the sample.
[0166] In one embodiment, the setting submodule is further configured to compare the hard disk failure time and the data acquisition time; if the interval between the data acquisition time and the hard disk failure time is less than N days, the label is set to 1; if the interval between the data acquisition time and the hard disk failure time is greater than N days and less than M days, and the original value of one of the multiple attribute fields of the SMART data is greater than 0, the label is set to 1; if the interval between the data acquisition time and the hard disk failure time is greater than N days and less than M days, and the original value of multiple attribute fields of the SMART data is equal to 0, the label is set to 0; if the interval between the data acquisition time and the hard disk failure time is greater than M days, the label is set to 0, where 1 represents risk and 0 represents no risk.
[0167] Figure 13 This is a block diagram of a hard disk failure prediction device according to an optional embodiment of this application, such as... Figure 13 As shown, it includes:
[0168] The data acquisition module 132 is used to implement the functions of the first acquisition module 122 and the second acquisition module. It is mainly responsible for acquiring data from the hard disk, acquiring data at fixed time intervals, and mainly includes the following two types of data: SMART data and hard disk performance data acquired within the operating system.
[0169] The hard disk SMART data includes the original value and the current value. The main SMART attribute fields are SMART5, SMART187, SMART188, SMART197, SMART198, etc.
[0170] Operating system-level hard disk performance data includes disk-level performance metrics, such as throughput and average I / O operation latency, and server-level performance metrics, such as CPU activity, paging and page-out activity.
[0171] The feature extraction module 134 is used to implement the functions of the first feature extraction module. It is mainly responsible for extracting features from the collected data, filtering out data columns that are not used by the detection and prediction algorithm, and only retaining data columns that will be used later.
[0172] The data verification module 136 is mainly responsible for verifying whether the data time length and the number of sampling points can meet the minimum data volume requirements for fault detection and prediction: the data collection interval is at least once a day, and the data collection takes place for at least two days.
[0173] The data combination module 138 is mainly responsible for combining SMART data and performance data to form a training dataset.
[0174] The label calculation module 1310 is primarily responsible for calculating the label for each data point in the training dataset based on the hard drive failure time. The specific calculation method compares the hard drive failure time and the data acquisition time. If the acquisition time interval is less than N days, it is labeled as 1, representing risk. If the interval between the acquisition time and the failure time is greater than N days but less than M days, and at least one of the five attributes (SMART5, SMART187, SMART188, SMART197, and SMART198) in the acquired data has a value greater than 0, it is also labeled as 1. If the interval is greater than N days but less than M days, and the original values of the above five attributes are all equal to 0, it is labeled as 0, representing health. If the interval between the acquisition time and the failure time is greater than M days, the data label is also labeled as 0.
[0175] The data cleaning module 1312 implements the functions of the aforementioned cleaning module. Its main responsibility is to clean invalid and noisy data from the dataset and fill in missing data. For faulty hard drives in the training set, it deletes data records with label 0 to reduce noise in the dataset.
[0176] AI training module 1314 is used to implement the functions of the above training module, and it is mainly responsible for machine learning training on the training dataset.
[0177] The training loss calculation module 1316 implements some of the functions of the aforementioned training module. Its main responsibility is to calculate the loss on the samples, guiding the model to train in the direction of minimizing loss. Specifically, the Focal Loss function is used as the loss function.
[0178] The inference module 1318 is used to input unknown hard disk data into the trained model for inference.
[0179] The expert system module 1320 inputs data into the expert system to reassess the status of hard drives that are predicted to be at risk by the AI model.
[0180] The risk disk processing module 1322 sets the risk level to high for hard drives that are identified as abnormal by both the AI model and the expert system, and recommends that the user replace the hard drive. For hard drives that are not identified as abnormal by the expert system, the risk level is set to medium.
[0181] The hard disk data backup module 1324 is used to implement the functions of the backup module mentioned above. For hard disks with a medium risk level, the hard disk data is compressed and the hard disk data is automatically backed up to other hard disks on a regular schedule. The other hard disks can be the hard disks of this server or backup disks dedicated to the data center. The later backup overwrites the previous backup. The automatic backup time is preferably during the early morning when there is less business traffic.
[0182] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above method embodiments when run.
[0183] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.
[0184] Embodiments of this application also provide an electronic device including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.
[0185] In one exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.
[0186] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.
[0187] Obviously, those skilled in the art should understand that the modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.
[0188] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this application should be included within the protection scope of this application.
Claims
1. A hard disk failure prediction method, characterized in that, The method includes: Collect hard disk data to be processed, wherein the hard disk data includes SMART data and performance data; The SMART data and the performance data are concatenated to obtain the target hard drive data; The target hard disk data is input into a pre-trained fault prediction model to obtain the first fault prediction result output by the fault prediction model. If the first fault prediction result indicates that there is a fault risk, the second fault prediction result is obtained by further determining whether there is a fault risk in the hard disk data to be processed based on the SMART data and performance data of the hard disk data to be processed. The hard drive failure risk level is determined based on the second failure prediction result; Before acquiring the hard disk data to be processed, the method further includes: Collect a preset amount of hard disk data, wherein the hard disk data includes SMART data and performance data; The preset number of SMART data and the performance data are concatenated to form a training dataset; The fault prediction model is trained based on the training dataset to obtain the trained fault prediction model, including: setting the label of each data in the training dataset according to the hard disk failure time; training the fault prediction model based on the training dataset, and obtaining the trained fault prediction model when the loss function meets preset conditions. The labeling of each data point in the training dataset based on the hard drive failure time includes: comparing the hard drive failure time and the data acquisition time; setting the label to 1 if the interval between the data acquisition time and the data acquisition time is less than N days; setting the label to 1 if the interval between the data acquisition time and the hard drive failure time is greater than or equal to N days and less than or equal to M days, and the original value of one of the multiple attribute fields of the SMART data is greater than 0; setting the label to 0 if the interval between the data acquisition time and the hard drive failure time is greater than or equal to N days and less than or equal to M days, and the original value of multiple attribute fields of the SMART data is equal to 0; and setting the label to 0 if the interval between the data acquisition time and the hard drive failure time is greater than M days. Here, 1 represents risk, and 0 represents no risk.
2. The method according to claim 1, characterized in that, Before concatenating the SMART data and the performance data to obtain the target hard disk data, the method further includes: Feature extraction is performed on the hard disk data to be processed to obtain the feature data to be processed; The feature data to be processed is filtered to obtain the target feature data to be processed. The target feature data to be processed is determined to meet the preset requirements.
3. The method according to claim 1, characterized in that, The hard drive failure risk level is determined based on the second failure prediction result, including: If the second fault prediction result indicates that there is a fault risk, the hard drive fault risk level is determined to be the first level. If the second fault prediction result is that there is no fault risk, the hard drive fault risk level is determined to be the second level, wherein the risk level of the first level is higher than that of the second level.
4. The method according to claim 3, characterized in that, After determining whether the hard disk data to be processed has a risk of failure based on the SMART data and performance data of the hard disk data to be processed, and obtaining a second failure prediction result, the method further includes: If the hard drive failure risk level is the first level, a prompt will be made to replace the hard drive corresponding to the hard drive data to be processed. If the hard drive failure risk level is the second level, the hard drive data to be processed is backed up.
5. The method according to claim 1, characterized in that, Based on the SMART data and performance data of the hard disk data to be processed, a second fault prediction result is obtained by determining whether the hard disk data to be processed has a risk of failure, including: Based on the SMART data of the hard disk data to be processed, determine whether the second fault prediction result indicates a hard disk risk. If the judgment result is yes, the second fault prediction result is determined to be that there is a hard drive risk; If the judgment result is negative, determine whether the second fault prediction result indicates a hard drive risk based on the performance data; if the judgment result is positive, determine that the second fault prediction result indicates a hard drive risk; if the judgment result is negative, determine that the second fault prediction result indicates no hard drive risk.
6. The method according to claim 5, characterized in that, Determining whether the second fault prediction result indicates a hard drive risk based on the SMART data of the hard drive data to be processed includes: The SMART data includes at least SMART5, SMART187, SMART188, SMART197 and SMART198. It is determined whether the original value of SMART5 is greater than a first preset value. If the determination result is yes, the second fault prediction result is determined to be that there is a hard drive risk. If the judgment result is negative, determine whether the original value of SMART187 is greater than the second preset value. If the judgment result is positive, determine that the second fault prediction result indicates that there is a hard drive risk. If the judgment result is negative, determine whether the original value of SMART188 is greater than the third preset value. If the judgment result is positive, determine that the second fault prediction result indicates that there is a hard drive risk. If the judgment result is negative, determine whether the original value of SMART197 or SMART198 is greater than the fourth preset value. If the judgment result is positive, determine that the second fault prediction result indicates that there is a hard drive risk. If the judgment result is negative, determining whether the second fault prediction result indicates the presence of hard disk risk based on the performance data includes: determining whether the average number of files successfully read per second by the hard disk in the performance data is greater than a fifth preset value; if the judgment result is positive, determining that the second fault prediction result indicates the presence of hard disk risk; if the judgment result is negative, determining that the second fault prediction result indicates the absence of hard disk risk.
7. The method according to claim 1, characterized in that, Before training the fault prediction model based on the training dataset to obtain the trained fault prediction model, the method further includes: The invalid and noisy data in the training dataset are cleaned, and the missing data is filled in.
8. The method according to claim 1, characterized in that, Before concatenating a preset number of SMART data points and the performance data to form a training dataset, the method further includes: Feature extraction is performed on the preset number of hard disk data to obtain feature data corresponding to the preset number of hard disk data; Filter the feature data corresponding to the preset number of hard disk data to obtain the target feature data corresponding to the preset number of hard disk data; The target feature data corresponding to the preset number of hard disk data is determined to meet the preset requirements.
9. The method according to claim 1, characterized in that, The loss function for: ； Where α is the balance factor and γ is the modulation parameter. y is the predicted value, and y is the actual value of the sample.
10. A hard disk failure prediction device, characterized in that, The device includes: The first acquisition module is used to acquire hard disk data to be processed, wherein the hard disk data includes SMART data and performance data; The first splicing module is used to splice the SMART data and the performance data to obtain the target hard disk data; The input module is used to input the target hard disk data into a pre-trained fault prediction model to obtain the first fault prediction result output by the fault prediction model. The secondary prediction module is used to determine whether the hard disk data to be processed has a risk of failure based on the SMART data and performance data of the hard disk data to be processed when the first fault prediction result indicates that there is a risk of failure, and to obtain a second fault prediction result. The first determining module is used to determine the hard disk failure risk level based on the second failure prediction result; The device further includes: The second acquisition module is used to acquire a preset amount of hard disk data, wherein the hard disk data includes SMART data and performance data; The second splicing module is used to splice a preset number of SMART data and the performance data together to form a training dataset. The training module is used to train the fault prediction model based on the training dataset to obtain the trained fault prediction model. The training module includes: The settings submodule is used to set the label for each data item in the training dataset based on the hard drive failure time. The training submodule is used to train the fault prediction model based on the training dataset, and obtain the trained fault prediction model when the loss function meets the preset conditions. The setting submodule is further used to compare the hard drive failure time and the data acquisition time. If the interval between the data acquisition time and the data acquisition time is less than N days, the label is set to 1. If the interval between the data acquisition time and the hard drive failure time is greater than or equal to N days and less than or equal to M days, and the original value of one of the multiple attribute fields of the SMART data is greater than 0, the label is set to 1. If the interval between the data acquisition time and the hard drive failure time is greater than or equal to N days and less than or equal to M days, and the original values of multiple attribute fields of the SMART data are all equal to 0, the label is set to 0. If the interval between the data acquisition time and the hard drive failure time is greater than M days, the label is set to 0. Here, 1 represents risk, and 0 represents no risk.
11. A computer-readable storage medium storing a computer program, wherein, The computer program is configured to execute the method described in any one of claims 1 to 9 when it is run.
12. An electronic device comprising a memory and a processor, the memory storing a computer program, the processor being configured to run the computer program to perform the method of any one of claims 1 to 9.