A disk fault detection method, device, computer equipment and storage medium

Abstract

The embodiment of the present invention discloses a disk fault detection method, device, computer equipment and storage medium. The method includes: obtaining the original SMART data of the disk to be detected, and standardizing the original SMART data to obtain standard SMART data; determining the change rate of each SMART attribute value according to the standard SMART data; inputting the standard SMART data and the change rate into the trained machine learning model to obtain a prediction result of whether the disk to be detected is faulty. The disk fault detection method provided by the embodiment of the present invention obtains the original SMART data of any type of disk and standardizes it uniformly, thereby removing the influence of different standardization processing methods of various manufacturers, so that the original SMART data of the disk can be directly used very conveniently, thereby improving the versatility of the fault detection method, and at the same time, by adding the attribute value change rate feature as the input of the prediction model, the speed of the disk health state deterioration is taken into account, thereby improving the accuracy of the disk fault detection result.

Description

Technical Field

[0001] The embodiments of the present invention relate to the technical field of data storage, and in particular to a disk fault detection method, device, computer equipment and storage medium. Background Art

[0002] With the emergence and rapid development of emerging information technologies such as industrial Internet of Things, cloud storage, cloud computing, and big data, the total amount of global data has increased exponentially, and nearly 90% of the world's data is stored on disks in data centers. Due to the disk's own structure and data storage mechanism, once a disk fails, the data stored on the disk may be permanently lost, causing serious losses to companies and individuals. Although the data redundancy backup mechanism can prevent data loss when a disk fails, it increases the cost of data storage, and disk failure prediction has become the most mainstream method at present.

[0003] Modern data centers widely use disk array technology to uniformly manage disks in storage systems. However, since the manufacturers of each disk in the disk array may be different, the types of SMART information parameters provided are different, and the naming rules are also different, making it difficult to obtain unified standard data for the prediction model. In addition, different manufacturers have different standardized processing methods for the original data values, so the original values ​​cannot be used directly. At the same time, existing fault prediction methods usually directly use the SMART data of the disk for prediction, and the prediction results are usually not accurate enough. Summary of the invention

[0004] The embodiments of the present invention provide a disk fault detection method, device, computer equipment and storage medium to achieve more universal disk fault detection and improve the accuracy of the detection result.

[0005] In a first aspect, an embodiment of the present invention provides a method for detecting a disk failure, the method comprising:

[0006] Acquire original SMART data of the disk to be detected, and standardize the original SMART data to obtain standard SMART data;

[0007] Determining a rate of change of each SMART attribute value based on the standard SMART data;

[0008] The standard SMART data and the change rate are input into the trained machine learning model to obtain a prediction result of whether the disk to be detected is faulty.

[0009] Optionally, obtaining original SMART data of the disk to be detected includes:

[0010] Use the collection tools smartmontools and MegaCli to read the SMART information of the disk to be detected;

[0011] The SMART information is preprocessed to obtain the original SMART data, wherein the preprocessing includes at least one of column-to-row conversion, deduplication, and null value removal.

[0012] Optionally, before inputting the standard SMART data and the change rate into the trained machine learning model to obtain a prediction result of whether the disk to be detected is faulty, the method further includes:

[0013] Acquire a training sample set, the training sample set comprising a plurality of training samples, each of the training samples comprising sample standard SMART data of a sample disk, a sample change rate of each SMART attribute value in the sample standard SMART data, and a bad disk attribute value of the sample disk;

[0014] The machine learning model is trained according to the training sample set.

[0015] Optionally, obtaining a training sample set includes:

[0016] Obtaining sample original SMART data of the sample disk;

[0017] Determine the correlation between each SMART attribute value in the sample original SMART data and the bad disk attribute value;

[0018] Screening the SMART attribute values ​​in the sample original SMART data according to the correlation;

[0019] Standardizing the original SMART data of the sample after screening to obtain the standard SMART data of the sample;

[0020] Accordingly, before the original SMART data is standardized to obtain standard SMART data, the method further includes:

[0021] The original SMART data is screened according to the sample standard SMART data.

[0022] Optionally, the training samples include positive samples;

[0023] Accordingly, the step of obtaining a training sample set includes:

[0024] Acquire sample original SMART data of the sample disk, wherein the sample disk includes a bad sample disk whose bad disk attribute value is a first preset value;

[0025] Selecting the sample original SMART data of the bad sample disk within a plurality of preset time windows before the failure moment as the positive samples respectively;

[0026] Accordingly, the training of the machine learning model according to the training sample set includes:

[0027] Pre-training the machine learning model using each of the positive samples respectively to test the accuracy corresponding to using different positive samples;

[0028] A target positive sample for training is determined according to the accuracy rate, and the target positive sample is used to train the machine learning model.

[0029] Optionally, the training samples include negative samples;

[0030] Accordingly, the step of obtaining a training sample set includes:

[0031] Acquire sample original SMART data of the sample disk, wherein the sample disk includes a good sample disk whose bad disk attribute value is a second preset value;

[0032] Randomly selecting the sample original SMART data of the good sample disk at multiple preset time points as the negative sample;

[0033] Accordingly, the training of the machine learning model according to the training sample set includes:

[0034] The machine learning model is trained using the negative samples.

[0035] Optionally, after inputting the standard SMART data and the change rate into the trained machine learning model to obtain a prediction result of whether the disk to be detected is faulty, the method further includes:

[0036] If the prediction result is a fault, an alarm is issued;

[0037] Acquire data information of the disk to be detected, and generate a fault warning report based on the data information and send it to the user;

[0038] The data information is backed up or transferred according to preset parameters.

[0039] In a second aspect, an embodiment of the present invention further provides a disk fault detection device, the device comprising:

[0040] A data acquisition module is used to acquire original SMART data of the disk to be detected, and standardize the original SMART data to obtain standard SMART data;

[0041] an attribute value change rate determination module, used to determine the change rate of each SMART attribute value according to the standard SMART data;

[0042] A fault prediction module is used to input the standard SMART data and the change rate into a trained machine learning model to obtain a prediction result of whether the disk to be detected is faulty.

[0043] In a third aspect, an embodiment of the present invention further provides a computer device, the computer device comprising:

[0044] one or more processors;

[0045] A memory for storing one or more programs;

[0046] When the one or more programs are executed by the one or more processors, the one or more processors implement the disk failure detection method provided by any embodiment of the present invention.

[0047] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the disk fault detection method provided by any embodiment of the present invention.

[0048] The embodiment of the present invention provides a disk fault detection method, which first obtains the original SMART data of the disk to be detected, and standardizes the original SMART data to obtain standard SMART data, then determines the change rate of each SMART attribute value according to the standard SMART data, and then uses the standard SMART data and the change rate as model input, and inputs them into the trained machine learning model to predict whether the disk to be detected is faulty. The disk fault detection method provided by the embodiment of the present invention obtains the original SMART data of any type of disk and standardizes it uniformly, thereby removing the influence of different standardization processing methods of various manufacturers, so that the original SMART data of the disk can be directly used very conveniently, thereby improving the versatility of the fault detection method, and at the same time, by adding the attribute value change rate feature as the input of the prediction model, the speed of the disk health state deterioration is taken into account, thereby improving the accuracy of the disk fault detection result. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A flowchart of a disk fault detection method provided in Embodiment 1 of the present invention;

[0050] Figure 2 A schematic diagram of the structure of a disk fault detection device provided in Embodiment 2 of the present invention;

[0051] Figure 3 This is a schematic diagram of the structure of a computer device provided in Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0052] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0053] It should be mentioned before discussing the exemplary embodiments in more detail that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe the steps as sequential processes, many of the steps therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of the steps can be rearranged. The process can be terminated when its operation is completed, but can also have additional steps not included in the accompanying drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0054] Embodiment 1

[0055] Figure 1This is a flow chart of a disk fault detection method provided in Embodiment 1 of the present invention. This embodiment is applicable to the case of performing fault detection on various types of disks in a disk array. This method can be executed by a disk fault detection device provided in an embodiment of the present invention. The device can be implemented in hardware and / or software and can generally be integrated into a computer device. Figure 1 As shown, the specific steps include:

[0056] S11, obtaining original SMART data of the disk to be detected, and standardizing the original SMART data to obtain standard SMART data.

[0057] The disk to be detected can be any type of disk in the disk array, that is, it can support disks produced by any manufacturer. Specifically, the original SMART (Self-Monitoring Analysis and Reporting Technology) data of the disk to be detected can be read by the array controller. Specifically, a read command of the original SMART data can be first sent to the array controller, and then the array controller converts the read command into a target read command of the original SMART data of the disk of the corresponding type (such as SCSI or ATA, etc.), so as to read the original SMART data according to the target read command, and return the read data through the array controller.

[0058] Optionally, the acquisition of the original SMART data of the disk to be detected includes: using the acquisition tools smartmontools and MegaCli to read the SMART information of the disk to be detected; preprocessing the SMART information to obtain the original SMART data, wherein the preprocessing includes at least one of column-to-row conversion, deduplication, and removal of null values. Specifically, due to the different support levels of array card manufacturers for each disk, it is difficult to directly obtain the original SMART information from the disk. In this embodiment, the disk monitoring tools provided by each manufacturer can be discarded, and the SMART information of each manufacturer's disk can be read through the general software smartmontools and MegaCli to make the process of reading disk information more universal. Among them, the MegaCli tool can be used to obtain the disk Symbol and Device Id, and then the SMART information can be obtained by looping through the Symbol and Device Id and using the smartmontools tool. The acquired SMART information may contain some null values, and may cause duplicate data in the reading process, but the key indicators are relatively complete. Therefore, the acquired SMART information can be pre-processed by removing duplicates and null values ​​to achieve the purpose of collecting original SMART data. At the same time, pre-processing such as column-to-row conversion can also be performed to make the data more convenient for subsequent use. Among them, the problem of data duplication can be removed through the disk's world wide name (WWN) field.

[0059] In order to remove the impact of different standardization processing methods of various manufacturers, the original various standard values ​​can be abandoned. After obtaining the original SMART data of the disk to be tested, the original SMART data can be standardized through a unified preset standardization method to obtain standard SMART data for subsequent use. Specifically, a relatively simple linear conversion function can be used to standardize the original SMART data, that is, normalization, where the linear conversion function can be:

[0060]

[0061] Among them, x normal represents a certain attribute value in the standard SMART data, x represents the corresponding attribute value in the original SMART data, and x max Indicates the maximum value of the attribute in the original SMART data, x minIndicates the minimum value of the attribute in the original SMART data. Through this linear transformation function, the value range of all attribute values ​​can be mapped to [-1,1].

[0062] S12. Determine the change rate of each SMART attribute value according to the standard SMART data.

[0063] In order to more completely describe the disk feature model, the change rate feature can be added to characterize the change trend of the SMART attribute value over time, so as to reflect the speed of the disk health deterioration, so as to make the subsequent prediction results more accurate. Specifically, after obtaining the standard SMART data, the corresponding change rate can be calculated for each SMART attribute value based on the past value records.

[0064] S13. Input the standard SMART data and the change rate into the trained machine learning model to obtain a prediction result of whether the disk to be detected is faulty.

[0065] Specifically, after obtaining the standard SMART data and the rate of change of each SMART attribute value, the standard SMART data and the rate of change can be used as input features and input into the trained machine learning model, so that the model can output the prediction result of whether the disk to be detected is faulty. The machine learning model can be a support vector machine (SVM), and specifically, when the output is +1, the disk to be detected can be determined to be a bad disk, and when the output is -1, the disk to be detected can be determined to be a good disk.

[0066] Optionally, before inputting the standard SMART data and the change rate into the trained machine learning model to obtain a prediction result of whether the disk to be detected is faulty, it also includes: obtaining a training sample set, the training sample set including multiple training samples, each of the training samples including sample standard SMART data of the sample disk, the sample change rate of each SMART attribute value in the sample standard SMART data, and the bad disk attribute value of the sample disk; training the machine learning model according to the training sample set.

[0067] Specifically, before using the machine learning model for prediction, a training sample set can be first obtained to train it. The training sample set can include a large number of training samples, and each training sample can include the sample standard SMART data of the sample disk, the sample change rate of each SMART attribute value in the sample standard SMART data, and the bad disk attribute value of the sample disk, that is, the classification label of the sample disk, which can specifically include +1 and -1. The training sample set can be formed by collecting relevant data of a large number of different sample disks, wherein the acquisition of the sample standard SMART data and the sample change rate can refer to the acquisition process of the standard SMART data and the change rate of the disk to be tested. After obtaining the training sample set, the training sample set can be used to train the machine learning model.

[0068] Further optionally, the obtaining of the training sample set includes: obtaining sample original SMART data of the sample disk; determining the correlation between each SMART attribute value in the sample original SMART data and the bad disk attribute value; filtering the SMART attribute values ​​in the sample original SMART data according to the correlation; standardizing the filtered sample original SMART data to obtain the sample standard SMART data; accordingly, before standardizing the original SMART data to obtain standard SMART data, it also includes: filtering the original SMART data according to the sample standard SMART data.

[0069] Specifically, the acquisition of the sample original SMART data can refer to the acquisition process of the original SMART data of the disk to be detected. After obtaining the sample original SMART data, the correlation between each SMART attribute value and the bad disk attribute value (such as -1 or +1) can be determined to filter out irrelevant variables in the sample original SMART data relative to fault detection based on the correlation. Specifically, the correlation coefficient method can be used to determine the correlation. The correlation coefficient is a quantity that studies the degree of linear correlation between variables, which is used to reflect the closeness of the correlation between variables. Specifically, the formula can be used:

[0070]

[0071] Wherein, r represents the correlation coefficient, i.e., the correlation, and x and y represent two object variables for studying the correlation, which may be the SMART attribute value and the bad disk attribute value in this embodiment. and Represents the average value of two object variables, that is, it can be the average value of SMART attribute values ​​and the average value of bad disk attribute values. Among them, 0<|r|<1, and the closer |r| is to 1, the higher the correlation is. Specifically, the SMART attribute values ​​can be screened by a preset threshold, that is, only the SMART attribute values ​​whose correlation exceeds the preset threshold are retained. After calculating the correlation, in this embodiment, the SMART attribute values ​​screened may include: airflow temperature Cel, command timeout, current pending sector, end-to-end error, hardware ECC recovered, high fly writes, load cycle count, power on hours, power-off retract count, raw read error rate, reallocated sector count, reported uncorrected error, seek error rate, spin up time and temperature celsius. Therefore, only the data of the above fifteen SMART attribute values ​​in the sample original SMART data can be retained to reduce the dimension of the sample original SMART data, thereby reducing the subsequent calculation amount and improving the accuracy of the prediction result. After the screening is completed, the sample standard SMART data can be obtained by standardizing the screened sample original SMART data, wherein the standardization can refer to the above-mentioned process of standardizing the original SMART data. Accordingly, in the process of using the trained machine learning model for fault prediction, the original SMART data can be first screened according to the type of SMART attribute values ​​included in the sample standard SMART data to use the same input data as the training process.

[0072] Further optionally, the training samples include positive samples; accordingly, obtaining the training sample set includes: obtaining sample original SMART data of the sample disk, the sample disk including a bad sample disk whose bad disk attribute value is a first preset value; respectively selecting the sample original SMART data of the bad sample disk within multiple preset time windows before the failure moment as the positive samples; accordingly, training the machine learning model according to the training sample set includes: pre-training the machine learning model using each of the positive samples respectively to test the accuracy corresponding to using different positive samples; determining the target positive sample for training according to the accuracy, and using the target positive sample to train the machine learning model.

[0073] Specifically, the training of the machine learning model requires the use of positive samples. In this embodiment, the data of the bad sample disk can be used as a positive sample, and the classification label can be set to +1. The acquisition of the sample original SMART data of the sample disk can refer to the acquisition process of the original SMART data of the disk to be detected. For the positive sample, the sample disk can be a bad sample disk, and its bad disk attribute value can be +1, which is the same as the corresponding classification label, to indicate that it is a bad disk. In the existing prediction methods, the model is usually trained directly using all the data of the faulty disk that can no longer be used. The practical application ability of the model obtained by such training is insufficient. Therefore, in this embodiment, the positive sample can be selected to participate in the training based on the time window strategy, in order to find out how long before the fault moment the original SMART data can be used as a positive sample to participate in the training, so as to be more in line with the actual environment. Specifically, the optimal time window can be determined by multi-sample multi-training. Then, after obtaining a certain amount of sample original SMART data of bad sample disks, the sample original SMART data in multiple preset time windows before the failure moment (such as within the first 1 hour, the first 12 hours, the first 1 day, and the first 2 days, etc.) can be selected as positive samples (of course, including the corresponding sample change rate and bad disk attribute values, etc.) to train the model, that is, the pre-training process, to test the accuracy of the model trained with different positive samples, that is, the performance of predicting disk failures. Then, by comparing the various accuracies, a more appropriate optimal time window can be selected according to actual needs, so as to determine the target positive sample required for actual training according to the optimal time window, and then the model can be trained using the target positive sample to obtain a trained model, wherein the target positive sample can be the sample original SMART data of a larger number of bad sample disks in the optimal time window compared to the pre-training process.

[0074] Further optionally, the training samples include negative samples; accordingly, obtaining the training sample set includes: obtaining sample original SMART data of the sample disk, the sample disk including a good sample disk whose bad disk attribute value is a second preset value; randomly selecting the sample original SMART data of the good sample disk at multiple preset moments as the negative samples; accordingly, training the machine learning model according to the training sample set includes: training the machine learning model using the negative samples.

[0075] Specifically, the training of the machine learning model also requires the use of negative samples. In this embodiment, the data of the good sample disk can be used as a negative sample, and the classification label can be set to -1. The acquisition of the sample original SMART data of the sample disk can refer to the acquisition process of the original SMART data of the disk to be detected. For the negative sample, the sample disk can be a good sample disk, and its bad disk attribute value can be -1, which is the same as the corresponding classification label, to indicate that it is a good disk. For the training samples of the good disk in the training sample set, not all training samples are used to train the model, but for each good sample disk, the sample original SMART data at multiple preset times (such as 4) can be randomly selected as negative samples (of course, the corresponding sample change rate and bad disk attribute value, etc.) to train the model to obtain a trained model, thereby reducing the input sample data and improving the training speed of the model. At the same time, the value deviation of the SMART data of the disk at a certain moment can also be eliminated to provide enough feature information to characterize the health status of the disk.

[0076] Before training the machine learning model, you can use Python to build the model algorithm. For example, for the support vector machine, you can use the SVM module in the scikit-learn library to implement the improved SVM model and set different classification labels for the samples of bad disks and good disks, namely +1 and -1. In the process of training the machine learning model, for the positive and negative samples in the training sample set that are misclassified by the decision hyperplane, you can adjust their penalty function parameters ω +1 and ω -1 To obtain different fault detection rates FDR and false alarm rates FAR, FDR and FAR can together constitute the above accuracy. As shown in the above example, when using positive and negative samples of four time windows to train the model, you can adjust ω +1 and ω -1 To make the prediction accuracy of the model compromise between FDR and FAR, so as to obtain multiple different results. Generally speaking, ω +1The larger the value of is, the greater the penalty for misclassified positive samples is, the fewer misclassified positive samples are, and the higher the FDR and FAR are. On the contrary, ω -1 The larger the value of , the greater the penalty for misclassified negative samples, the fewer misclassified negative samples, and the lower the FDR and FAR. At the same time, different time windows will also affect the performance of the model. As the time window of positive samples becomes longer, the FDR and FAR will become higher. The reason is that a longer time window brings more positive samples close to negative samples, which makes the model biased towards the detection of positive samples and unfavorable to the detection of negative samples, resulting in higher FDR and FAR. In practical applications, in order to reduce the processing overhead caused by a large number of false faults, it is more inclined to choose a model with a lower FAR. According to the test results, the model can be trained with positive samples within a time window of 1 hour or 12 hours. When a 1-hour time window is used, the penalty function pair (1.5, 6) can be set to train the model. At this time, the lowest FAR of 0.03% can be obtained, and the corresponding FDR is 69.9%. When a 12-hour time window is used, the penalty function pair (3, 1) can be set to train the model. At this time, the FAR is 0.45% and the FDR is 82%. According to actual needs, the selection of the time window and the setting of the penalty function may not be limited to the above examples, thereby improving the accuracy and practicality of the model.

[0077] On the basis of the above technical solution, optionally, after inputting the standard SMART data and the change rate into the trained machine learning model to obtain a prediction result of whether the disk to be detected is faulty, it also includes: if the prediction result is a fault, an alarm is issued; data information of the disk to be detected is obtained, and a fault warning report is generated based on the data information and sent to the user; and the data information is backed up or transferred according to preset parameters.

[0078] Specifically, most existing disk monitoring systems only use hardware alarms, which require maintenance personnel to check 24 hours a day, cannot be notified in a timely and effective manner, and the alarm information is relatively simple. In this embodiment, if the machine learning model predicts that the disk to be detected is a faulty disk, an alarm can be issued first to indicate that a fault has occurred. After the alarm, the data information of the disk to be detected can be automatically obtained, and a detailed fault warning report and disposal plan can be generated through analysis and integration, and then sent to users (such as operation and maintenance engineers, etc.) through multiple channels (such as Web / SMS / Email) to achieve the effect of making the disk failure transparent to the user. The data information of the disk to be detected can also be backed up or transferred according to preset parameters to maximize data security, wherein the preset parameters may include the required execution actions corresponding to the disk type (such as system disk or data disk, etc.), alarm level (different colors can be used to represent the level to perform different actions, such as green, yellow and red from light to heavy), data importance (such as very important, important, general and impact, etc.), whether there is hot standby and the specific disposal strategy adopted, etc., so as to automatically execute different backup and transfer plans for different situations. Accordingly, before automatically executing various backup and transfer schemes, the disk type, alarm level, data importance, whether there is hot spare, and the adopted disposal strategy of the disk to be detected can be determined first according to the acquired data information. In particular, if the above scheme for determining the best time window is adopted, the time within the best time window after the alarm is the golden time period for handling the fault, and the work of data backup and transfer can be done in advance, so as to better ensure the security of the data. Exemplarily, for the RAID0 strategy, for data information with data importance of very important, disk type of system disk, alarm level of red, and hot spare, the execution action may include compressing and backing up the system image to the hot spare; for data information with data importance of very important, disk type of system disk, alarm level of red, but no hot spare, the execution action may include compressing and backing up the system image to the data disk; for data information with data importance of important, disk type of data disk, alarm level of red, and hot spare, the execution action may include importing the data information into the backup disk; other data may not be processed. For RAID1 strategy or RAID5 strategy, for data information with data importance of very important, disk type of system disk, alarm level of red, and hot spare, the execution action may include failover and RAID reconstruction; for data information with data importance of important, disk type of data disk, alarm level of green, and hot spare, the execution action may include failover and RAID reconstruction; other data may not be processed.

[0079] The disk fault detection method provided in this embodiment is used for users with a small number of clusters or racks. Due to the small number of professionals, disk faults are often not discovered in time, which often causes data loss and affects business when they are discovered. By automatically obtaining data information of the faulty disk and generating a fault warning report to send to the user, the user can predict the fault in advance and handle it in time. For users of distributed clusters or racks, there is a greater need to solve the dilemma caused by the large and scattered clusters, different hardware standards of manufacturers, and uneven levels of operation and maintenance personnel, which cannot form unified and effective management and cannot detect and record faults in time. The method provided in this embodiment can give full play to the versatility of disk information collection and warning, and can combine experts to make resource allocation according to on-site conditions, thereby building an enterprise-level system for unified information collection, warning release, and operation and maintenance resource allocation.

[0080] The technical solution provided by the embodiment of the present invention first obtains the original SMART data of the disk to be detected, and standardizes the original SMART data to obtain standard SMART data, then determines the change rate of each SMART attribute value according to the standard SMART data, and then uses the standard SMART data and the change rate as model input, and inputs them into the trained machine learning model to predict whether the disk to be detected is faulty. By obtaining the original SMART data of any type of disk and standardizing them uniformly, the influence of different standardization processing methods of various manufacturers is removed, so that the original SMART data of the disk can be used directly and conveniently, which improves the versatility of the fault detection method. At the same time, by adding the attribute value change rate feature as the input of the prediction model, the speed at which the health status of the disk deteriorates is taken into account, thereby improving the accuracy of the disk fault detection results.

[0081] Embodiment 2

[0082] Figure 2 This is a schematic diagram of the structure of a disk fault detection device provided in the second embodiment of the present invention. The device can be implemented by hardware and / or software, and can generally be integrated into a computer device to execute the disk fault detection method provided in any embodiment of the present invention. Figure 2 As shown, the device comprises:

[0083] The data acquisition module 21 is used to acquire the original SMART data of the disk to be detected, and standardize the original SMART data to obtain standard SMART data;

[0084] an attribute value change rate determination module 22, for determining a change rate of each SMART attribute value according to the standard SMART data;

[0085] The fault prediction module 23 is used to input the standard SMART data and the change rate into the trained machine learning model to obtain a prediction result of whether the disk to be detected is faulty.

[0086] The technical solution provided by the embodiment of the present invention first obtains the original SMART data of the disk to be detected, and standardizes the original SMART data to obtain standard SMART data, then determines the change rate of each SMART attribute value according to the standard SMART data, and then uses the standard SMART data and the change rate as model input, and inputs them into the trained machine learning model to predict whether the disk to be detected is faulty. By obtaining the original SMART data of any type of disk and standardizing them uniformly, the influence of different standardization processing methods of various manufacturers is removed, so that the original SMART data of the disk can be used directly and conveniently, which improves the versatility of the fault detection method. At the same time, by adding the attribute value change rate feature as the input of the prediction model, the speed at which the health status of the disk deteriorates is taken into account, thereby improving the accuracy of the disk fault detection results.

[0087] On the basis of the above technical solution, optionally, the data acquisition module 21 includes:

[0088] An information acquisition unit, used for reading the SMART information of the disk to be detected by using acquisition tools smartmontools and MegaCli;

[0089] A preprocessing unit is used to preprocess the SMART information to obtain the original SMART data, wherein the preprocessing includes at least one of column-to-row conversion, deduplication and null value removal.

[0090] On the basis of the above technical solution, optionally, the disk fault detection device further includes:

[0091] A training sample set acquisition module, used for acquiring a training sample set before inputting the standard SMART data and the change rate into the trained machine learning model to obtain a prediction result of whether the disk to be detected is faulty, wherein the training sample set includes a plurality of training samples, each of which includes sample standard SMART data of a sample disk, a sample change rate of each SMART attribute value in the sample standard SMART data, and a bad disk attribute value of the sample disk;

[0092] A model training module is used to train the machine learning model according to the training sample set.

[0093] On the basis of the above technical solution, optionally, the training sample set acquisition module includes:

[0094] A sample original data acquisition unit, used for acquiring sample original SMART data of the sample disk;

[0095] A correlation determination unit, used to determine the correlation between each SMART attribute value in the sample original SMART data and the bad disk attribute value;

[0096] An attribute value screening unit, configured to screen the SMART attribute values ​​in the sample original SMART data according to the correlation;

[0097] A standardization unit, used for standardizing the original SMART data of the sample after screening to obtain the standard SMART data of the sample;

[0098] Correspondingly, the disk fault detection device further includes:

[0099] The attribute value screening module is used for screening the original SMART data according to the sample standard SMART data before standardizing the original SMART data to obtain standard SMART data.

[0100] Based on the above technical solution, optionally, the training samples include positive samples;

[0101] Correspondingly, the training sample set acquisition module includes:

[0102] A bad sample original data acquisition unit, used for acquiring sample original SMART data of the sample disk, wherein the sample disk includes a bad sample disk whose bad disk attribute value is a first preset value;

[0103] A positive sample selection unit, used to select the sample original SMART data of the bad sample disk within a plurality of preset time windows before the failure moment as the positive sample;

[0104] Correspondingly, the model training module includes:

[0105] A pre-training unit, used to pre-train the machine learning model using each of the positive samples respectively, so as to test the accuracy corresponding to using different positive samples;

[0106] A positive sample training unit is used to determine a target positive sample for training according to the accuracy rate, and use the target positive sample to train the machine learning model.

[0107] Based on the above technical solution, optionally, the training samples include negative samples;

[0108] Correspondingly, the training sample set acquisition module includes:

[0109] A good sample original data acquisition unit, used for acquiring sample original SMART data of the sample disk, wherein the sample disk includes a good sample disk whose bad disk attribute value is a second preset value;

[0110] A negative sample selection unit, used for randomly selecting the sample original SMART data of the good sample disk at multiple preset time points as the negative sample;

[0111] Correspondingly, the model training module includes:

[0112] A negative sample training unit is used to train the machine learning model using the negative samples.

[0113] On the basis of the above technical solution, optionally, the disk fault detection device further includes:

[0114] An alarm module is used for, after inputting the standard SMART data and the change rate into the trained machine learning model to obtain a prediction result of whether the disk to be detected is faulty, to issue an alarm if the prediction result is a fault;

[0115] A fault warning report generating module is used to obtain data information of the disk to be detected, and generate a fault warning report according to the data information and send it to the user;

[0116] The backup transfer module is used to back up or transfer the data information according to preset parameters.

[0117] The disk fault detection device provided in the embodiment of the present invention can execute the disk fault detection method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0118] It is worth noting that in the embodiment of the above-mentioned disk fault detection device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0119] Embodiment 3

[0120] Figure 3 The schematic diagram of the structure of the computer device provided for the third embodiment of the present invention shows a block diagram of an exemplary computer device suitable for implementing the implementation mode of the present invention. Figure 3 The computer device shown is only an example and should not limit the functions and scope of use of the embodiments of the present invention. Figure 3 As shown, the computer device includes a processor 31, a memory 32, an input device 33 and an output device 34; the number of processors 31 in the computer device can be one or more. Figure 3 Taking a processor 31 as an example, the processor 31, memory 32, input device 33 and output device 34 in the computer device can be connected through a bus or other means. Figure 3 The example of connecting through bus is taken in the following.

[0121] The memory 32 is a computer-readable storage medium that can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the disk fault detection method in the embodiment of the present invention (for example, the data acquisition module 21, the attribute value change rate determination module 22 and the fault prediction module 23 in the disk fault detection device). The processor 31 executes various functional applications and data processing of the computer device by running the software programs, instructions and modules stored in the memory 32, that is, realizing the above-mentioned disk fault detection method.

[0122] The memory 32 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 32 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 32 may further include a memory remotely arranged relative to the processor 31, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0123] The input device 33 can be used to obtain the original SMART data of the disk to be detected, and generate key signal input related to the user settings and function control of the computer device, etc. The output device 34 can include a display screen and other devices, which can be used to show the user the prediction result of whether the disk to be detected is faulty, etc.

[0124] Embodiment 4

[0125] Embodiment 4 of the present invention further provides a storage medium including computer executable instructions, wherein the computer executable instructions are used to execute a disk fault detection method when executed by a computer processor, the method comprising:

[0126] Acquire original SMART data of the disk to be detected, and standardize the original SMART data to obtain standard SMART data;

[0127] Determining a rate of change of each SMART attribute value based on the standard SMART data;

[0128] The standard SMART data and the change rate are input into the trained machine learning model to obtain a prediction result of whether the disk to be detected is faulty.

[0129] The storage medium may be any of various types of memory devices or storage devices. The term "storage medium" is intended to include: installation media, such as CD-ROM, floppy disk or tape device; computer system memory or random access memory, such as DRAM, DDR RAM, SRAM, EDO RAM, Rambus RAM, etc.; non-volatile memory, such as flash memory, magnetic media (such as hard disk or optical storage); registers or other similar types of memory elements, etc. The storage medium may also include other types of memory or combinations thereof. In addition, the storage medium may be located in the computer system in which the program is executed, or may be located in a different second computer system, which is connected to the computer system via a network (such as the Internet). The second computer system may provide program instructions to the computer for execution. The term "storage medium" may include two or more storage media that may reside in different locations (e.g., in different computer systems connected via a network). The storage medium may store program instructions (e.g., embodied as a computer program) that may be executed by one or more processors.

[0130] Of course, the computer executable instructions of a storage medium including computer executable instructions provided by an embodiment of the present invention are not limited to the operations of the method described above, and can also execute related operations in the disk fault detection method provided by any embodiment of the present invention.

[0131] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0132] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0133] Through the above description of the implementation methods, the technicians in the relevant field can clearly understand that the present invention can be implemented by means of software and necessary general hardware, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0134] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A disk fault detection method, characterized in that: include: Obtaining original SMART data of a disk to be detected, and standardizing the original SMART data to obtain standard SMART data; the disk to be detected is a disk of any type; Determining a rate of change of each SMART attribute value based on the standard SMART data; Inputting the standard SMART data and the change rate into the trained machine learning model to obtain a prediction result of whether the disk to be detected is faulty; The obtaining of the original SMART data of the disk to be detected includes: Use the collection tools smartmontools and MegaCli to read the SMART information of the disk to be detected; Preprocessing the SMART information to obtain the original SMART data, wherein the preprocessing includes at least one of column-to-row conversion, duplicate removal, and null value removal; The step of normalizing the original SMART data to obtain standard SMART data comprises: The raw SMART data is normalized using a linear transformation function: Among them, x normal represents a certain attribute value in the standard SMART data, x represents the corresponding attribute value in the original SMART data, and x max Indicates the maximum value of the attribute in the original SMART data, x min Indicates the minimum value of the attribute in the original SMART data.

2. The disk fault detection method according to claim 1, characterized in that: Before inputting the standard SMART data and the change rate into the trained machine learning model to obtain a prediction result of whether the disk to be detected is faulty, the method further includes: Acquire a training sample set, the training sample set comprising a plurality of training samples, each of the training samples comprising sample standard SMART data of a sample disk, a sample change rate of each SMART attribute value in the sample standard SMART data, and a bad disk attribute value of the sample disk; The machine learning model is trained according to the training sample set.

3. The disk fault detection method according to claim 2, characterized in that: The step of obtaining a training sample set includes: Obtaining sample original SMART data of the sample disk; Determine the correlation between each SMART attribute value in the sample original SMART data and the bad disk attribute value; Screening the SMART attribute values ​​in the sample original SMART data according to the correlation; Standardizing the original SMART data of the sample after screening to obtain the standard SMART data of the sample; Accordingly, before the original SMART data is standardized to obtain standard SMART data, the method further includes: The original SMART data is screened according to the sample standard SMART data.

4. The disk fault detection method according to claim 2, characterized in that: The training samples include positive samples; Accordingly, the step of obtaining a training sample set includes: Acquire sample original SMART data of the sample disk, wherein the sample disk includes a bad sample disk whose bad disk attribute value is a first preset value; Selecting the sample original SMART data of the bad sample disk within a plurality of preset time windows before the failure moment as the positive samples respectively; Accordingly, the training of the machine learning model according to the training sample set includes: Pre-training the machine learning model using each of the positive samples respectively to test the accuracy corresponding to using different positive samples; A target positive sample for training is determined according to the accuracy rate, and the target positive sample is used to train the machine learning model.

5. The disk fault detection method according to claim 2, characterized in that: The training samples include negative samples; Accordingly, the step of obtaining a training sample set includes: Acquire sample original SMART data of the sample disk, wherein the sample disk includes a good sample disk whose bad disk attribute value is a second preset value; Randomly selecting the sample original SMART data of the good sample disk at multiple preset time points as the negative sample; Accordingly, the training of the machine learning model according to the training sample set includes: The machine learning model is trained using the negative samples.

6. The disk fault detection method according to claim 1, characterized in that: After inputting the standard SMART data and the change rate into the trained machine learning model to obtain a prediction result of whether the disk to be detected is faulty, the method further includes: If the prediction result is a fault, an alarm is issued; Acquire data information of the disk to be detected, and generate a fault warning report based on the data information and send it to the user; The data information is backed up or transferred according to preset parameters.

7. A disk fault detection device, characterized in that: include: A data acquisition module, used for acquiring original SMART data of a disk to be detected, and standardizing the original SMART data to obtain standard SMART data; the disk to be detected is a disk of any type; an attribute value change rate determination module, used to determine the change rate of each SMART attribute value according to the standard SMART data; A fault prediction module, used for inputting the standard SMART data and the change rate into the trained machine learning model to obtain a prediction result of whether the disk to be detected is faulty; The data acquisition module comprises: An information acquisition unit, used for reading the SMART information of the disk to be detected by using acquisition tools smartmontools and MegaCli; A preprocessing unit, configured to preprocess the SMART information to obtain the original SMART data, wherein the preprocessing includes at least one of column-to-row conversion, duplicate removal, and null value removal; The data acquisition module is specifically used for: The raw SMART data is normalized using a linear transformation function: Among them, x normal represents a certain attribute value in the standard SMART data, x represents the corresponding attribute value in the original SMART data, and x max Indicates the maximum value of the attribute in the original SMART data, x min Indicates the minimum value of the attribute in the original SMART data.

8. A computer device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the disk failure detection method as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the disk failure detection method as described in any one of claims 1 to 6 is implemented.

Images