Disk failure prediction and processing method, device, medium and computer program product
By training the disk state classification model and generating the disk failure prediction model, the prediction and pre-processing of disk array failures is realized, and the problems of low disk failure processing efficiency and RAID0-level data reliability in the existing technology are solved, and the operation of disk arrays is improved. reliability.
Patent Information
- Application Number
- CN202411149171.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-21
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-08-21
AI Technical Summary
Existing disk arrays require manual processing by operation and maintenance personnel when disk failure occurs, resulting in low operation and maintenance efficiency and long business stagnation. Especially at the RAID0 level, there is no data redundancy and low data reliability.
Using disk failure prediction processing method, by training the disk state classification model, using weighted disk running data for multiple iterative training, generating a disk failure prediction model, predicting the fault status of the target disk, and setting it to detect the expected fault disk The data is synchronized to the backup disk and switched to input and output tasks for pre-processing.
It improves the reliability of the disk array, reduces the probability of business stagnation and frequent disk replacements caused by prediction errors, solves the problem of low data reliability at RAID0 level, and realizes accurate prediction and pre-processing of disk failures.
Smart Images

Figure CN118673393B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of storage technology, and in particular to a disk failure prediction processing method, device, medium and computer program product. Background Art
[0002] Disk arrays (such as Redundant Arrays of Independent Disks, RAID) are large-capacity disk groups composed of many independent disks. Using disk array technology, data is divided into many segments and stored on each disk. Disk arrays provide a fault-tolerant mechanism. When any disk in the disk array fails, the data on the failed disk can be calculated based on other disks and reconstructed on a new disk, greatly improving storage reliability.
[0003] However, the current fault tolerance mechanism of disk arrays can only provide post-disaster recovery functions, and requires operation and maintenance personnel to discover faults and manually control disk replacement, which has low operation and maintenance efficiency and correspondingly leads to long business downtime. At the same time, the disk array controller controls the disks in the disk array at levels including RAID0, RAID1, RAID5, and RAID6. Among them, RAID0 has no data redundancy and cannot achieve the above post-disaster recovery function, resulting in low data reliability.
[0004] How to improve the reliability of disk array operation is a technical problem that those skilled in the art need to solve. Summary of the invention
[0005] The purpose of the present invention is to provide a disk failure prediction processing method, device, medium and computer program product for improving the reliability of disk array operation.
[0006] In order to solve the above technical problems, the present invention provides a disk failure prediction processing method, comprising:
[0007] The disk status classification model is trained by using disk operation data training samples, and the disk status classification models obtained in multiple rounds of iterative training are combined to obtain a disk failure prediction model; in each round of iterative training, the disk status classification ability of the disk status classification model of the current round is trained by using the weighted disk operation data training samples, and the weight of the disk status classification model finally obtained in the current round is determined according to the disk status classification error of the disk status classification model finally obtained in the current round, and the weight of the disk operation data training samples for entering the next round of iterative training is updated; the disk status includes a normal state and a fault state;
[0008] Determine the disk operation status of the target disk according to the disk operation data of the target disk of the target disk array by using the disk failure prediction model;
[0009] If it is determined that there is an expected fault disk in the target disk, the data of the expected fault disk is synchronized to the backup disk, and after the data synchronization is completed, the data input and output tasks of the expected fault disk are switched to the backup disk.
[0010] On the one hand, the disk status classification model is a support vector machine model;
[0011] The training steps of the disk status classification model in each round of iterative training include:
[0012] Determining a kernel function of the support vector machine model;
[0013] Taking minimizing the disk status classification error as the goal, constructing the objective function of the disk status classification model using the kernel function;
[0014] Utilizing the objective function of the disk status classification model and the weighted disk operation data training samples to solve the optimal solution of the disk status classification model;
[0015] The disk status classification model finally obtained in the current round is constructed using the optimal solution of the disk status classification model.
[0016] On the other hand, determining the kernel function of the support vector machine model includes:
[0017] The disk operation data training samples are used to train the disk status classification models corresponding to multiple kernel functions, and the type of the kernel function of the disk status classification model to be used in the iterative training of the disk failure prediction model is determined based on the disk status classification error of the corresponding disk status classification model.
[0018] On the other hand, determining the kernel function of the support vector machine model includes:
[0019] The type of the kernel function is determined according to the computing power of the training device.
[0020] On the other hand, the method of determining the weight of the disk status classification model finally obtained in the current round according to the disk status classification error of the disk status classification model finally obtained in the current round and updating the weight of the disk operation data training sample to enter the next round of iterative training includes:
[0021] After determining the weight of the disk status classification model finally obtained in the current round according to the disk status classification error of the disk status classification model finally obtained in the current round, the weight update parameter of the current round is calculated according to the weight of the disk status classification model finally obtained in the current round, the disk status classification result of each disk operation data training sample by the disk status classification model finally obtained in the current round, the actual disk status of each disk operation data training sample and the weight of each disk operation data training sample in the current round;
[0022] The weight update parameter is used to update the weight of the disk operation data training sample in the current round, so as to obtain the weight of the disk operation data training sample for entering the next round of iterative training.
[0023] On the other hand, the disk status classification model is trained using disk operation data training samples, and the disk status classification models obtained in multiple rounds of iterative training are combined to obtain a disk failure prediction model, including:
[0024] After multiple rounds of iterative training using the disk operation data training samples, the disk status classification models obtained from each round of iterative training are combined to obtain an intermediate disk failure prediction model;
[0025] Testing the intermediate disk failure prediction model using a disk operation data test sample;
[0026] If the test result meets the preset disk failure prediction requirements, the iterative training is terminated to obtain the disk failure prediction model;
[0027] If the test result does not meet the disk failure prediction requirement, the iterative training continues.
[0028] On the other hand, the test result meets the preset disk failure prediction requirement, including:
[0029] The fault detection rate of the intermediate disk failure prediction model for the disk operation data training samples is not less than the fault detection rate threshold, and the false alarm rate of the intermediate disk failure prediction model for the disk operation data training samples is not greater than the false alarm rate threshold, and the advance prediction time of the faulty disk of the intermediate disk failure prediction model for the disk operation data training samples meets the preset time range.
[0030] On the other hand, the failure detection rate is the ratio of the number of failed disks predicted by the intermediate disk failure prediction model to the number of actual failed disks.
[0031] On the other hand, the false alarm rate is the ratio of the number of disks misjudged as healthy by the intermediate disk failure prediction model to the number of actual healthy disks.
[0032] On the other hand, using the disk failure prediction model to determine the disk operation status of the target disk according to the disk operation data of the target disk of the target disk array includes:
[0033] Reading disk operation data of each target disk of the target disk array;
[0034] Obtaining a failure prediction result of the target disk according to the disk operation data of the target disk using the disk failure prediction model;
[0035] If the failure prediction result of the target disk is an expected failure, the target disk is determined to be the expected failure disk.
[0036] On the other hand, using the disk failure prediction model to determine the disk operation status of the target disk according to the disk operation data of the target disk of the target disk array includes:
[0037] Reading disk operation data of each target disk of the target disk array;
[0038] Obtaining a failure prediction result of the target disk according to the disk operation data of the target disk using the disk failure prediction model;
[0039] If there is a candidate expected failure disk whose failure prediction result is an expected failure, performing bad track detection on the candidate expected failure disk;
[0040] The expected failure disk is determined from the candidate expected failure disks according to the bad sector detection result.
[0041] On the other hand, performing bad track detection on the candidate expected failure disk includes:
[0042] Performing a full disk read operation on the candidate expected failure disk;
[0043] Determining the operation status of the disk storage unit according to the read operation result of each disk storage unit of the candidate expected failure disk;
[0044] The step of determining the expected failure disk from the candidate expected failure disks according to the bad track detection result includes:
[0045] If the number of the disk storage units in the candidate expected failure disk whose operating status is an abnormal operating status exceeds a threshold value of the number of abnormal storage units, the candidate expected failure disk is determined to be the expected failure disk.
[0046] On the other hand, determining the expected failure disk from the candidate expected failure disks according to the bad track detection result includes:
[0047] If it is determined according to the bad sector detection result that the candidate expected failure disk is the expected failure disk, then entering the step of synchronizing the data of the expected failure disk to a backup disk;
[0048] If it is determined according to the bad sector detection result that the candidate expected failure disk is not the expected failure disk, the monitoring frequency of the expected failure disk is increased.
[0049] On the other hand, if it is determined that there is an expected fault disk in the target disk, synchronizing the data of the expected fault disk to the backup disk, and after the data synchronization is completed, switching the data input and output tasks of the expected fault disk to the backup disk, including:
[0050] After determining that the candidate expected failure disk is the expected failure disk according to the bad sector detection result, the expected failure disk is included in a disk replacement queue according to the expected failure level corresponding to the bad sector detection result, so that the expected failure disk with a higher expected failure level is placed at the front of the disk replacement queue;
[0051] According to the order of the disk replacement queue, the data of the expected failure disk in the disk replacement queue is synchronized to the backup disk, and after the data synchronization is completed, the data input and output task of the expected failure disk is switched to the backup disk.
[0052] On the other hand, synchronizing the data of the expected faulty disk to the backup disk, and after the data synchronization is completed, switching the data input and output tasks of the expected faulty disk to the backup disk, including:
[0053] Adding the first backup disk in the backup disk group to the target disk array as a target backup disk;
[0054] In the process of synchronizing the data of the expected failure disk to the target backup disk, the target backup disk and the expected failure disk are grouped into a mirror group to synchronously execute the data input and output tasks received by the expected failure disk;
[0055] After all the data of the expected failure disk is synchronized to the target backup disk, the data input and output tasks of the expected failure disk are suspended, the expected failure disk is removed from the target disk array, and the target backup disk is replaced with the identifier of the expected failure disk in the target disk array.
[0056] On the other hand, the disk array controller is applied to monitor the disk operation status of the target disk array using the disk failure prediction model, including:
[0057] The execution time of monitoring the disk operation status of the target disk array by using the disk failure prediction model is determined according to the operation status of the disk array controller.
[0058] In order to solve the above technical problems, the present invention further provides a disk failure prediction processing device, comprising:
[0059] A model generation module is used to train a disk status classification model using disk operation data training samples, and to combine the disk status classification models obtained in multiple rounds of iterative training to obtain a disk failure prediction model; in each round of iterative training, the weighted disk operation data training samples are used to train the disk status classification capability of the disk status classification model of the current round, and the weight of the disk status classification model finally obtained in the current round is determined according to the disk status classification error of the disk status classification model finally obtained in the current round, and the weight of the disk operation data training samples for entering the next round of iterative training is updated; the disk status includes a normal state and a fault state;
[0060] A disk failure prediction module, used to determine the disk operation status of the target disk according to the disk operation data of the target disk of the target disk array by using the disk failure prediction model;
[0061] The disk management module is used to synchronize the data of the expected faulty disk to the backup disk if it is determined that there is an expected faulty disk in the target disk, and after the data synchronization is completed, switch the data input and output tasks of the expected faulty disk to the backup disk.
[0062] In order to solve the above technical problems, the present invention also provides a disk failure prediction processing device, comprising:
[0063] Memory for storing computer programs;
[0064] The processor is used to execute the computer program, and when the computer program is executed by the processor, the steps of the disk failure prediction processing method as described in any one of the above items are implemented.
[0065] In order to solve the above technical problems, the present invention also provides a non-volatile storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the disk failure prediction processing method as described in any one of the above items are implemented.
[0066] To solve the above technical problems, the present invention also provides a computer program product, including a computer program / instruction, wherein the computer program / instruction, when executed by a processor, implements the steps of the disk failure prediction processing method as described in any one of the above items.
[0067] The disk failure prediction processing method provided by the present invention has the beneficial effect that when solving the disk failure processing problem of the disk array, a disk failure prediction model is used to predict the disk failure, and when the expected failure disk is detected, the data of the expected failure disk is synchronized to the backup disk, and after the data synchronization is completed, the data input and output tasks of the expected failure disk are switched to the backup disk to realize the pre-processing of the expected failure disk, instead of waiting until the disk actually fails to recover the data, so as not to cause business stagnation, and can also solve the problem of low data reliability of the RAID0 level disk array. Among them, by using the weighted disk operation data training samples to perform multiple rounds of iterative training of the disk state classification ability of the disk state classification model, in each round, the weight of the disk state classification model finally obtained in the current round is determined according to the disk state classification error of the disk state classification model finally obtained, and the weight of the disk operation data training samples entering the next round of iterative training is updated, and the disk state classification model obtained in multiple rounds of iterative training is used to obtain a disk failure prediction model, and a strong classifier is generated to accurately predict the occurrence of disk failure in the disk array, reducing the probability of business stagnation or frequent disk replacement to increase operating costs when the disk fails due to prediction errors.
[0068] The present invention also provides a disk failure prediction processing device, medium and computer program product, which have the above-mentioned beneficial effects and will not be described in detail here. BRIEF DESCRIPTION OF THE DRAWINGS
[0069] In order to more clearly illustrate the embodiments of the present invention or the technical solutions of the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0070] Figure 1 An architecture diagram of a disk failure prediction and processing system provided by an embodiment of the present invention;
[0071] Figure 2 An architecture diagram of a disk failure prediction subsystem provided by an embodiment of the present invention;
[0072] Figure 3 A flowchart of a disk failure prediction and processing method provided by an embodiment of the present invention;
[0073] Figure 4 A flowchart of a disk failure prediction and replacement process provided by an embodiment of the present invention;
[0074] Figure 5 It is a structural schematic diagram of a RAID0 level disk array;
[0075] Figure 6 A schematic diagram of the management structure of a RAID0 level disk array provided in an embodiment of the present invention;
[0076] Figure 7 A schematic diagram of the first link of a disk failure switching process provided by an embodiment of the present invention;
[0077] Figure 8 A schematic diagram of the second step of a disk failure switching process provided by an embodiment of the present invention;
[0078] Fig. 9 A schematic diagram of the third step of a disk failure switching process provided by an embodiment of the present invention;
[0079] Fig.10 A schematic diagram of the structure of a disk failure prediction processing device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0080] The core of the present invention is to provide a disk failure prediction processing method, device, medium and computer program product for improving the reliability of disk array operation.
[0081] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0082] Disk arrays (such as Redundant Arrays of Independent Disks, RAID) are large-capacity disk groups composed of many independent disks. The types of disks in a disk array are mainly mechanical hard disk drives (HDD), solid state drives (SSD), and can also be other types of storage devices. Disk arrays provide data redundancy, performance improvement, and capacity expansion through different RAID levels. Data striping and parity striping are two basic technologies used when building disk arrays. Data striping is to divide data into multiple blocks (called stripe units or blocks) and then store them in parallel on multiple disks in the array. Parity striping is used to store parity information, which can be used to detect and repair single or multiple disk failures in the data. RAID 0 is the most typical level that uses data striping, which provides performance improvement but no data redundancy. RAID 5 and RAID 6 also use data striping, but also use parity striping to provide data redundancy.
[0083] When a disk fails in a RAID 5 or RAID 6 disk array, the data and checksum information on the remaining disks can be used to reconstruct the lost data, but the reconstruction process takes a certain amount of time, during which the write performance of the disk array will be affected. RAID 0 only uses data stripes to improve read and write performance, has no data redundancy, and cannot protect data security. It can only be used in situations where data input / output (I / O) performance requirements are high and data security requirements are low, and its application has great limitations.
[0084] Therefore, it is necessary to provide a solution for predicting disk failures in disk arrays for pre-processing, so as to improve the data reliability of disk arrays without disk redundancy such as RAID 0 level and solve the problem that business is affected once a failed disk occurs in other levels of disk arrays.
[0085] To solve the above problems, an embodiment of the present invention provides a disk failure processing solution to solve the problem of accurately predicting disk failures in a disk array and the problem of pre-coping with disk failures.
[0086] Figure 1 An architectural diagram of a disk failure prediction and processing system provided in an embodiment of the present invention.
[0087] To facilitate understanding, the system architecture to which the present invention is applicable is first introduced.
[0088] like Figure 1 As shown, the disk failure prediction processing system provided by the embodiment of the present invention mainly consists of a model generation subsystem and a disk failure prediction subsystem. The model generation subsystem and the disk failure prediction subsystem can be deployed on the same hardware device or different hardware devices.
[0089] The model generation subsystem is used to train and test the disk failure prediction model. The model generation subsystem can be deployed on a single server or a cluster of multiple servers. The model generation subsystem can include a first disk operation data processing module, a model training module, and a model testing module. Among them, the first disk operation data processing module is used to extract disk status attributes from historical disk operation data, perform dimensionality reduction processing, and perform preprocessing to obtain disk operation data training samples; the model training module is used to use the disk operation data training samples to train multiple disk status classification models, which are weak classification models. Then use the obtained disk status classification model 1, disk status classification model 2, ..., disk status classification model The disk failure prediction model is obtained. The model testing module is used to pre-process the disk status attribute data extracted from the historical disk operation data into disk operation data test samples, and use the disk operation data test samples to test the disk failure prediction model, so that the disk failure prediction model outputs the disk failure prediction result, and the disk failure prediction model is evaluated according to the disk failure prediction result. If the evaluation passes, it will enter the next stage.
[0090] The disk failure prediction subsystem is used to monitor the target disks in the target disk array using the trained and tested disk failure prediction model. Once an expected faulty disk is found, the target disk array is triggered to execute the automatic disk replacement process to respond before the disk failure occurs, thereby solving the data reliability problem of disk arrays without disk redundancy such as RAID 0 level, and solving the problem of business impact once a faulty disk occurs in other levels of disk arrays.
[0091] The disk failure prediction subsystem can be deployed on the monitoring node or on the disk array controller of the target disk array. Considering the load problem of the disk array controller, a microprocessor can be installed on the board of the disk array controller to deploy the disk failure prediction subsystem. The disk failure prediction subsystem can include a second disk operation data processing module and a disk online status diagnosis module. The disk failure prediction task performed by the disk failure prediction subsystem can import the disk failure prediction model into the disk array controller of the target disk, and then execute the disk failure prediction task according to the set time loop. During the execution process, the second disk operation data processing module regularly obtains the disk operation data, extracts the disk status attribute data from it for preprocessing, and then the disk online status diagnosis module inputs it into the disk failure prediction model, outputs the disk failure prediction result, and determines whether there is an expected failure disk in the target disk array based on the disk failure prediction result.
[0092] Figure 2 An architectural diagram of a disk failure prediction subsystem provided in an embodiment of the present invention.
[0093] like Figure 2 As shown, if the disk failure prediction subsystem is deployed in the target disk array controller, the disk failure prediction subsystem can be divided into a disk array management module, a cache controller, a buffer memory, a disk online status diagnosis module and a peripheral interface module according to the tasks to be performed. Among them, the disk array management module can be deployed in the disk array controller, that is, the disk switching process is added to the original management process of the disk array controller for the target disk. The disk online status diagnosis module (including the disk failure prediction model) can be deployed in the newly added microprocessor on the board of the disk array controller.
[0094] The peripheral interface module of the disk array controller can be used to connect the peripheral interface. The peripheral interface can adopt the Universal Asynchronous Receiver / Transmitter (UART), Serial Peripheral Interface (SPI), Joint Test Action Group (JTAG), Inter-Integrated Circuit (I2C), General-purpose input / output (GPIO) and other types of interfaces, which can be used to provide users with access to the disk array controller.
[0095] In addition, the disk array controller is connected to the host computer through a high-speed serial computer expansion bus switching device (Peripheral Component Interconnect Express switch, PCIe switch) or a direct connection. The bus interface processing module of the disk array controller is used to process the connection with the host computer. The host computer can be deployed with disk array controller management software, disk array controller driver, etc. for interaction with the disk array controller.
[0096] The disk array controller is connected to the target disk in the target disk array (such as Figure 2 Target Disk 0, Target Disk 1, Target Disk 3) and backup disks (such as Figure 2 The memory, system startup module and power management module in the disk array controller can all refer to the relevant technical specifications.
[0097] Based on the above architecture, the disk failure prediction and processing method provided by the embodiment of the present invention is described below in conjunction with the accompanying drawings.
[0098] Figure 3 The present invention provides a flowchart of a disk failure prediction and processing method according to an embodiment of the present invention.
[0099] like Figure 3 As shown, the disk failure prediction processing method provided by the embodiment of the present invention includes:
[0100] S301: using disk operation data training samples to train a disk status classification model, combining the disk status classification models obtained in multiple rounds of iterative training to obtain a disk failure prediction model; in each round of iterative training, using weighted disk operation data training samples to train the disk status classification capability of the disk status classification model of the current round, and determining the weight of the disk status classification model finally obtained in the current round according to the disk status classification error of the disk status classification model finally obtained in the current round, and updating the weight of the disk operation data training samples to enter the next round of iterative training; the disk status includes a normal state and a fault state;
[0101] S302: Determine the disk operation status of the target disk according to the disk operation data of the target disk of the target disk array by using the disk failure prediction model;
[0102] S303: If it is determined that there is an expected failure disk in the target disk, the data of the expected failure disk is synchronized to the backup disk, and after the data synchronization is completed, the data input and output tasks of the expected failure disk are switched to the backup disk.
[0103] In the specific implementation, disk operation data needs to be collected during the model training phase and the execution phase of the disk failure prediction task.
[0104] In the embodiment of the present invention, a disk self-monitoring analysis and reporting technology (abbreviated as SMART or SMART) may be used to collect SMART data of the disk as the disk operation status data.
[0105] According to the SMART technical standard, there are 255 SMART attributes, including a large amount of high-dimensional data. There will be complex associations and interactions between these attributes. A large number of such high-dimensional attributes will reduce the effect of machine learning. In order to reduce the impact of noise characteristics in the original data and make the trained model generalizable, the embodiment of the present invention selects attributes that are strongly correlated with disk failure prediction from the SMART attributes as disk operation data. In some optional implementations of the embodiments of the present invention, the information gain ratio method can be used to select SMART attributes, and by testing the correlation between each SMART attribute and the disk failure prediction, a preset number of SMART attributes with the highest correlation are selected as disk operation data. In other optional implementations of the embodiments of the present invention, the SMART attributes with the highest correlation selected by calculation recorded in Table 1 can also be used.
[0106] Table 1
[0107]
[0108] In order to unify the dimensions and make machine learning more stable, the disk operation data needs to be normalized so that the attribute values of each selected target attribute (SMART attribute) are normalized to the [-1,1] range. In an embodiment of the present invention, the normalization calculation formula of the disk operation data can be:
[0109] ;in, represents the normalized attribute value of the target attribute, Represents the original attribute value of the target attribute, represents the minimum value of the target attribute, Indicates the maximum value of the target attribute.
[0110] For S301, a disk operation data sample is constructed by collecting a large amount of disk SMART data, and the disk corresponding to the sample is identified as a normal disk or a faulty disk. The SMART information of each normal disk can be recorded for one week, and the SMART information of each faulty disk can be recorded for 20 days before the failure. The sampling interval is 1 hour. After collecting the disk SMART data and preprocessing (selecting the attribute value of the target attribute and normalizing it), 70% of the samples in the disk operation data sample can be used as disk operation data training samples, and the remaining 30% can be used as disk operation data test samples. The disk operation data training samples are used to train the disk failure prediction model, and the disk operation data test samples are used to test the trained disk failure prediction model to verify the reliability of the trained model. If the trained disk failure prediction model meets the requirements, the disk failure prediction model is imported into the disk array controller. If it does not meet the requirements, the training and parameter adjustment will continue until the required position is met.
[0111] In the embodiment of the present invention, since the predicted expected faulty disk involves the disk replacement problem, a disk fault prediction model with higher accuracy is required. To solve this problem, the embodiment of the present invention innovatively proposes a disk fault prediction model based on a disk status classification model, and obtains a disk fault prediction model by combining disk status classification models obtained by multiple rounds of iterative training. In each round, the weight of the disk status classification model finally obtained in the current round is determined according to the disk status classification error of the finally obtained disk status classification model, and the weight of the disk operation data training sample entering the next round of iterative training is updated. Then, relative to the disk status classification model, the disk fault prediction model becomes a strong classification model with higher prediction accuracy, so as to avoid the failure to detect the expected faulty disk or the frequent replacement of the disk due to misjudgment.
[0112] For S302, the trained disk failure prediction model is deployed on the disk array controller of the target disk, and specifically can be deployed on an independent acceleration module with an internally integrated microprocessor on the disk array controller to perform data calculations on the spot without affecting the disk array management. Figure 2 As shown, the second disk operation data processing module is used to read the SMART data of each target disk in the target disk array, and a preprocessing process consistent with the sample set data is performed; the disk online status diagnosis module is used to determine the disk status through the obtained disk fault prediction model. The disk online status diagnosis module can perform fault diagnosis on the target disk at fixed time intervals. Corresponding to the above-mentioned embodiment, if the data 20 days before the failure is used in the model training stage, three consecutive diagnoses can be performed per week during the execution stage of the disk fault monitoring task. If a target disk is monitored to be about to fail more than 2 times (including 2 times) within a week, it is tentatively determined that the disk may be about to fail, and the disk failure prediction result is sent to the disk array management module. At the same time, the disk failure prediction result is stored in the non-volatile buffer memory so that the user can read it through the peripheral interface.
[0113] In an embodiment of the present invention, if the disk failure prediction processing method is applied to a disk array controller, then using a disk failure prediction model in S302 to determine the disk operating status of a target disk according to disk operating data of a target disk of a target disk array may include: determining, according to the operating status of the disk array controller, the execution time of determining the disk operating status of a target disk according to disk operating data of a target disk of a target disk array using a disk failure prediction model.
[0114] For S303, for the expected faulty disk that is determined to be about to fail, data synchronization is used to gradually synchronize the data and data input and output tasks of the expected faulty disk to the backup disk, thereby achieving business-aware disk switching, and the business will not be stagnant or affected due to disk failure.
[0115] In addition, the disk failure prediction processing method provided by the embodiment of the present invention may further include: when an expected failed disk is detected, sending an alarm message to a preset address. The preset address may be a communication address of an operation and maintenance personnel.
[0116] The disk failure prediction processing method provided by the embodiment of the present invention has the beneficial effect that when solving the disk failure processing problem of the disk array, a disk failure prediction model is used to predict the disk failure, and when the expected failure disk is detected, the data of the expected failure disk is synchronized to the backup disk, and after the data synchronization is completed, the data input and output tasks of the expected failure disk are switched to the backup disk to realize the pre-processing of the expected failure disk, instead of waiting until the disk actually fails to recover the data, so as not to cause business stagnation, and can also solve the problem of low data reliability of the RAID0 level disk array. Among them, by using the weighted disk operation data training samples to perform multiple rounds of iterative training of the disk state classification ability of the disk state classification model, in each round, the weight of the disk state classification model finally obtained in the current round is determined according to the disk state classification error of the disk state classification model finally obtained, and the weight of the disk operation data training samples entering the next round of iterative training is updated, and the disk state classification model obtained in multiple rounds of iterative training is used to obtain a disk failure prediction model, and a strong classifier is generated to accurately predict the occurrence of disk failure in the disk array, reducing the probability of business stagnation or frequent disk replacement to increase operating costs when the disk fails due to prediction errors.
[0117] Based on the above embodiment, the embodiment of the present invention further illustrates the training and testing process of the disk failure prediction model.
[0118] In the disk failure prediction processing method provided by the embodiment of the present invention, the disk status classification model may adopt a support vector machine model.
[0119] The training steps for the disk status classification model in each round of iterative training in S301 may include: determining the kernel function of the support vector machine model; constructing the objective function of the disk status classification model using the kernel function with the goal of minimizing the disk status classification error; solving the optimal solution of the disk status classification model using the objective function of the disk status classification model and weighted disk operation data training samples; and constructing the final disk status classification model obtained in the current round using the optimal solution of the disk status classification model.
[0120] The embodiment of the present invention uses an adaptive boosting algorithm (Adaptive Boosting, Adaboost) based on a support vector machine to train the disk failure prediction model.
[0121] The disk operation data training sample is ;in, , , ; represents the disk operation data training sample set, Indicates The attribute value of the target attribute in the training sample of disk running data, Represents a set of disk attribute indicators. express dimensional real number space, Indicates the number of types of target attributes, Indicates The disk status corresponding to the disk operation data training sample, Represents a set of disk status types. Indicates that the disk status is faulty. Indicates that the disk status is normal. Indicates the total number of disk operation data training samples in the disk operation data training sample set.
[0122] Choose an appropriate kernel function and appropriate penalty parameters , The larger the value, the greater the penalty for misclassification. The smaller the value, the smaller the penalty for misclassification.
[0123] In the embodiment of the present invention, the objective function of the disk status classification model and the weighted disk operation data training samples are used to solve the optimal solution of the disk status classification model. The optimal problem can be solved by constructing and using the Lagrangian operator. Then the objective function of the disk status classification model can be expressed as follows:
[0124] ;in, , , ;
[0125] The optimal solution of the objective function of the disk status classification model is .
[0126] Select A positive component of ,calculate .
[0127] Then the decision function for constructing the disk status classification model is ;in, represents the Lagrange multiplier, Denotes the Lagrange multiplier corresponding to the The coefficients of the disk running data training samples, Denotes the Lagrange multiplier corresponding to the The coefficients of the disk running data training samples, Indicates The disk status corresponding to the disk operation data training sample, Indicates The disk status corresponding to the disk operation data training sample. represents the selected kernel function, represents the penalty parameter; represents the optimal solution of the objective function of the disk status classification model, represents the coefficient of the optimal solution corresponding to the first disk operation data training sample, represents the coefficient of the optimal solution corresponding to the second disk operation data training sample, Indicates The coefficient of the optimal solution corresponding to the disk operation data training sample, Function representing a disk status classification model.
[0128] Using the above method, in each round of iterative training of the disk failure prediction model, the disk status classification model of the current round is obtained through one or more iterative trainings.
[0129] For each round of iterative training of the disk failure prediction model, for the first round of iterative training of the disk failure prediction model, the weights of the training samples of the operation data of each disk can be initialized by randomly generating: ;in, , , Represents the disk operation data training sample weight set corresponding to the first round of iterative training, Indicates the total number of disk operation data training samples in the disk operation data training sample set.
[0130] make , perform the following iterative training:
[0131] In the In the round of iterative training, the disk status classification model is , using the disk running data training sample set with added weights Train the disk operation status classification model;
[0132] Calculate the The disk status classification error of the disk status classification model finally obtained after rounds of iterative training is:
[0133] ;in, Indicates The disk status classification error of the disk status classification model in rounds of iterative training, Indicates In the training round The weight of the disk running data training sample, Indicates the correctness of the disk status classification result. If the disk status classification result output by the disk status classification model is consistent with the disk status corresponding to the disk operation data training sample, then is 1, otherwise is 0, Indicates Disk status classification model in round training, Indicates The disk status corresponding to the disk operation data training sample.
[0134] In S301, the weight of the disk status classification model finally obtained in the current round is determined according to the disk status classification error of the disk status classification model finally obtained in the current round, and the weight of the disk operation data training samples entering the next round of iterative training is updated. It can include: after determining the weight of the disk status classification model finally obtained in the current round according to the disk status classification error of the disk status classification model finally obtained in the current round, according to the weight of the disk status classification model finally obtained in the current round, the disk status classification results of each disk operation data training sample of the disk status classification model finally obtained in the current round, the actual disk status of each disk operation data training sample and the weight of each disk operation data training sample in the current round, the weight update parameter of the current round is calculated; the weight of the disk operation data training sample in the current round is updated using the weight update parameter to obtain the weight of the disk operation data training sample entering the next round of iterative training.
[0135] In the embodiment of the present invention, according to The disk status classification error calculation of the disk status classification model finally obtained by rounds of iterative training The weight of the disk status classification model finally obtained by rounds of iterative training can be expressed as follows:
[0136] .
[0137] According to The disk status classification error of the disk status classification model finally obtained by rounds of iterative training is used to calculate and update the weight of the disk operation data training samples to enter the next round of iterative training, which can be expressed as follows:
[0138] ;
[0139] , ;
[0140] ;
[0141] in, Indicates The set of weight coefficients in round iterative training, Indicated in The weight of the first disk run data training sample in the round of iterative training, Indicated in The weight of the second disk run data training sample in the round of iterative training, Indicated in The weight of the second disk run data training sample in the round of iterative training, Indicated in In the first round of iterative training The weight of the disk running data training sample, Indicates The first The weight coefficient of the samples, Indicates that according to The result calculation of the iterative training round +1 round weight calculation formula, represents the natural exponential function with base e, Indicates The set of coefficients of the optimal solution in the Lagrange multiplier calculated in the round of iterative training.
[0142] The disk status classification models obtained in multiple rounds of iterative training are combined to obtain a disk failure prediction model, which is obtained by the following formula:
[0143] ;
[0144] in, represents a disk failure prediction model; It is a sign function, which means that when the value in the brackets is less than 0, -1 is output; when the value in the brackets is greater than 0, 1 is output; and when the value in the brackets is equal to 0, 0 is output.
[0145] In the actual disk failure prediction process, different types of kernel functions can be selected, such as polynomial, Sigmoid, etc. The kernel functions shown in Table 2 can be selected.
[0146] Table 2
[0147]
[0148] In some optional implementations of the embodiments of the present invention, determining the kernel function of the support vector machine model may include: using disk operation data training samples to train disk status classification models corresponding to multiple kernel functions, and selecting and determining the type of kernel function of the disk status classification model to be used in the iterative training of the disk failure prediction model according to the disk status classification error of the corresponding disk status classification model. That is, the disk status classification error when different kernel functions are selected by the disk operation data training samples can be used to test the disk status classification model, and the kernel function corresponding to the minimum disk status classification error can be selected.
[0149] In some other optional implementations of the embodiments of the present invention, determining the kernel function of the support vector machine model may also include: determining the type of kernel function according to the computing power of the training device. Since different kernel functions have different computational difficulties, considering the device cost, the kernel function may also be selected according to the device computing power, or, if the device computing power allows, multiple optional kernel functions may be tested to obtain the kernel function corresponding to the minimum disk status classification error.
[0150] After training the disk failure prediction model, the disk operation data test sample is used to test the obtained disk failure prediction model. Conduct a test. In the disk failure prediction processing method provided by the embodiment of the present invention, in S301, a disk state classification model is trained using a disk operation data training sample, and the disk state classification models obtained in multiple rounds of iterative training are combined to obtain a disk failure prediction model, which may include: after multiple rounds of iterative training using the disk operation data training sample, the disk state classification models obtained in each round of iterative training are combined to obtain an intermediate disk failure prediction model; the intermediate disk failure prediction model is tested using a disk operation data test sample; if the test result meets the preset disk failure prediction requirement, the iterative training is terminated to obtain the disk failure prediction model; if the test result does not meet the disk failure prediction requirement, the iterative training is continued.
[0151] In some optional implementations of the embodiments of the present invention, the test results meet the preset disk failure prediction requirements, which may include: the failure detection rate (FDR) of the intermediate disk failure prediction model for the disk operation data training samples is not less than the failure detection rate threshold, and the false alarm rate (FAR) of the intermediate disk failure prediction model for the disk operation data training samples is not greater than the false alarm rate threshold, and the failed disk prediction time (time in advance, TIA) of the intermediate disk failure prediction model for the disk operation data training samples meets the preset time range.
[0152] Table 3 is the confusion matrix of the classification results. The fault detection rate and false alarm rate of the intermediate disk failure prediction model can be calculated by referring to Table 3.
[0153] Table 3
[0154]
[0155] Referring to Table 3, the fault detection rate may be the ratio of the number of faulty disks predicted by the intermediate disk fault prediction model to the number of actual faulty disks. A larger value of the fault detection rate indicates a better recognition effect.
[0156] In the embodiment of the present invention, the fault detection rate can be expressed as ;in, represents the fault detection rate, represents the exact number of failed disks expected, Indicates the number of failed disks that were incorrectly predicted as healthy disks.
[0157] Referring to Table 3, the false alarm rate can be the ratio of the number of disks that are misjudged as healthy by the intermediate disk failure prediction model to the number of actual healthy disks. The smaller the false alarm rate, the better the recognition effect.
[0158] In the embodiment of the present invention, the false alarm rate can be expressed as ;in, represents the false alarm rate, Indicates the number of healthy disks that are incorrectly predicted as failed disks. Indicates the number of healthy disks that are accurately predicted.
[0159] The advance prediction time for a failed disk is the time from when the disk failure prediction model determines that the disk may fail to when the disk actually fails. This time should be neither too early nor too late to avoid frequent disk replacement and to reserve enough time for disk replacement.
[0160] Based on the above embodiment, the embodiment of the present invention further illustrates the process of performing disk monitoring using a disk failure prediction model.
[0161] In some optional implementations of the embodiments of the present invention, determining the disk operation status of the target disk according to the disk operation data of the target disk of the target disk array using the disk failure prediction model in S302 may include: reading the disk operation data of each target disk of the target disk array; obtaining a failure prediction result of the target disk according to the disk operation data of the target disk using the disk failure prediction model; if the failure prediction result of the target disk is an expected failure, determining the target disk as an expected failure disk. That is, the expected failure disk may be determined directly according to the disk failure prediction result obtained by the disk failure prediction model.
[0162] In some other optional implementations of the embodiments of the present invention, in order to further improve the accuracy of disk failure prediction, S302 uses a disk failure prediction model to determine the disk operation status of the target disk according to the disk operation data of the target disk of the target disk array, which can include: reading the disk operation data of each target disk of the target disk array; using the disk failure prediction model to obtain a failure prediction result for the target disk according to the disk operation data of the target disk; if there is a candidate expected failure disk whose failure prediction result is an expected failure, performing bad sector detection on the candidate expected failure disk; and determining the expected failure disk from the candidate expected failure disks according to the bad sector detection result.
[0163] In an embodiment of the present invention, performing bad track detection on a candidate expected failure disk may include: performing a full disk read operation on the candidate expected failure disk; and determining the operation status of the disk storage unit according to the read operation result of each disk storage unit of the candidate expected failure disk. The operation status of the disk storage unit may include a timeout time and a status of reading data, based on which the health status of the disk storage unit is judged, and statistical information is recorded to provide specific location information for subsequent disk repair and fault judgment.
[0164] Determining an expected failure disk from candidate expected failure disks according to the bad sector detection result may include: if the number of disk storage units in the candidate expected failure disks whose operating status is an abnormal operating status exceeds a threshold value of the number of abnormal storage units, determining the candidate expected failure disk as the expected failure disk.
[0165] In an embodiment of the present invention, determining an expected failure disk from candidate expected failure disks according to the bad track detection result may include: if the candidate expected failure disk is determined to be an expected failure disk according to the bad track detection result, then entering the step of synchronizing the data of the expected failure disk to the backup disk; if the candidate expected failure disk is determined not to be an expected failure disk according to the bad track detection result, then increasing the monitoring frequency of the expected failure disk. That is, for a target disk that is determined to be likely to fail by the disk failure prediction model but has not met the conditions of an expected failure disk according to the bad track detection, the monitoring frequency is increased for focused observation.
[0166] In an embodiment of the present invention, if it is determined in S303 that there is an expected failure disk in the target disk, the data of the expected failure disk is synchronized to the backup disk, and after the data synchronization is completed, the data input and output tasks of the expected failure disk are switched to the backup disk, which may include: after determining that the candidate expected failure disk is the expected failure disk according to the bad track detection result, the expected failure disk is included in the disk replacement queue according to the expected failure level corresponding to the bad track detection result, so that the expected failure disk with a higher expected failure level is closer to the front in the disk replacement queue; according to the order of the disk replacement queue, the expected failure disk in the disk replacement queue is synchronized to the backup disk, and after the data synchronization is completed, the data input and output tasks of the expected failure disk are switched to the backup disk. In order to cope with the situation where there may be multiple expected failure disks, a disk replacement queue can be set to determine the order of replacement of each expected failure disk. According to the computing power of the disk array controller, the number of expected failure disks that can be replaced at the same time is determined, and the corresponding number of expected failure disks at the front of the disk replacement array are dequeued to start replacement.
[0167] On the basis of the above-mentioned embodiments, in the disk failure prediction processing method provided in the embodiments of the present invention, in S303, the data of the expected failure disk is synchronized to the backup disk, and after the data synchronization is completed, the data input and output tasks of the expected failure disk are switched to the backup disk, which may include: adding the first backup disk in the backup disk group to the target disk array as the target backup disk; in the process of synchronizing the data of the expected failure disk to the target backup disk, grouping the target backup disk and the expected failure disk into a mirror group to synchronously execute the data input and output tasks received by the expected failure disk; after all the data of the expected failure disk is synchronized to the target backup disk, suspending the data input and output tasks of the expected failure disk, removing the expected failure disk from the target disk array, and replacing the target backup disk with the identifier of the expected failure disk in the target disk array.
[0168] Based on the above embodiment, the embodiment of the present invention further describes the management process of the target disk array.
[0169] Figure 4 A flowchart of a disk failure prediction and replacement process provided by an embodiment of the present invention; Figure 5 It is a structural schematic diagram of a RAID0 level disk array; Figure 6 A schematic diagram of the management structure of a RAID0 level disk array provided in an embodiment of the present invention; Figure 7 A schematic diagram of the first link of a disk failure switching process provided by an embodiment of the present invention; Figure 8 A schematic diagram of the second step of a disk failure switching process provided by an embodiment of the present invention; Fig. 9A schematic diagram of the third step of a disk failure switching process provided by an embodiment of the present invention.
[0170] Assume that the target disk array type is RAID0. When creating the target disk array, first operate through the management interface of the disk array controller, select RAID level as RAID0, create a new RAID group, then add disk members to the RAID group, and finally confirm and save the created RAID group. Figure 5 As shown, the RAID0 level target disk array includes target disk 0, target disk 1 and target disk 3, and data is stored in each target disk in the form of data stripes. Chunk data blocks in each target disk are horizontally organized into data stripes, such as A0, A1, and A2.
[0171] Configure the backup disk. In the embodiment of the present invention, the backup disk is used to replace the expected failed disk. Multiple backup disks (all global hot spare disks) can be set. When an expected failed disk appears in the target disk array, the disk array management module is used to perform automatic replacement to prevent the loss of disk array data and further deterioration of the status. The steps for configuring the backup disk are to operate through the management interface of the disk array controller. First, enter the RAID group created in the previous step, select Manage hot spare disk, then add the disk used for hot spare disk to the RAID group, and finally confirm and save the created RAID group hot spare disk. The final status is as follows: Figure 6 shown.
[0172] Enable the disk array fault diagnosis function and set the diagnosis interval. The default is once a week. Set the number of disk storage units that are poor, serious, or damaged to be tolerated. The default is 0. Configure other functions of the target disk array. After the configuration is complete, run the RAID group.
[0173] The disk online status diagnosis module performs regular diagnosis on the target disk array, diagnoses the disk status judgment conditions and performs operations.
[0174] like Figure 4As shown, if the disk is not diagnosed to be about to fail, the target disk array operates normally, and the disk failure prediction module continues to perform fault diagnosis. If it is diagnosed that the disk may fail, but it is not diagnosed that the disk is about to fail, the disk failure prediction module sends an alarm message of possible disk failure to the disk array management module, and the disk array management module sends an abnormal alarm message to the host through an interrupt. The diagnosis time interval can be shortened to 5 / 7 of the original time, rounded up, and the disk failure status is monitored more closely. If it is diagnosed that the disk may fail and it is also diagnosed that the disk is about to fail, the disk failure prediction module sends an alarm message of impending disk failure to the disk array management module, and stores the monitored information in the non-volatile buffer memory of the disk array controller. The disk array management module sends an abnormal alarm message to the host through an interrupt, and sends an asynchronous operation event to notify the host disk management module to perform an automatic disk replacement operation.
[0175] In addition, if the user determines that a disk is about to fail through the data sent by the disk array controller, the data stored by the disk array controller, and the actual application status, or performs a routine disk replacement check and issues a disk replacement operation through the controller's management tool, then it is also determined that the disk is about to fail and an automatic disk replacement operation is performed.
[0176] When an expected failed disk is detected, the disk management module automatically replaces the disk. Figure 7 As shown in steps ①②③, assuming that the target disk 1 is the expected fault disk, the disk management module uses the first unused backup disk (backup disk 0) to join the target disk array, and can identify backup disk 0 as Disk1-S, and synchronize the data in the target disk 1 (Disk1) to the backup disk Disk1-S. The target disk 1 and the backup disk Disk1-S temporarily form a mirror pair, as shown in Figure 8 As shown in step ④, when the host sends a write request or other management request such as refresh during synchronization, the disk management module copies the same request to the backup disk Disk1-S, and the backup disk Disk1-S synchronously executes the disk write and management commands. Since the read request does not affect the disk synchronization, there is no need to perform synchronization operations. When all the data on the target disk 1 is synchronized to the backup disk Disk1-S, as shown in Fig. 9 As shown in step ⑤, all requests are suspended, and the target disk 1 that is about to fail is marked as Disk1-E and removed from the RAID group. The backup disk Disk1-S is marked as the target disk 1, and the array information is written to the new target disk 1. The disk replacement is completed and the request execution is resumed. Since this process only performs the change of the disk identification, the speed is very fast and will not cause much impact on normal business operations.
[0177] Then continue to monitor each target disk in the target disk array. If the expected faulty disk is diagnosed again, the above steps can be continued until all backup disks are used up.
[0178] In the embodiment of the present invention, the non-volatile buffer memory (NVM) can be accessed through the peripheral interface module, and the peripheral interface module manages the external interface of the disk array controller, such as UART, I2C, GPIO and other interfaces. The user can view the disk fault diagnosis history results stored in the NVM through the I2C or UART interface. At the same time, the disk diagnosis command can be sent to the disk array management module through the I2C or UART interface to obtain the disk fault status at this time.
[0179] It should be noted that in the embodiments of the disk failure prediction processing method of the present invention, some of the steps or features can be ignored or not executed. The hardware or software functional modules divided for the convenience of description are not the only implementation form of the disk failure prediction processing method provided by the embodiment of the present invention.
[0180] The above describes in detail various embodiments corresponding to the disk failure prediction processing method. On this basis, the present invention also discloses a disk failure prediction processing device, equipment, non-volatile storage medium and computer program product corresponding to the above method.
[0181] The disk failure prediction processing device provided by the embodiment of the present invention may include:
[0182] The model generation module is used to train the disk status classification model using the disk operation data training samples, and combine the disk status classification models obtained in multiple rounds of iterative training to obtain a disk failure prediction model; in each round of iterative training, the disk status classification ability of the disk status classification model of the current round is trained using the weighted disk operation data training samples, and the weight of the disk status classification model finally obtained in the current round is determined according to the disk status classification error of the disk status classification model finally obtained in the current round, and the weight of the disk operation data training samples entering the next round of iterative training is updated; the disk status includes a normal state and a fault state;
[0183] A disk failure prediction module, used to determine the disk operation status of a target disk according to disk operation data of a target disk of a target disk array by using a disk failure prediction model;
[0184] The disk management module is used to synchronize the data of the expected fault disk to the backup disk if it is determined that there is an expected fault disk in the target disk, and after the data synchronization is completed, switch the data input and output tasks of the expected fault disk to the backup disk.
[0185] The model generation module in the embodiment of the present invention can refer to the introduction of the model generation subsystem in the above embodiment. The disk failure prediction module in the embodiment of the present invention can refer to the introduction of the disk failure prediction subsystem in the above embodiment. The disk management module in the embodiment of the present invention can refer to the introduction of the disk management module in the above embodiment.
[0186] It should be noted that in each implementation of the disk fault prediction processing device provided by the embodiment of the present invention, the division of units is only a logical functional division, and other division methods can be used. The connection method between different units can be electrical, mechanical or other connection methods. The separated units can be located in the same physical location or distributed on multiple network nodes. Each unit can be implemented in the form of hardware or in the form of a software functional unit. That is, according to actual needs, part or all of the units provided by the embodiment of the present invention can be selected and the corresponding connection method or integration method can be adopted to achieve the purpose of the embodiment of the present invention.
[0187] Since the embodiments of the apparatus part correspond to the embodiments of the method part, please refer to the description of the embodiments of the method part for the embodiments of the apparatus part, which will not be repeated here.
[0188] Fig.10 A schematic diagram of the structure of a disk failure prediction processing device provided by an embodiment of the present invention.
[0189] like Fig.10 As shown, the disk failure prediction processing device provided in an embodiment of the present invention includes: a memory 1010, used to store a computer program 1011; a processor 1020, used to execute the computer program 1011, and when the computer program 1011 is executed by the processor 1020, the steps of the disk failure prediction processing method provided in any of the above embodiments are implemented.
[0190] Among them, the processor 1020 may include one or more processing cores, such as a 3-core processor, an 8-core processor, etc. The processor 1020 may be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 1020 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a central processing unit (CPU); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 1020 may be integrated with a graphics processing unit (GPU), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1020 may also include an artificial intelligence (AI) processor, which is used to process computing operations related to machine learning.
[0191] The memory 1010 may include one or more non-volatile storage media, which may be non-transitory. The memory 1010 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 1010 is at least used to store the following computer program 1011, wherein the computer program 1011, after being loaded and executed by the processor 1020, can implement the relevant steps in the disk failure prediction processing method disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 1010 may also include an operating system 1012 and data 1013, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 1012 may be Windows or other types of operating systems. The data 1013 may include but is not limited to the data involved in the above method.
[0192] In some embodiments, the disk failure prediction processing device may further include a display screen 1030 , a power supply 1040 , a communication interface 1050 , an input / output interface 1060 , a sensor 1070 , and a communication bus 1080 .
[0193] Those skilled in the art will understand that Fig.10 The structure shown in the figure does not constitute a limitation on the disk failure prediction processing device, and may include more or less components than those shown in the figure.
[0194] The disk failure prediction processing device provided in the embodiment of the present invention includes a memory and a processor. When the processor executes the program stored in the memory, it can implement the steps of the disk failure prediction processing method provided in the above embodiment, and the effect is the same as above.
[0195] An embodiment of the present invention provides a non-volatile storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the disk failure prediction processing method provided in any one of the above embodiments can be implemented.
[0196] The non-volatile storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.
[0197] For the introduction of the non-volatile storage medium provided in the embodiment of the present invention, please refer to the above method embodiment, and the effect thereof is the same as the disk failure prediction and processing method provided in the embodiment of the present invention, and the present invention will not elaborate on it here.
[0198] An embodiment of the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the disk failure prediction processing method provided in any one of the above embodiments.
[0199] For an introduction to the computer program product provided by the embodiment of the present invention, please refer to the above method embodiment, and the effect thereof is the same as the disk failure prediction and processing method provided by the embodiment of the present invention, and the present invention will not elaborate on it here.
[0200] The above is a detailed introduction to the disk failure prediction processing method, device, medium and computer program product provided by the present invention. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referenced to each other. For the devices, equipment, non-volatile storage media and computer program products disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, the present invention can also be improved and modified in a number of ways, and these improvements and modifications also fall within the scope of protection of the present invention.
[0201] It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.
Claims
1. A disk failure prediction processing method, characterized in that: include: The disk status classification model is trained using disk operation data training samples, and the disk status classification models obtained in multiple rounds of iterative training are combined to obtain a disk failure prediction model; in each round of iterative training, the disk status classification ability of the disk status classification model of the current round is trained using the weighted disk operation data training samples, and the weight of the disk status classification model finally obtained in the current round is determined according to the disk status classification error of the disk status classification model finally obtained in the current round, and the weight of the disk operation data training samples entering the next round of iterative training is updated; The disk status includes normal status and fault status; Determine the disk operation status of the target disk according to the disk operation data of the target disk of the target disk array by using the disk failure prediction model; If it is determined that there is an expected fault disk in the target disk, the data of the expected fault disk is synchronized to the backup disk, and after the data synchronization is completed, the data input and output tasks of the expected fault disk are switched to the backup disk; The disk operation data training sample is generated according to the SMART information of the normal disk and the SMART information of the faulty disk before the fault occurs; The disk status classification model is a support vector machine model; The training steps of the disk status classification model in each round of iterative training include: Determining a kernel function of the support vector machine model; Taking minimizing the disk status classification error as the goal, constructing the objective function of the disk status classification model using the kernel function; Utilizing the objective function of the disk status classification model and the weighted disk operation data training samples to solve the optimal solution of the disk status classification model; Using the optimal solution of the disk status classification model, construct the disk status classification model finally obtained in the current round; Determining the weight of the disk status classification model finally obtained in the current round according to the disk status classification error of the disk status classification model finally obtained in the current round and updating the weight of the disk operation data training sample to enter the next round of iterative training includes: After determining the weight of the disk status classification model finally obtained in the current round according to the disk status classification error of the disk status classification model finally obtained in the current round, the weight update parameter of the current round is calculated according to the weight of the disk status classification model finally obtained in the current round, the disk status classification result of each disk operation data training sample by the disk status classification model finally obtained in the current round, the actual disk status of each disk operation data training sample and the weight of each disk operation data training sample in the current round; Using the weight update parameter to update the weight of the disk operation data training sample in the current round, to obtain the weight of the disk operation data training sample for the next round of iterative training; The disk status classification model is trained using disk operation data training samples, and the disk status classification models obtained in multiple rounds of iterative training are combined to obtain a disk failure prediction model, including: After multiple rounds of iterative training using the disk operation data training samples, the disk status classification models obtained from each round of iterative training are combined to obtain an intermediate disk failure prediction model; Testing the intermediate disk failure prediction model using a disk operation data test sample; If the test result meets the preset disk failure prediction requirements, the iterative training is terminated to obtain the disk failure prediction model; If the test result does not meet the disk failure prediction requirement, the iterative training continues.
2. The disk failure prediction processing method according to claim 1, characterized in that: Determining a kernel function of the support vector machine model includes: The disk operation data training samples are used to train the disk status classification models corresponding to multiple kernel functions, and the type of the kernel function of the disk status classification model to be used in the iterative training of the disk failure prediction model is determined based on the disk status classification error of the corresponding disk status classification model.
3. The disk failure prediction processing method according to claim 1, characterized in that: Determining a kernel function of the support vector machine model includes: The type of the kernel function is determined according to the computing power of the training device.
4. The disk failure prediction processing method according to claim 1, characterized in that: The test result meets the preset disk failure prediction requirement, including: The fault detection rate of the intermediate disk failure prediction model for the disk operation data training samples is not less than the fault detection rate threshold, and the false alarm rate of the intermediate disk failure prediction model for the disk operation data training samples is not greater than the false alarm rate threshold, and the advance prediction time of the faulty disk of the intermediate disk failure prediction model for the disk operation data training samples meets the preset time range.
5. The disk failure prediction processing method according to claim 4, characterized in that: The failure detection rate is a ratio of the number of failed disks predicted by the intermediate disk failure prediction model to the number of actual failed disks.
6. The disk failure prediction processing method according to claim 4, characterized in that: The false alarm rate is the ratio of the number of disks misjudged as healthy by the intermediate disk failure prediction model to the number of actual healthy disks.
7. The disk failure prediction processing method according to claim 1, characterized in that: Determining the disk operation status of the target disk of the target disk array according to the disk operation data of the target disk by using the disk failure prediction model includes: Reading disk operation data of each target disk of the target disk array; Obtaining a failure prediction result of the target disk according to the disk operation data of the target disk using the disk failure prediction model; If the failure prediction result of the target disk is an expected failure, the target disk is determined to be the expected failure disk.
8. The disk failure prediction processing method according to claim 1, characterized in that: Determining the disk operation status of the target disk of the target disk array according to the disk operation data of the target disk by using the disk failure prediction model includes: Reading disk operation data of each target disk of the target disk array; Obtaining a failure prediction result of the target disk according to the disk operation data of the target disk using the disk failure prediction model; If there is a candidate expected failure disk whose failure prediction result is an expected failure, performing bad track detection on the candidate expected failure disk; The expected failure disk is determined from the candidate expected failure disks according to the bad sector detection result.
9. The disk failure prediction processing method according to claim 8, characterized in that: Performing bad track detection on the candidate expected failure disk includes: Performing a full disk read operation on the candidate expected failure disk; Determining the operating status of the disk storage unit according to the read operation result of each disk storage unit of the candidate expected failure disk; The step of determining the expected failure disk from the candidate expected failure disks according to the bad track detection result includes: If the number of the disk storage units in the candidate expected failure disk whose operating status is an abnormal operating status exceeds a threshold value of the number of abnormal storage units, the candidate expected failure disk is determined to be the expected failure disk.
10. The disk failure prediction processing method according to claim 8, characterized in that: The step of determining the expected failure disk from the candidate expected failure disks according to the bad track detection result includes: If it is determined according to the bad sector detection result that the candidate expected failure disk is the expected failure disk, then entering the step of synchronizing the data of the expected failure disk to a backup disk; If it is determined according to the bad sector detection result that the candidate expected failure disk is not the expected failure disk, the monitoring frequency of the expected failure disk is increased.
11. The disk failure prediction processing method according to claim 10, characterized in that: If it is determined that there is an expected fault disk in the target disk, synchronizing the data of the expected fault disk to the backup disk, and after the data synchronization is completed, switching the data input and output tasks of the expected fault disk to the backup disk, including: After determining that the candidate expected failure disk is the expected failure disk according to the bad sector detection result, the expected failure disk is included in a disk replacement queue according to the expected failure level corresponding to the bad sector detection result, so that the expected failure disk with a higher expected failure level is placed at the front of the disk replacement queue; According to the order of the disk replacement queue, the data of the expected failure disk in the disk replacement queue is synchronized to the backup disk, and after the data synchronization is completed, the data input and output task of the expected failure disk is switched to the backup disk.
12. The disk failure prediction processing method according to claim 1, characterized in that: Synchronizing the data of the expected faulty disk to the backup disk, and after the data synchronization is completed, switching the data input and output tasks of the expected faulty disk to the backup disk, including: Adding the first backup disk in the backup disk group to the target disk array as a target backup disk; In the process of synchronizing the data of the expected failure disk to the target backup disk, the target backup disk and the expected failure disk are grouped into a mirror group to synchronously execute the data input and output tasks received by the expected failure disk; After all the data of the expected failure disk is synchronized to the target backup disk, the data input and output tasks of the expected failure disk are suspended, the expected failure disk is removed from the target disk array, and the target backup disk is replaced with the identifier of the expected failure disk in the target disk array.
13. The disk failure prediction processing method according to claim 1, characterized in that: Applied to a disk array controller, the disk failure prediction model is used to monitor the disk operation status of a target disk array, including: The execution time of monitoring the disk operation status of the target disk array by using the disk failure prediction model is determined according to the operation status of the disk array controller.
14. A disk failure prediction processing device, characterized in that: include: A model generation module is used to train a disk status classification model using disk operation data training samples, and to combine the disk status classification models obtained in multiple rounds of iterative training to obtain a disk failure prediction model; in each round of iterative training, the disk status classification capability of the disk status classification model of the current round is trained using the weighted disk operation data training samples, and the weight of the disk status classification model finally obtained in the current round is determined according to the disk status classification error of the disk status classification model finally obtained in the current round, and the weight of the disk operation data training samples for entering the next round of iterative training is updated; The disk status includes normal status and fault status; A disk failure prediction module, used to determine the disk operation status of the target disk according to the disk operation data of the target disk of the target disk array by using the disk failure prediction model; A disk management module, configured to synchronize the data of the expected faulty disk to a backup disk if it is determined that there is an expected faulty disk in the target disk, and switch the data input and output tasks of the expected faulty disk to the backup disk after the data synchronization is completed; The disk operation data training sample is generated according to the SMART information of the normal disk and the SMART information of the faulty disk before the fault occurs; The disk status classification model is a support vector machine model; The training steps of the disk status classification model in each round of iterative training include: Determining a kernel function of the support vector machine model; Taking minimizing the disk status classification error as the goal, constructing the objective function of the disk status classification model using the kernel function; Utilizing the objective function of the disk status classification model and the weighted disk operation data training samples to solve the optimal solution of the disk status classification model; Using the optimal solution of the disk status classification model, construct the disk status classification model finally obtained in the current round; Determining the weight of the disk status classification model finally obtained in the current round according to the disk status classification error of the disk status classification model finally obtained in the current round and updating the weight of the disk operation data training sample to enter the next round of iterative training includes: After determining the weight of the disk status classification model finally obtained in the current round according to the disk status classification error of the disk status classification model finally obtained in the current round, the weight update parameter of the current round is calculated according to the weight of the disk status classification model finally obtained in the current round, the disk status classification result of each disk operation data training sample by the disk status classification model finally obtained in the current round, the actual disk status of each disk operation data training sample and the weight of each disk operation data training sample in the current round; Using the weight update parameter to update the weight of the disk operation data training sample in the current round, to obtain the weight of the disk operation data training sample for the next round of iterative training; The disk status classification model is trained using disk operation data training samples, and the disk status classification models obtained in multiple rounds of iterative training are combined to obtain a disk failure prediction model, including: After multiple rounds of iterative training using the disk operation data training samples, the disk status classification models obtained from each round of iterative training are combined to obtain an intermediate disk failure prediction model; Testing the intermediate disk failure prediction model using a disk operation data test sample; If the test result meets the preset disk failure prediction requirements, the iterative training is terminated to obtain the disk failure prediction model; If the test result does not meet the disk failure prediction requirement, the iterative training continues.
15. A disk failure prediction processing device, characterized in that: include: Memory for storing computer programs; A processor is used to execute the computer program, and when the computer program is executed by the processor, the steps of the disk failure prediction processing method as described in any one of claims 1 to 13 are implemented.
16. A non-volatile storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the disk failure prediction processing method according to any one of claims 1 to 13 are implemented.
17. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the disk failure prediction processing method according to any one of claims 1 to 13 are implemented.
