Disk failure prediction method, prediction model training method, and electronic device

Abstract

The application provides a disk failure prediction method, a prediction model training method and an electronic device. The disk failure prediction method comprises: obtaining a prediction data set of a to-be-predicted disk, wherein the prediction data set comprises IO information of a prediction sample IO and SMART information corresponding to the prediction sample IO, and the prediction data set is collected in a cache disk acceleration scene of the to-be-predicted disk; inputting the prediction data set into a pre-trained prediction model to obtain a prediction result of the to-be-predicted disk. According to the scheme provided in the embodiment of the application, the IO information and the SMART information can be combined to predict the disk failure of all types of disks, and the risk of data loss is effectively reduced.

Description

Technical Field

[0001] This invention relates to, but is not limited to, the field of data storage, and particularly to a disk failure prediction method, a prediction model training method, and an electronic device. Background Technology

[0002] With the development of network and communication technologies, the amount of data stored in server data centers is increasing rapidly. Disks are crucial hardware devices for data storage, and larger data centers typically have a large number of disks. Disks generally have a limited lifespan, and the probability of failure increases significantly towards the end of their lifespan. To address this issue, replication technology or erasure coding-based data redundancy is commonly used. However, this only prevents data loss due to the failure of a single disk; when multiple disks fail simultaneously, the risk of data loss still exists.

[0003] Therefore, it is usually necessary to predict disk failure during disk operation and replace the disk in a timely manner when a high risk of failure is detected, thereby reducing the risk of data loss. A common approach is to use a trained prediction model for failure prediction. However, the training data used by existing prediction models is usually Self-Monitoring Analysis and Reporting Technology (SMART) information. SMART information is only applicable to Serial Advanced Technology Attachment (SATA) hard drives, which have many different disk parameters. It cannot provide accurate predictions for Serial Attached SCSI (SAS) hard drives, which have fewer disk parameters. Summary of the Invention

[0004] The following is an overview of the subject matter described in detail herein. This overview is not intended to limit the scope of the claims.

[0005] This invention provides a disk failure prediction method, a prediction model training method, and an electronic device capable of predicting failures for all types of disks.

[0006] In a first aspect, embodiments of the present invention provide a disk failure prediction method, comprising:

[0007] Obtain the prediction dataset of the disk to be predicted. The prediction dataset includes IO information of the prediction sample input and output (Input and Output, IO) and SMART information corresponding to the prediction sample IO. The prediction dataset is collected in the cache disk acceleration scenario of the disk to be predicted.

[0008] The prediction dataset is input into a pre-trained prediction model to obtain the prediction result for the disk to be predicted.

[0009] This invention includes: acquiring a prediction dataset for a disk to be predicted, the prediction dataset including IO information of predicted sample IOs and SMART information corresponding to the predicted sample IOs, wherein the prediction dataset is collected in a cache disk acceleration scenario of the disk to be predicted; inputting the prediction dataset into a pre-trained prediction model to obtain the prediction result for the disk to be predicted. According to the solution provided by this invention, disk failure prediction can be performed on all types of disks by combining IO information and SMART information, effectively reducing the risk of data loss.

[0010] Secondly, embodiments of the present invention also provide a prediction model training method, including:

[0011] Obtain the predicted training sample set of the training sample disk. The predicted training sample set includes the training sample IO information of the training sample IO and the training sample SMART information corresponding to the training sample IO. The predicted training sample set is collected from the cache disk acceleration scenario of the training sample disk.

[0012] The prediction model is trained based on the prediction training sample set.

[0013] Thirdly, embodiments of the present invention also provide an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the disk failure prediction method as described in the first aspect, or to execute the prediction model training method as described in the second aspect.

[0014] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing computer-executable instructions, the computer-executable instructions being used to execute the disk failure prediction method as described in the first aspect, or to execute the prediction model training method as described in the second aspect.

[0015] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the description, claims and drawings. Attached Figure Description

[0016] The accompanying drawings are provided to further understand the technical solutions of the present invention and constitute a part of the specification. They are used together with the embodiments of the present invention to explain the technical solutions of the present invention, and do not constitute a limitation on the technical solutions of the present invention.

[0017] Figure 1 This is a flowchart of a disk failure prediction method provided in one embodiment of the present invention;

[0018] Figure 2 This is a schematic diagram of the module framework provided in another embodiment of the present invention;

[0019] Figure 3 This is a flowchart of determining the prediction result based on the prediction period provided in another embodiment of the present invention;

[0020] Figure 4 This is a flowchart of determining the prediction result based on the periodic failure probability provided in another embodiment of the present invention;

[0021] Figure 5 This is a flowchart of another embodiment of the present invention for determining prediction results based on the number of times a cycle is identified as high-risk;

[0022] Figure 6 This is a flowchart of determining whether the disk to be predicted is in a cache disk acceleration scenario, provided by another embodiment of the present invention;

[0023] Figure 7 This is a flowchart of a prediction model training method provided in another embodiment of the present invention;

[0024] Figure 8 This is a flowchart of determining the training sample disk in a cache disk acceleration scenario provided by another embodiment of the present invention;

[0025] Figure 9 This is a flowchart of obtaining a prediction training sample set provided in another embodiment of the present invention;

[0026] Figure 10 This is a flowchart of determining training sample IO information according to preset conditions provided in another embodiment of the present invention;

[0027] Figure 11 This is a flowchart of training a prediction model according to a training cycle provided in another embodiment of the present invention;

[0028] Figure 12 This is a flowchart of dividing the prediction sample training set into a training sample set and a test sample set, provided by another embodiment of the present invention;

[0029] Figure 13This is a schematic diagram of the structure of an electronic device provided in another embodiment of the present invention. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0031] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, or the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0032] This invention provides a disk failure prediction method, a prediction model training method, and an electronic device. The disk failure prediction method includes: acquiring a prediction dataset of the disk to be predicted, the prediction dataset including IO information of predicted sample IO and SMART information corresponding to the predicted sample IO, wherein the prediction dataset is collected in a cache disk acceleration scenario of the disk to be predicted; inputting the prediction dataset into a pre-trained prediction model to obtain the prediction result of the disk to be predicted. According to the solution provided by the embodiments of this invention, disk failure prediction can be performed on all types of disks by combining IO information and SMART information, effectively reducing the risk of data loss.

[0033] The embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0034] like Figure 1 As shown, Figure 1 This is a flowchart of a disk failure prediction method provided in an embodiment of the present invention. The disk failure prediction method includes, but is not limited to, steps S110 and S120.

[0035] Step S110: Obtain the prediction dataset of the disk to be predicted. The prediction dataset includes the IO information of the prediction sample IO and the SMART information corresponding to the prediction sample IO. The prediction dataset is collected from the cache disk acceleration scenario of the disk to be predicted.

[0036] It should be noted that the IO information of each prediction sample IO of the disk to be predicted includes multiple attributes such as IO latency, IO size, and IO status information. Therefore, using IO information as the input of the prediction model can effectively alleviate the problem of insufficient attributes.

[0037] It should be noted that this embodiment can determine the minimum allowable time for predicted sample IO by the number of read / write operations per second (IOPS) of the disk. This minimum allowable time is set as a duration threshold. When the sum of the durations of several IOs exceeds this duration threshold, it can be determined that the predicted sample IOs are all large block IOs, that is, the disk working scenario corresponding to the IOs belongs to the cache disk acceleration scenario. It is worth noting that in the cache disk acceleration scenario, it is easy to know that the disk IOs sent tend to be large block reads and writes, with fewer small block IOs. At the same time, the maximum IO at the block level is usually 512K, so the range of IO sizes is relatively small. This provides a basis for tracking the size of disk IOs. Those skilled in the art will understand that in storage systems, in order to ensure the quality of storage services, front-end applications usually perform service capability matching, and the storage side sets front-end Quality of Service (QoS) and back-end QoS, etc. These QoS settings effectively prevent IO bursts in most cases, avoid excessive IO queue depth, and prevent disk overload, which would prevent the provision of stable services. In summary, in the context of cache disk acceleration, statistical information on the latency of large-block I / O under a certain load is of significant importance. It provides rich disk status indicators from the application level. Therefore, I / O information in the context of cache disk acceleration can be used for failure prediction.

[0038] It is worth noting that the prediction dataset in this embodiment includes both IO information and corresponding SMART information. The corresponding SMART information can be the SMART information during the execution of the prediction sample IO process, or it can be the SMART information collected according to the collection cycle, such as once a day. The specific collection method and cycle can be adjusted according to the actual situation, as long as the SMART information and IO information have a certain correlation.

[0039] It should be noted that, due to the different physical properties of different types of disks, their failure criteria differ. When training the predictive model, disk type information can be used as one of the selected features, enabling the model's predictions to characterize the failure risk of that type of disk. Based on this, disk type information can also be used as input to the predictive model, allowing it to predict failures for different types of disks. It is understood that disk type information may include disk manufacturer, disk model, disk capacity, disk serial number, and rotational speed; this embodiment does not impose further limitations on this.

[0040] Step S120: Input the prediction dataset into the pre-trained prediction model to obtain the prediction result of the disk to be predicted.

[0041] In one embodiment, the specific timing of the forecast can be determined by making a forecast once a day based on operational needs to reduce the risk of data loss, or it can be made after each collection of IO and SMART information, which can be adjusted according to actual needs.

[0042] It's important to note that the I / O path is influenced not only by the disk itself but also by the controller, expansion cards, cables, and even the operating system. I / O information is a comprehensive set of data that needs to be considered in conjunction with other disk-related information for accurate judgment. It cannot be used in isolation for prediction; otherwise, false alarms may occur, such as reporting a problem with a physical port on an expansion card as a disk failure. SMART information, on the other hand, characterizes the disk's parameter status. Therefore, combining the characteristics of I / O information with those of SMART information allows for a more accurate prediction of the disk's failure risk.

[0043] Understandably, when using a prediction model to obtain prediction results, the prediction model can be trained in advance based on IO information and SMART information. For example, the latency information in IO information, the rate of change of each parameter in SMART information, and the absolute value of the increase in the rate of change of each parameter in SMART information can be used as training features. After labeling the features, they are input into the prediction module for training, so that the prediction model can obtain the prediction result of the disk to be predicted based on the above features.

[0044] In one embodiment, the prediction result can take any form, such as the current failure risk probability of the disk to be predicted, or a specific risk value, etc. It can also be determined according to a specific collection period. For example, if the prediction dataset is data within one week, the prediction result is the failure risk probability in the next week. As long as it can reflect the failure risk of the disk to be predicted, no further limitations are imposed here.

[0045] In another embodiment, the system architecture diagram for applying the disk failure prediction method of this application can be as follows: Figure 2As shown, the system includes a prediction center and several agent nodes. Both the prediction center and agent nodes can take the form of electronic devices or servers, without further limitation. The prediction center may include an alarm management module, a prediction module, and a disk information management module. The alarm management module is used to issue alarms when a disk is detected to be at high risk. The prediction module is used to predict the failure risk of the disk to be predicted based on the prediction dataset. The disk information management module is used to receive and manage IO information and SMART information sent by the agent nodes, and form a prediction dataset. Each agent node includes an IO module and a SMART module. The IO module is used to acquire the IO information of the agent node's disk and filter the IO information according to preset rules, so that the filtered IO information can be used to form the prediction dataset. The SMART module is used to collect the SMART information of the agent node's disk. It should be noted that this application does not involve specific structural improvements to the agent nodes and the prediction center, but only to the processing of the collected data, which will not be elaborated upon here.

[0046] Additionally, refer to Figure 3 In one embodiment, the IO information also includes IO time information. Figure 1 Step S120 in the illustrated embodiment also includes, but is not limited to, the following steps:

[0047] Step S310: Determine the prediction period by determining the period dataset from the prediction dataset based on the prediction period and IO time information.

[0048] Step S320: Based on the periodic dataset and the prediction model, the probability of periodic failure of the disk to be predicted in the prediction period is obtained.

[0049] Step S330: Determine the prediction result of the disk to be predicted based on the periodic failure probability.

[0050] In one embodiment, the prediction period can be selected according to actual needs. For example, to determine the near-term failure risk of the disk to be predicted, the prediction period can be set to several days, one week, or two weeks. If the goal is to determine the failure risk of the disk to be predicted over a longer period, the prediction period can be set to one month. The specific period can be adjusted according to actual needs. It is understood that the prediction period can be any number. For example, data from one week, two weeks, and four weeks can be obtained simultaneously, and the prediction results for each prediction period can be obtained, making the failure risk prediction of the disk to be predicted more accurate.

[0051] In one embodiment, such as Figure 2As shown, the disk information management module stores the collected prediction dataset. In order to reduce storage pressure, the collected data can also be managed for lifecycle. For example, if the prediction period is set to one week, lifecycle management can be used to ensure that the IO information and SMART information in the prediction dataset in the disk information management module are all data collected within one week. The specific lifecycle management method is not an improvement made in this embodiment and will not be described in detail here.

[0052] In one embodiment, the IO time information can be the specific time information of the disk to be predicted to execute the IO, such as the time when the IO starts to be executed or the time when the IO is completed. The specific selection criteria can be adjusted according to the actual situation, and will not be limited here.

[0053] In one embodiment, the periodic failure probability is the probability of failure of the disk to be predicted with the prediction period as the length. For example, the prediction period is one week. The obtained periodic dataset consists of IO information and SMART information collected in the past week. Failure prediction is performed based on the above information, and the resulting periodic failure probability is the probability of failure of the disk to be predicted in the next week.

[0054] Additionally, refer to Figure 4 In one embodiment, Figure 3 Step S330 in the illustrated embodiment also includes, but is not limited to, the following steps:

[0055] Step S410: When the probability of periodic failure is greater than the preset probability threshold corresponding to the prediction period, the prediction result is determined to be high risk.

[0056] Step S420: When the probability of periodic failure is less than or equal to the probability threshold, the prediction result is determined to be low risk.

[0057] It should be noted that the probability threshold can be determined according to the actual risk management needs. For example, a probability threshold greater than 80% can be set as a high failure risk, and a probability threshold less than or equal to 80% can be set as a low failure risk. This embodiment does not limit the specific probability threshold. Of course, several probability thresholds corresponding to several risk levels can also be set according to actual needs, which will not be elaborated here.

[0058] It should be noted that using risk level as the prediction result reflects the probability of disk failure, allowing for advance scheduling of disk replacement when a high-risk prediction is detected. This is particularly beneficial in reducing the risk of multiple disks failing simultaneously, thus minimizing data loss. Furthermore, by setting a probability threshold for low risk, alarms are not generated when the periodic failure probability is below this threshold, effectively reducing false alarm rates.

[0059] It is understandable that the cycle failure probability is the prediction result corresponding to the prediction period and will not affect the prediction results of different prediction periods. For example, the cycle failure probability of a prediction period of one week and the cycle failure probability of a prediction period of two weeks are independent parameters. That is, when determining the prediction result of a prediction period of two weeks, the cycle failure probability of a prediction period of one week will not be considered. This will not be elaborated on here.

[0060] Additionally, refer to Figure 5 In one embodiment, Figure 4 Step S410 in the illustrated embodiment also includes, but is not limited to, the following steps:

[0061] Step S510: When the probability of periodic failure is greater than the probability threshold, the predicted period corresponding to the probability of periodic failure is determined to be a high-risk period.

[0062] Step S520: When the number of times a prediction period is determined to be a high-risk period exceeds the preset alarm threshold, the prediction result is determined to be high-risk.

[0063] In one embodiment, using an alarm number threshold can effectively reduce the number of false alarms. In the actual prediction process, since the periodic prediction method is used, the data on a certain day may be abnormal, which may lead to a high-risk prediction result. Therefore, by making multiple predictions, the deviation of the prediction result caused by occasional anomalies can be effectively reduced. The specific alarm number threshold can be adjusted according to actual needs, and will not be limited here.

[0064] In one embodiment, when the prediction result is determined to be high-risk, alarm information can also be generated based on the basic information of the disk to be predicted. The basic information may include the disk model, installation location, etc., and is not limited here. It is understood that the alarm information can be generated through... Figure 2 The alarm management module shown generates alarm information, such as pushing alarm information to the backend management system, and carries basic disk information in the alarm information so that maintenance personnel can perform disk maintenance in a timely and accurate manner.

[0065] Additionally, refer to Figure 6 In one embodiment, the IO information also includes IO duration and IO size, and the cache disk acceleration scenario of the disk to be predicted is determined by the following steps:

[0066] Step S610: Obtain the IOPS of the disk to be predicted, and determine the duration threshold based on the IOPS and IO size of the disk to be predicted.

[0067] Step S620: When the IO duration exceeds the duration threshold, determine that the disk to be predicted is in the cache disk acceleration scenario.

[0068] It should be noted that the IOPS of the disk to be predicted can be obtained in any way, such as reading the IOPS performance parameters of the disk to be predicted, or by performing several IO tests on the disk to be predicted through actual testing. The specific method can be selected according to the actual needs.

[0069] Understandably, the duration threshold can be obtained by dividing the IO size by IOPS. Since IOPS is used to characterize the disk's read and write operation capabilities, the duration threshold can be used to characterize the minimum allowed time required for the disk to process a specific IO size. When the IO duration exceeds this minimum allowed time, it can be determined that the IO execution is in a cache disk acceleration scenario.

[0070] It is worth noting that this can be achieved through an IO.

[0071] In another embodiment, the SMART information includes at least one of the following:

[0072] Total number of start-stop cycles;

[0073] Total number of load and uninstall counts;

[0074] Bad habits of growth;

[0075] Non-media error count;

[0076] The number of irreparable errors.

[0077] In one embodiment, SMART information may include any available attributes, such as SMART Health Status, accumulated start-stop cycles, accumulated load-unload cycles, elements in grown defectlist, non-medium error count, and number of uncorrectable errors. The number of uncorrectable errors may include total uncorrected read errors and total uncorrected write errors. Those skilled in the art will be motivated to add or reduce specific disk parameters according to actual needs, but no limitations are made here.

[0078] Understandably, based on the disk parameters mentioned above, in order to characterize the risk of disk failure, the rate of change and increment value of each disk parameter can be used. The rate of change of a disk parameter can be a parameter that characterizes how fast a specific value changes, and the increment value of a disk parameter can be the absolute value of the increment value. Anything that can be used to characterize the magnitude of change of the disk parameter is sufficient. The greater the magnitude of change of the disk parameter, the greater the risk of disk failure.

[0079] Additionally, refer to Figure 7 The present invention also provides a prediction model training method, including but not limited to steps S710 and S720.

[0080] Step S710: Obtain the predicted training sample set of the training sample disk. The predicted training sample set includes training sample IO information of several training sample IOs and training sample SMART information corresponding to the training sample IOs. The predicted training sample set is collected from the cache disk acceleration scenario of the training sample disk.

[0081] In one embodiment, the prediction training sample set can be obtained through an IOPS performance model from... Figure 1 In the cache disk acceleration scenario described in the illustrated embodiment, the IOPS performance model can be obtained by manually testing the IOPS performance of different large IO blocks at different queue depths. It is understood that the IOPS performance model can be used to characterize the disk's read and write capabilities. Therefore, for a certain number of IOs, the estimated allowable time, i.e., the aforementioned duration threshold, can be calculated using the IOPS performance model. When the actual processing time of several IOs exceeds this duration threshold, the IOs can be considered to originate from the cache disk acceleration scenario and can be identified as valid samples. It is understood that the prediction training sample set can be collected periodically, such as once a day, with the specific period selected according to actual needs.

[0082] It is worth noting that the principle behind collecting prediction training samples in a cache disk-accelerated scenario can be found in [reference needed]. Figure 2 The principles described in the embodiments will not be elaborated upon here.

[0083] Step S720: Train the prediction model based on the prediction training sample set.

[0084] In one embodiment, the prediction model can be trained once a day, or adjusted according to actual needs, without further limitation. It is understood that when the training sample set includes several sample subsets, training can be performed separately for each sample subset. For example, if prediction training sample sets for one week, two weeks, and four weeks are collected according to different periods, then training can be performed for the corresponding periods, thereby enabling the prediction model to predict failures for different periods.

[0085] In one embodiment, the prediction model can employ a common model framework, such as the LightGBM framework. It should be noted that before training the prediction model, the basic parameters of the model need to be set. For example, when using the LightGBM framework, the framework parameters can be set as shown in Table 1 below:

[0086] Parameter name value Learing rate 0.35 Iteration rounds 110 Cross validation 5 Total sample number 5160 Terminal condition <![CDATA[10 -4 ]]>

[0087] Table 1 Model Framework Parameter Configuration Table

[0088] Additionally, refer to Figure 8 In one embodiment, the training sample I / O information includes the training sample I / O duration and the training sample I / O size. Figure 7 Step S710 in the illustrated embodiment also includes, but is not limited to, the following steps:

[0089] Step S810: Obtain the IOPS of the training sample disk, and determine the training sample duration threshold based on the IOPS of the training sample disk and the IO size of the training sample.

[0090] Step S820: When the training sample IO duration exceeds the training sample duration threshold, it is determined that the training sample disk is in the cache disk acceleration scenario.

[0091] It should be noted that the principle behind determining that the training sample disk is in a cache disk acceleration scenario can be found in [reference needed]. Figure 6 The description of the embodiments shown is omitted here for the sake of simplicity.

[0092] Additionally, refer to Figure 9 In one embodiment, Figure 8 Step S810 in the illustrated embodiment also includes, but is not limited to, the following steps:

[0093] Step S910: Determine all I / O operations of the training sample disk in the cache disk acceleration scenario as candidate I / O operations;

[0094] Based on preset conditions, the training sample IOs are determined from all IOs of the training sample disk, and the IO information of the training sample IOs is determined as the training sample IO information.

[0095] Step S920: Determine the training sample IO from the candidate IO according to preset conditions, and determine the IO information of the training sample IO as the training sample IO information.

[0096] Step S930: Obtain the training sample SMART information corresponding to the training sample IO from the SMART information of the training sample disk;

[0097] Step S940: Preprocess the training sample IO information and training sample SMART information, and generate a prediction training sample set based on the preprocessed training sample IO information and training sample SMART information.

[0098] It should be noted that although most of the I / O in the cache disk acceleration scenario is large-block I / O, not all I / O can be used for model training. Therefore, it is necessary to first identify the I / O in the cache disk acceleration scenario as candidate I / O, and then select the training sample I / O from the candidate I / O according to preset conditions.

[0099] In one embodiment, after obtaining the training sample IO and training sample SMART information, the preprocessing performed on the training samples can check the validity of the training samples and whether the training samples meet the time requirements. It can also add or reduce corresponding operations according to actual needs, such as handling positive and negative sample imbalance, which will not be elaborated here. It is understood that checking the validity of the training samples is mainly used to ensure that the acquired candidate IOs are continuous, avoiding the use of IOs whose acquisition process is interrupted as training samples. For example, if a disk power failure occurs during the execution of an IO, the IO becomes a discontinuous IO with significant deviations in its IO information, making it unusable for training. Therefore, this type of sample can be removed through preprocessing. It is understood that checking whether the training samples meet the time requirements can be determined based on the set training period. For example, if the set maximum training period is four weeks, training samples older than four weeks are removed to ensure the timeliness of the data.

[0100] In one embodiment, after obtaining the training sample IO and training sample SMART information, feature expansion can be performed to increase data discretization. For example, based on the IO information including latency information, status information, and IO time information, the following feature expansions are performed on the training sample IO: several latency segments are pre-set, such as 0 to 32 milliseconds, 32 milliseconds to 64 milliseconds, 64 milliseconds to 128 milliseconds, 128 milliseconds to 512 milliseconds, and >= 512 milliseconds. Based on the latency information in the training sample IO information, the latency segment in which each training sample IO information is located is determined, and the percentage of each latency segment is determined; disk health score evaluation, the proportion of high latency segments is appropriately weighted, so that the proportion of higher latency segments has a higher say, highlighting the health threat caused by high latency; IO error rate, the proportion of IOs with IO error status information in each latency segment to the total IO; the average latency of the top N latency segments sorted from largest to smallest in each latency segment, where the value of N can be selected according to actual needs. It is understandable that SMART information is mostly statistical data. Therefore, we can obtain the rate of change and the absolute value of increase of each statistical data in SMART information to achieve feature expansion, which will not be elaborated on here.

[0101] Additionally, refer to Figure 10 In one embodiment, the training sample IO information further includes state information, latency information, and IO time information, and the preset conditions include at least one of the following:

[0102] Status information refers to error states used to characterize I / O errors;

[0103] The I / O size exceeds the preset I / O size threshold;

[0104] The number of training sample I / Os has been determined to be less than a preset threshold.

[0105] The IO time information conforms to the preset sample acquisition cycle;

[0106] The latency information meets the pre-defined latency distribution range.

[0107] In one embodiment, the preset conditions can be determined by... Figure 2 The IO module shown performs judgments and data filtering; for example, it can determine the statistical list based on the collection period. For ease of explanation, the following will combine... Figure 10 The preset conditions of this embodiment are illustrated by example:

[0108] After the IO module obtains candidate IOs, it first determines whether the status information of the candidate IO is an IO error. If so, it directly adds it to the statistics list. By collecting IOs with IO error status information and using them for training, the prediction model can more accurately predict the probability of disk errors. If the status information of the candidate IO is a correct IO, it determines whether the IO size meets the IO size threshold. Based on the analysis of the above embodiment, the latency characteristics of large IO blocks can be used to predict disk failures. Therefore, this embodiment can only count IOs of a specific size, such as only IOs in the range of 128K to 512K. If the size of the candidate IO is greater than the IO size threshold, the candidate IO is added to the candidate list; otherwise, the number of IOs is judged to avoid insufficient IO collection. When the number of collected candidate IOs exceeds the threshold, sufficient training sample IOs have been collected. At this point, candidate IOs in the statistics list and the candidate list can be designated as training sample IOs, and the current candidate IO is no longer collected. If the number does not exceed the threshold, the current candidate IO needs to be evaluated. For example, by checking the IO time information of the candidate IO, it can be determined that the candidate IO is within the sample collection period. If not, it can be determined that the collection time of the candidate IO does not meet the collection period. Since the candidate IOs are collected in chronological order, it can be determined that the collection of the candidate IO has expired. Therefore, the candidate list can be cleared, and the collection of candidate IOs can be stopped. If the collection period has not expired, the candidate IO is a valid IO, and it is added to the statistics list. The candidate IOs in the statistics list and the candidate list are designated as training sample IOs.

[0109] It is understandable that the pre-set latency distribution range can be set according to actual needs. For example, the latency distribution range in the above embodiment is: 0 to 32 milliseconds, 32 milliseconds to 64 milliseconds, 64 milliseconds to 128 milliseconds, 128 milliseconds to 512 milliseconds, and >= 512 milliseconds. When the latency information of the training sample IO meets the above latency distribution range, it can be further determined as a usable training sample IO.

[0110] Additionally, refer to Figure 11 In one embodiment, Figure 7 Step S720 in the illustrated embodiment also includes, but is not limited to, the following steps:

[0111] Step S1110: Obtain the pre-set training period and determine the periodic sample set corresponding to the training period based on the IO time information of the training sample IO.

[0112] Step S1120: Train the prediction model based on the periodic sample set.

[0113] In one embodiment, the training period can be selected according to actual needs. For example, based on the current time, the training samples from the first week, the second week, and the fourth week can be obtained as the periodic sample set, so that the prediction model can predict disk failure according to different prediction periods.

[0114] It is understandable that after determining the training period, when performing disk failure prediction, the prediction dataset can be collected according to the same period to obtain the prediction results within the corresponding period.

[0115] In addition, in one embodiment, the SMART information of the training samples includes at least one of the following:

[0116] Total number of start-stop cycles;

[0117] Total number of load and uninstall counts;

[0118] Bad habits of growth;

[0119] Non-media error count;

[0120] The number of irreparable errors.

[0121] It should be noted that the selection of SMART information for training samples can refer to the selection principle of SMART information in the disk failure prediction method mentioned above, and will not be repeated here for the sake of simplicity.

[0122] Additionally, refer to Figure 12 In one embodiment, Figure 7 Step S720 in the illustrated embodiment also includes, but is not limited to, the following steps:

[0123] Step S1210: Divide the training sample set and the test sample set from the predicted training sample set according to a preset ratio;

[0124] Step S1220: Train the prediction model based on the training sample set, and perform validation testing on the trained prediction model based on the test sample set.

[0125] In one embodiment, the preset ratio can be any value and can be adjusted according to actual needs, for example, the training sample set and the test sample set can be split in a ratio of 8:2.

[0126] It should be noted that the feature expansion operation in the above embodiments can be performed before or after segmenting the prediction training sample set; this embodiment does not impose any limitations on this.

[0127] It should be noted that when validating the prediction model using a test sample set, common test metrics can be used and thresholds can be set for judgment, such as False Discovery Rate (FDR) and False Accept Rate (FAR). The specific threshold setting criteria can be adjusted according to actual needs, and will not be limited here.

[0128] Additionally, refer to Figure 13 An embodiment of the present invention also provides an electronic device 1300, which includes a memory 1310, a processor 1320, and a computer program stored in the memory 1310 and executable on the processor 1320.

[0129] The processor 1320 and the memory 1310 can be connected via a bus or other means.

[0130] The non-transient software program and instructions required to implement the disk failure prediction method of the above embodiments are stored in the memory 1310. When executed by the processor 1320, the disk failure prediction method applied to the electronic device 1300 in the above embodiments is executed, for example, the disk failure prediction method described above is executed. Figure 1 Method steps S110 to S120, Figure 3 Method steps S310 to S330, Figure 4 Method steps S410 to S420, Figure 5 Method steps S510 to S520, Figure 6 Method steps S610 to S620, Figure 7 Method steps S710 to S720, Figure 8 Method steps S810 to S820, Figure 9 Method steps S910 to S940, Figure 11 Method steps S1110 to S1120, Figure 12 The method steps S1210 to S1220.

[0131] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0132] Furthermore, one embodiment of the present invention provides a computer-readable storage medium storing computer-executable instructions that are executed by a processor or controller, for example, by a processor in the above-described electronic device embodiment, causing the processor to execute the disk failure prediction method for electronic devices described above, for example, performing the above-described... Figure 1 Method steps S110 to S120, Figure 3 Method steps S310 to S330, Figure 4 Method steps S410 to S420, Figure 5 Method steps S510 to S520, Figure 6 Method steps S610 to S620, Figure 7 Method steps S710 to S720, Figure 8 Method steps S810 to S820, Figure 9 Method steps S910 to S940, Figure 11 Method steps S1110 to S1120, Figure 12 The method steps S1210 to S1220 are described above. Those skilled in the art will understand that all or some of the steps and systems disclosed in the above-disclosed methods can be implemented as software, firmware, hardware, or suitable combinations thereof. Some or all physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0133] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of the present invention.

Claims

1. A method for predicting failure of a disk, comprising: obtaining a prediction data set of a disk to be predicted, the prediction data set comprising IO information of a prediction sample IO and SMART information corresponding to the prediction sample IO, wherein the prediction data set is collected in a cache disk acceleration scenario of the disk to be predicted, the IO information comprises an IO duration and an IO size, and the cache disk acceleration scenario indicates that the IO duration is greater than a minimum allowed time required for a corresponding disk to process the IO size; inputting the prediction data set into a pre-trained prediction model to obtain a prediction result of the disk to be predicted.

2. The method of claim 1, wherein, The IO information further comprises IO time information, and the inputting the prediction data set into the pre-trained prediction model to obtain the prediction result of the disk to be predicted comprises: determining a prediction period, determining a period data set from the prediction data set according to the prediction period and the IO time information; obtaining a period failure probability of the disk to be predicted in the prediction period according to the period data set and the prediction model; and determining the prediction result of the disk to be predicted according to the period failure probability.

3. The method of claim 2, wherein, The determining the prediction result of the disk to be predicted according to the period failure probability comprises: when the period failure probability is greater than a preset probability threshold corresponding to the prediction period, determining the prediction result as high risk; and when the period failure probability is less than or equal to the probability threshold, determining the prediction result as low risk.

4. The method of claim 3, wherein, The determining the prediction result as high risk when the period failure probability is greater than the probability threshold comprises: when the period failure probability is greater than the probability threshold, determining a prediction period corresponding to the period failure probability as a high risk period; and when a number of times that the prediction period is determined as the high risk period is greater than a preset alarm number threshold, determining the prediction result as high risk.

5. The method of claim 1, wherein, The cache disk acceleration scenario of the disk to be predicted is determined by the following steps: obtaining IOPS of the disk to be predicted, determining a duration threshold according to the IOPS of the disk to be predicted and the IO size; and when the IO duration is greater than the duration threshold, determining that the disk to be predicted is in the cache disk acceleration scenario.

6. The method of claim 1, wherein, The SMART information at least comprises one of: cumulative start-stop times; cumulative load-unload times; growing bad track number; non-medium error count; unrepairable error number. 7.A method for training a prediction model, comprising: obtaining a prediction training sample set of a training sample disk, the prediction training sample set comprising training sample IO information of a training sample IO and training sample SMART information corresponding to the training sample IO, wherein the prediction training sample set is collected in a cache disk acceleration scenario of the training sample disk, the training sample IO information comprises an IO duration and an IO size, and the cache disk acceleration scenario indicates that the IO duration is greater than a minimum allowed time required for a corresponding training sample disk to process the IO size; and training the prediction model according to the prediction training sample set.

8. The method of claim 7, wherein, The training sample IO information includes a training sample IO duration and a training sample IO size, and the cache disk acceleration scenario of the training sample disk is determined by the following steps: An IOPS of the training sample disk is obtained, and a training sample duration threshold is determined according to the IOPS of the training sample disk and the training sample IO size; When the training sample IO duration is greater than the training sample duration threshold, it is determined that the training sample disk is in the cache disk acceleration scenario.

9. The method of claim 8, wherein, The obtained prediction training sample set of the training sample disk includes: All IOs of the training sample disk in the cache disk acceleration scenario are determined as candidate IOs; The training sample IO is determined from the candidate IOs according to a preset condition, and IO information of the training sample IO is determined as training sample IO information; Training sample SMART information corresponding to the training sample IO is obtained from SMART information of the training sample disk; The training sample IO information and the training sample SMART information are preprocessed, and a prediction training sample set is generated according to the preprocessed training sample IO information and the training sample SMART information.

10. The method of claim 9, wherein, The training sample IO information further includes state information, latency information and IO time information, and the preset condition at least includes one of the following: The state information is an error state for representing IO errors; The IO size is greater than a preset IO size threshold; The number of currently determined training sample IOs is less than a preset number threshold; The IO time information meets a preset sample collection period; The latency information meets a preset latency distribution interval.

11. The method of claim 10, wherein, The training of the prediction model according to the prediction training sample set further includes: A preset training period is obtained, and a period sample set corresponding to the training period is determined according to the IO time information of the training sample IO; The prediction model is trained according to the period sample set.

12. The method according to any one of claims 7 to 9, characterized in that, The training sample SMART information at least includes one of the following: Cumulative start-stop times; Cumulative load-unload times; Growth bad track number; Non-medium error count; Number of unrepairable errors.

13. The method of claim 8, wherein, The training of the prediction model according to the prediction training sample set further includes: A training sample set and a test sample set are segmented from the prediction training sample set according to a preset ratio; The prediction model is trained according to the training sample set, and the trained prediction model is verified and tested according to the test sample set.

14. An electronic device comprising: A memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the disk failure prediction method of any one of claims 1 to 6, or the prediction model training method of any one of claims 7 to 13.

15. A computer readable storage medium storing computer executable instructions for executing the disk failure prediction method of any one of claims 1 to 6, or the prediction model training method of any one of claims 7 to 13.

Images