RAID failure probability calculation device, RAID failure probability calculation method, and RAID failure probability calculation program

Abstract

To provide a RAID failure probability calculation device that calculates failure probability of a RAID group composed of a plurality of hard disk drives.SOLUTION: A RAID failure probability calculation device includes: an HDD error information collection unit that collects an error on hard disk drives that constitute a RAID group; a hard disk drive failure probability calculation unit that calculates, on the basis of the hard disk drive error collected by a hard disk drive information processing unit, hard disk drive failure probability; a RAID configuration information record table that stores RAID configuration information for the RAID group configured with the hard disk drives; a RAID failure probability calculation unit that calculates, on the basis of the hard disk drive failure probability calculated by the hard disk drive failure probability calculation unit and the RAID configuration information obtained from the RAID configuration information record table, the failure probability of the RAID group; and an output unit that outputs failure probability of the RAID group.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a RAID failure probability calculation device, a RAID failure probability calculation method, and a RAID failure probability calculation program.

Background Art

[0002] For the stable operation of an IT system, it is necessary to perform preventive maintenance that detects signs of failure before the devices constituting the system fail and takes actions such as replacement as needed. By taking action before a failure occurs, it is possible to avoid the unexpected stoppage of the system due to a sudden failure and ensure availability.

[0003] In addition, in a HDD (Hard Disk Drive), which is one of the devices constituting an IT system, RAID (Redundant Arrays of Inexpensive Disks), a technology that enhances fault tolerance by redundantly recording data on two or more HDDs when handling important data, is used.

[0004] A combination of two or more HDDs in which data is redundantly stored by RAID is called a RAID group. Also, RAID has levels such as RAID0, 1, 5, 6, etc., depending on the method of dividing and arranging data to each HDD constituting the RAID group and the redundancy.

[0005] For example, in RAID5, data can be restored even if one of the HDDs constituting the RAID group fails. A situation where multiple HDDs among the HDDs constituting the RAID group fail simultaneously is called a multiple failure. When a multiple failure occurs due to a number of HDDs exceeding the number of failures allowed at the level of the RAID group, the data cannot be recovered and is lost.

[0006] In preventive maintenance of HDDs, in order to prevent data loss due to multiple failures, it is necessary to monitor not only the signs of failure of each HDD but also at the RAID group level.

[0007] In contrast, the technology described in Patent Document 1 collects status information for each HDD under monitoring, compares it with status information during normal operation, and applies a predefined threshold to the resulting deviation to diagnose the degradation state in multiple stages. Furthermore, it diagnoses the degree of degradation at the RAID group level in multiple stages based on the combination of degradation states of the HDDs constituting the RAID group, according to a predefined correspondence rule. [Prior art documents] [Patent Documents]

[0008] [Patent Document 1] Japanese Patent Publication No. 2011-180673 [Overview of the Initiative] [Problems that the invention aims to solve]

[0009] Predictive detection of failures requiring parts replacement necessitates setting appropriate detection conditions. If the conditions are too lenient and an excessive number of issues are detected, the number of parts that need replacing will also be excessive, increasing replacement costs.

[0010] Furthermore, system availability decreases during replacement work, and there is a risk of increased load on other HDDs, potentially leading to failures. On the other hand, if the conditions are too strict, warning signs that should be detected may be missed, leading to an increase in failures, data loss, and decreased system availability. Therefore, it is necessary to fine-tune the detection conditions to find an appropriate balance between replacement costs and availability for each IT system environment.

[0011] Patent Document 1 diagnoses the degree of degradation of each HDD by calculating the degree of discrepancy between the operating status information under normal conditions and the operating status information at the time of diagnosis, and applying a threshold to the degree of discrepancy. However, it is difficult to intuitively understand the risk of failure based on the degree of degradation diagnosed by the degree of discrepancy.

[0012] Furthermore, while the threshold applied to the degree of deviation should be adjusted depending on the application environment, changing the threshold alters the diagnostic results for each HDD and each RAID group, making it unsuitable for fine-tuning to balance availability and cost.

[0013] Furthermore, HDDs can experience sudden and unexpected failures. The technology described in Patent Document 1 does not take into account the possibility of sudden failures in its diagnostic results.

[0014] This invention has been made in view of these circumstances, and aims to provide an HDD, a RAID failure probability calculation device for calculating the failure probability of a RAID group composed of HDDs, a RAID failure probability calculation method, and a RAID failure probability calculation program. [Means for solving the problem]

[0015] One aspect of the present invention that solves the above problem is to collect errors in HDDs constituting a RAID group. The HDD error information collection unit collects, The aforementioned HDD Error Information Collection Department A hard disk drive failure probability calculation unit that calculates the probability of hard disk drive failure based on hard disk drive errors collected by the system; a RAID configuration information recording table that stores RAID configuration information for RAID groups composed of hard disk drives; and a RAID failure probability calculation unit that calculates the probability of failure for RAID groups based on the hard disk drive failure probability calculated by the system and the RAID configuration information recording table. The aforementioned This is a RAID failure probability calculation device equipped with an output unit that outputs the failure probability of a RAID group. [Effects of the Invention]

[0016] According to the present invention, the risk of a RAID system composed of multiple HDDs can be calculated using the probability of failure, and by setting a threshold for the probability of failure, it becomes possible to notify the system of the possibility of RAID system failure. [Brief explanation of the drawing]

[0017] [Figure 1] An example of a block diagram showing a configuration example of a RAID failure probability calculation system according to an embodiment of the present invention. [Figure 2] An example of a flowchart for explaining an outline of RAID failure probability calculation processing according to an embodiment of the present invention. [Figure 3] An example of an HDD failure probability information table according to an embodiment of the present invention. [Figure 4] An example of a RAID reconstruction time table according to an embodiment of the present invention. [Figure 5] An example of an HDD information recording table according to an embodiment of the present invention. [Figure 6] An example of a RAID configuration information recording table according to an embodiment of the present invention. [Figure 7] An example of a flowchart for explaining HDD failure probability calculation processing according to an embodiment of the present invention. [Figure 8] An example of an HDD failure probability recording table according to an embodiment of the present invention. [Figure 9] An example of a flowchart for explaining RAID reconstruction time calculation processing according to an embodiment of the present invention. [Figure 10] An example of a flowchart for explaining RAID failure probability calculation processing according to an embodiment of the present invention. [Figure 11] An example of a flowchart for explaining HDD simultaneous failure probability calculation processing according to an embodiment of the present invention. [Figure 12] An example of a RAID failure probability recording table according to an embodiment of the present invention. [Figure 13] An example of a RAID failure probability monitoring screen according to an embodiment of the present invention. [Figure 14] An example of a block diagram showing a configuration example of a RAID failure probability calculation system of the first modification example in the present embodiment. [Figure 15] An example of a flowchart for explaining RAID failure probability calculation processing of the first modification example according to an embodiment of the present invention. [Figure 16]This is an example of an HDD information recording table of a first modified example in an embodiment of the present invention. [Figure 17] This is an example of an HDD status information table in an embodiment of the present invention. [Figure 18] This is an example flowchart illustrating the HDD failure probability calculation process in a first modified embodiment of the present invention. [Figure 19] This is an example of a block diagram showing the configuration of a RAID failure probability calculation system in a second modified example of this embodiment. [Figure 20] This is an example flowchart illustrating the RAID failure probability calculation process in a second modified embodiment of the present invention. [Figure 21] This is an example of a past I / O information table in an embodiment of the present invention. [Figure 22] This is an example of a RAID reconstruction time table in an embodiment of the present invention. [Figure 23] This is an example flowchart illustrating the RAID reconstruction time calculation process for a second modified example in an embodiment of the present invention. [Figure 24] This is an example of a predictive I / O information table in an embodiment of the present invention. [Modes for carrying out the invention]

[0018] Embodiments of the present invention will be described in detail below with reference to the drawings.

[0019] Figure 1 is an example of a block diagram showing an example configuration of a RAID failure probability calculation system in an embodiment of the present invention.

[0020] The RAID failure probability calculation system 1 comprises a management computer 100 (RAID failure probability calculation device), a monitored system (storage system) 120, and an input / output device 108.

[0021] The monitored system 120 is a RAID group for which the probability of RAID failure is calculated. The monitored system 120 consists of one or more storage systems (RAID groups) and is configured to include multiple HDDs 131, 132, 141, and 142.

[0022] HDDs 131 and 132 are part of the HDDs that make up RAID group 130 (RAID #1), and HDDs 141 and 142 are part of the HDDs that make up RAID group 140 (RAID #2). In this embodiment, the monitored system 120 consists of a storage system, but any configuration that includes a RAID group consisting of multiple HDDs is acceptable.

[0023] The management computer 100 is an information processing device that calculates the failure probability of 130 and 140 by collecting information (hereinafter referred to as status information) representing the operating status of HDDs 131, 132, 141, and 142 that constitute RAID groups 130 and 140 operating in the monitored system 120.

[0024] The input / output device 108 consists of an output device such as a display and touch panel connected to the management computer 100, and an input device such as a keyboard, mouse, or touch panel. The input / output device 108, for example, receives information input from the user and stores it in the management computer 100, and also outputs information such as the processing results of the management computer 100.

[0025] The management computer may be configured as part of the storage controller that manages the storage system.

[0026] Next, the management computer 100 includes a processing unit 101 such as a CPU (Central Processing Unit), memory 102 such as RAM (Random Access Memory) and ROM (Read Only Memory), and storage devices 103 such as an HDD and SSD (Solid State Drive). Furthermore, the management computer 100 stores the following in its storage device 103: an HDD failure probability calculation unit 104, an HDD information collection unit 105, a RAID reconstruction time calculation unit 106, a RAID failure probability calculation unit 107, an HDD information record table 200, an HDD failure probability information table 300, a RAID reconstruction time table 400, a RAID configuration information record table 500, an HDD failure probability record table 600, and a RAID failure probability record table 700.

[0027] The HDD failure probability calculation unit 104 is a unit that calculates the failure probability of HDDs 131, 132, 141, and 142 that constitute RAID groups 130 and 140 included in the monitored system 120.

[0028] The HDD information collection unit 105 acquires from the monitored system 120 RAID configuration information, which is the correspondence between RAID groups and HDDs in the monitored system 120, the RAID level of each RAID group, attribute information of each HDD such as the model, capacity, and I / O amount of each HDD, and status information of each HDD.

[0029] The RAID reconstruction time calculation unit 106 determines the time required for RAID reconstruction when an HDD is replaced for maintenance, based on information such as the model, capacity, and I / O volume of the HDDs that make up the RAID group.

[0030] The RAID failure probability calculation unit 107 calculates the failure probability of the RAID group included in the monitored system 120 from the failure probability of each HDD obtained by the HDD failure probability calculation unit 104, the RAID configuration information obtained by the HDD information collection unit 105, and the RAID reconstruction time obtained by the RAID reconstruction time calculation unit 106.

[0031] The HDD error information collection unit 109 collects write errors, read errors, timeout errors, and other errors from the HDDs that make up the monitored system.

[0032] The HDD information recording table 200 is a table that stores attribute information of each HDD, such as the model, capacity, and I / O amount of each HDD, as well as status information of each HDD, which is acquired by the HDD information collection unit 105.

[0033] The HDD failure probability information table 300 is a table that stores HDD errors associated with failures and failure probabilities calculated based on whether or not the HDD in which the error was confirmed actually failed afterward. It is defined based on past failure cases.

[0034] The RAID Reconfiguration Time Table 400 is a table that stores the RAID reconfiguration time calculated from the HDD model, capacity, I / O volume, and RAID level of the RAID group being configured, and is defined based on past maintenance and replacement cases.

[0035] The RAID configuration information record table 500 is a table that stores the RAID level of a RAID group and the HDDs that make up the RAID group.

[0036] The HDD failure probability record table 600 is a table that stores the failure probability of HDDs and the RAID reconstruction time for RAID groups.

[0037] The RAID failure probability record table 700 is a table that stores the failure probability of each RAID group obtained by the RAID failure probability calculation unit 107.

[0038] Figure 2 is an example flowchart illustrating the overview of the RAID failure probability calculation process in an embodiment of the present invention. The RAID failure probability calculation process is performed, for example, when a predetermined instruction to calculate the RAID failure probability is input from a user to the management computer 100, or at a predetermined timing (for example, a predetermined time, day of the week, or predetermined time interval).

[0039] First, as shown in Figure 2, the RAID failure probability calculation process pre-defines and maintains an HDD failure probability information table 300 and a RAID reconstruction time table 400 (S10, S20).

[0040] Next, the HDD information collection unit 105 collects HDD information and HDD status information and records them in the HDD information recording table 200 (S30).

[0041] Next, the HDD information collection unit 105 collects RAID configuration information and the RAID level of each RAID group and stores them in the RAID configuration information record table 500 (S40).

[0042] Furthermore, the processes in S30 and S40 may be executed repeatedly at predetermined timings (for example, at predetermined times, days of the week, or time intervals) independently of the RAID failure probability calculation process.

[0043] Next, the HDD failure probability calculation unit 104 performs an HDD failure probability calculation process (S100) to calculate the failure probability from the HDD status information of each HDD included in the monitored system 120. Next, the RAID reconstruction time calculation unit 106 performs a RAID reconstruction time calculation process (S200) which outputs the RAID reconstruction time from the HDD information of each HDD included in the monitored system 120 and the RAID level of each RAID.

[0044] Finally, the RAID failure probability calculation unit 107 performs a RAID failure probability calculation process (S300) to calculate the RAID failure probability of each RAID based on the HDD failure probability, RAID configuration information, and RAID reconstruction time.

[0045] Figure 3 shows an example of an HDD failure probability information table 300 in an embodiment of the present invention. The HDD failure probability information table 300 includes HDD status information 301 and failure probability 302. The HDD status information 301 is information that stores the error items of errors that occur in the HDD and the number of occurrences for each item.

[0046] The error count stores, for example, the cumulative number of errors since the start of operation, or the number of errors over a certain period (e.g., one day or one week). In addition, the failure probability (within one week) 302a and failure probability (within two weeks) 302b shown in the figure store the probability that an HDD that has experienced an error as shown in the HDD status information 301 will fail within one week and two weeks thereafter.

[0047] Failure probability is defined based on past failure cases. For example, if there have been 10 past cases where Error 1 occurred twice within a certain period, and 6 of those failed within a week, then the HDD failure probability information table 300 will record Error 1 (301a) twice in the HDD status information 301, other status information (301b, ...) zero times, and failure probability (within one week) 302a as 60%.

[0048] In this embodiment, the number of errors observed on the HDD is used as the HDD status information representing HDD errors, but parameters representing the operation of the HDD or information on each inspection item of the HDD self-diagnostic function (SMART) may also be used.

[0049] Furthermore, HDDs can fail suddenly without any error output or unusual values ​​in parameters representing HDD operation. Therefore, the HDD failure probability information table 300 also stores the probability of sudden HDD failure, which is calculated from past cases of sudden failures, where the HDD status information shows zero errors.

[0050] Since the storage controller may not collect all the test items from the HDD self-diagnostic function (SMART), the storage controller or devices or programs added to the storage controller may collect write errors, read errors, timeout errors, etc., that occur on the HDD, and predict HDD failure based on the number and frequency of these collected errors. By using this information, error information obtained from the HDD can be identified, making it possible to make more accurate predictions.

[0051] Figure 4 shows an example of a RAID reconstruction time table 400 in an embodiment of the present invention. The RAID reconstruction time table 400 stores the RAID reconstruction time 405, which is calculated by aggregating the RAID reconstruction time performed in the past for each HDD model 401, capacity 402, I / O volume 403, and RAID level 404 of the constituent RAID group. The RAID reconstruction time is the value of the RAID reconstruction time required when an HDD is replaced during maintenance. The RAID reconstruction time 405 is defined based on past maintenance replacement cases.

[0052] Figure 5 shows an example of an HDD information recording table in an embodiment of the present invention. The HDD information recording table 200 includes an HDD identifier 201 for each HDD, HDD information 202 in which attribute information for each HDD is set, and HDD status information 203 in which status information for each HDD is set.

[0053] In this example, the HDD information 202 consists of the HDD model 202a, capacity 202b, and I / O amount 202c for each HDD.

[0054] In this example, HDD status information 203 records the number of errors that occurred on the HDD, categorized by error type. The error count can be, for example, the cumulative number of errors since the start of operation, or the number of errors over a certain period (e.g., one day or one week). The error types can be write errors, read errors, timeouts, etc.

[0055] In addition, it is possible to judge the degree of HDD degradation by using the number of replacement tracks used when a track on the HDD becomes unusable.

[0056] Figure 6 shows an example of a RAID configuration information recording table in an embodiment of the present invention. The RAID configuration information recording table 500 includes identifiers 501 for each RAID group included in the monitored system 120, the RAID level 502 for each RAID group, and identifiers 503 for the HDDs constituting each RAID group. The number of HDDs recorded on HDD 503 varies depending on the RAID configuration.

[0057] Figure 7 is an example of a flowchart illustrating the HDD failure probability calculation process (S100) in an embodiment of the present invention.

[0058] First, the HDD failure probability calculation unit 104 obtains HDD status information 203 from the HDD information recording table 200 (S101).

[0059] Next, the HDD failure probability calculation unit 104 refers to the HDD failure probability information table 300 (S102), and by referring to the HDD status information 301 in the HDD failure probability information table 300 and the HDD status information obtained in S101, it calculates the failure probability by obtaining the failure probability for each period of the HDD from the failure probabilities 302a and 302b of the corresponding row (S103).

[0060] Here, the probability of sudden failure, calculated from past cases of sudden failures stored in the HDD failure probability information table 300, can be used to indicate the probability of sudden failure even for HDDs that showed no warning signs in the HDD status information acquired in S101.

[0061] A sudden failure is a type of malfunction where the HDD suddenly dies and becomes unusable, even though no errors are being collected.

[0062] Next, the HDD failure probability calculation unit 104 stores the identifier of each HDD in the failure probability 602 corresponding to the HDD identifier 601 in the HDD failure probability record table 600, and stores the HDD failure probability of each HDD obtained in S103 (S104).

[0063] Figure 8 shows an example of an HDD failure probability record table in an embodiment of the present invention. The HDD failure probability storage table 600 stores an HDD identifier 601 that records the identifier of an HDD included in the monitored system 120, an HDD failure probability 602 for the HDD indicated by 601, and a RAID reconstruction time 603 required when the HDD indicated by 601 is replaced for maintenance. Details of the information stored in the RAID reconstruction time 603 will be described later.

[0064] In this embodiment, an HDD failure probability information table 300 is defined that holds HDD status information and corresponding failure probabilities. Failure probabilities are obtained by comparing this table with the HDD status information included in the monitored system 120. However, as a first modification of this embodiment, a model is trained in which HDD status information is used as an explanatory variable and a value representing the presence or absence of a failure is used as an objective variable. Failure probabilities are then obtained by inputting the HDD status information included in the monitored system 120 into the model. Details of this first modification of this embodiment will be described later.

[0065] Figure 9 is an example flowchart illustrating the RAID reconstruction time calculation process (S200) in an embodiment of the present invention.

[0066] First, the RAID reconstruction time calculation unit 106 obtains the HDD information 202 for each HDD from the HDD information recording table 200 (S201).

[0067] Next, the RAID reconstruction time calculation unit 106 obtains information on the RAID level 502 of the RAID group to which each HDD belongs from the RAID configuration information recording table 500 (S202).

[0068] Next, the RAID reconstruction time calculation unit 106 refers to the RAID reconstruction time table 400 and, using the HDD information and RAID group information for each HDD obtained in S201 and S202, refers to the information 401 to 404 in the RAID reconstruction time table 400 to obtain the RAID reconstruction time 405 for the corresponding row (S203).

[0069] Next, the RAID reconstruction time calculation unit 106 stores the acquired RAID reconstruction time in the RAID reconstruction time 603 of the HDD failure probability storage table 600 (S204).

[0070] In this embodiment, a RAID reconstruction time table 400 is defined that holds HDD information and the corresponding RAID reconstruction time. The RAID reconstruction time is determined by comparing this table with the HDD information of the HDDs included in the monitored system 120. However, as a second modification of this embodiment, a model may be trained with HDD information and I / O transitions as explanatory variables and RAID reconstruction time as the objective variable. The RAID reconstruction time may then be obtained by inputting the HDD information of the HDDs included in the monitored system 120 and the predicted I / O transition values ​​into the model. Details of this second modification of this embodiment will be described later.

[0071] Figure 10 is an example flowchart illustrating the RAID failure probability calculation process (S300) in an embodiment of the present invention.

[0072] First, the RAID failure probability calculation unit 107 obtains the HDD failure probability 602 for each HDD and the RAID reconstruction time 603 from the HDD failure probability record table 600 (S301).

[0073] Next, the RAID failure probability calculation unit 107 obtains the correspondence between the RAID group 501 and the HDD 503 that constitute the RAID from the RAID configuration information record table 500 (S302).

[0074] Next, the RAID failure probability calculation unit 107 uses the HDD failure probability 602 obtained in S301 and the RAID configuration information obtained in S302 to perform an HDD simultaneous failure probability calculation process (S310) which calculates the probability of simultaneous (multiple) failures in which the HDDs constituting the RAID fail within the same period.

[0075] The probability of simultaneous HDD failure calculated by S310 calculates the probability that more HDDs will fail during the same period than the number of failures allowed by the RAID level of the RAID group. For example, if the RAID level is RAID 5, RAID 5 allows for the failure of up to one drive, so it calculates the probability that two or more HDDs will fail during the same period.

[0076] Next, a multiplier is accumulated so that the probability of simultaneous HDD failure calculated in S310 increases according to the RAID reconstruction time.

[0077] This process reflects the risk of other HDDs failing during the RAID reconstruction process after one HDD in a RAID group is replaced for maintenance purposes, in addition to the RAID failure probability. For example, the RAID reconstruction time associated with the HDD with the highest probability of failure among the HDDs in the RAID group is increased by a higher multiplier the longer the time. If the multiplier for RAID reconstruction time is increased by 0.01 per hour, a RAID group including an HDD with a RAID reconstruction time of 500 minutes (8 hours and 20 minutes) will have 1.08 added to the simultaneous HDD failure probability calculated in S310 to determine the RAID failure probability (S303).

[0078] Next, the RAID failure probability obtained by multiplying the RAID failure probability of the RAID group calculated in S303 by the failure probability considering the RAID reconstruction time for each RAID group is stored in the failure probability 702 of the RAID failure probability record table 700 corresponding to RAID group 701 (S304).

[0079] Finally, the RAID failure probability calculation unit 107 obtains the HDD failure probability 602 for each HDD from the HDD failure probability record table 600 and the RAID failure probability 702 for each RAID group from the RAID failure probability storage table 700, and displays them on the RAID failure probability monitoring screen (S305).

[0080] Figure 11 is an example flowchart illustrating the HDD simultaneous failure probability calculation process (S310) in an embodiment of the present invention. This embodiment describes the flow for calculating the HDD simultaneous failure probability for a RAID level that allows for up to one failure.

[0081] First, the RAID failure probability calculation unit 107 calculates the probability X that no HDDs fail by summing up the probabilities that each HDD constituting the RAID group does not fail (S311). In a RAID group consisting of N HDDs, if the HDD failure probabilities of each HDD are x1, x2, ..., xn, then the probabilities that each HDD does not fail are (1-x1), (1-x2), ..., (1-xn), respectively, and the probability X that no HDDs fail is given by (1-x1)(1-x2) ..., (1-xn).

[0082] Next, the RAID failure probability calculation unit 107 defines the variable p and assigns it the value 1 (S312).

[0083] Next, we calculate the probability Xp that only the p-th HDD among the n HDDs that make up the RAID group will fail (S313). Xp is calculated by replacing the probability (1-xp) that the p-th HDD does not fail in the X=(1-x1)(1-x2)...(1-xn) calculated in S312 with the probability xp that the p-th HDD does fail, so it can be calculated as Xp=X×xp / (1-xp).

[0084] Next, the RAID failure probability calculation unit 107 compares p and n (S314).

[0085] If p=n is not the case (S314:NO), the RAID failure probability calculation unit 107 increments p by substituting p+1 and executes S313.

[0086] If p=n (S314:YES), the RAID failure probability calculation unit 107 calculates the HDD simultaneous failure probability, which is the probability that two or more HDDs will fail during the same period (S316). The HDD simultaneous failure probability is calculated by subtracting the sum of X calculated in S311 and X1, X2, ..., Xn calculated in S313 from 1 (1-(X+X1+X2+...+Xn)).

[0087] In this embodiment, the probability of a number of HDDs failing during the same period that exceeds the number of failures allowed by the RAID level of the RAID group is defined as the simultaneous HDD failure probability. However, the simultaneous HDD failure probability can be derived from the HDD failure probability of each HDD constituting the RAID group.

[0088] For example, a simple average of HDD failure probabilities or a weighted average would suffice. A weighted average, for instance, in the case of RAID 5 where more than two failures are unacceptable, would be a weighted average that assigns weight to the top two HDD failure probabilities among the HDDs making up the RAID group.

[0089] The probability of simultaneous HDD failure is calculated from the HDD failure probability of each HDD that makes up the RAID group. Since the HDD failure probability of each HDD includes the probability of sudden failure, the probability of simultaneous HDD failure also takes into account the risk of sudden failure.

[0090] Figure 12 shows an example of a RAID failure probability record table (700) in an embodiment of the present invention. The RAID failure probability record table 700 stores the names of RAID groups configured within the monitored system 120, a RAID group 701, and the failure probability 702 of the RAID group indicated by RAID group 701.

[0091] Figure 13 shows an example of a RAID failure probability monitoring screen in an embodiment of the present invention. The RAID failure probability monitoring screen 800 includes a results display unit 810 and a management unit 820.

[0092] The results display unit 810 displays the HDD failure probability calculation results 811, which show the HDD failure probability for each HDD calculated by the HDD failure probability calculation unit 104 and stored in the HDD failure probability record table 600, and the RAID failure probability for each RAID group calculated by the RAID failure probability calculation unit 107 and stored in the HDD failure probability record table 700. The management unit 820 is used by the user to determine whether maintenance replacement is necessary based on the calculated RAID failure probability.

[0093] The failure probability (1 week) threshold setting unit 821 and the failure probability (2 weeks) threshold setting unit 822 accept RAID failure probability thresholds from the user for the failure probability within one week and the failure probability within two weeks, respectively. If there are RAID groups with a RAID failure probability that exceeds the accepted threshold, the threshold setting result 823 displays the RAID failure probability of the corresponding RAID group and the HDD failure probability of the HDDs constituting the RAID group.

[0094] In this example, the HDD with the highest probability of failure among the HDDs constituting the RAID group that exceeds the threshold is highlighted by color, and the HDD that is recommended for replacement is shown. In this embodiment, the RAID failure probability of a RAID group that exceeds the set threshold is displayed, but it is also possible to display the rows of the RAID groups that exceed the threshold by color in the failure probability result 811.

[0095] Furthermore, the management unit 820 may display the necessary actions based on the calculated RAID failure probability in the recommended action display unit 824. The recommended actions displayed in the recommended action display unit 824 are, for example, the RAID groups and HDDs that require maintenance replacement, as displayed in the threshold setting result 823. In addition, if the number of maintenance replacements required is large, the system may recommend adjusting the threshold to balance failure risk and replacement costs, and may suggest solutions such as migrating to the cloud if replacement costs become high.

[0096] In this embodiment, the HDD failure probability and RAID failure probability were explained using failure probabilities within one week and within two weeks, but the time divisions are not limited to two; there may be one or three or more, and it is also possible to perform the test with different time periods such as within one day and within three days, and the settings may be set according to the level and purpose of maintenance.

[0097] If an HDD is replaced during maintenance, the HDD-compatible replacement switch on the storage system's chassis will be turned ON. By receiving the signal from the replacement switch and resetting the error record of the HDD, it is possible to avoid misdiagnosis due to incorrect error information.

[0098] Furthermore, if the RAID failure probability calculation system cannot read the HDD model, specifications, etc., displaying a message prompting the user to manually input this information will help avoid input errors and enable more accurate diagnosis.

[0099] [First modified example of this embodiment] As a first modification of this embodiment, a model is trained in which the state information of the HDD is used as an explanatory variable and a value representing the presence or absence of a failure is used as an objective variable. By inputting the state information of the HDDs included in the monitored target 120 into the model instead of S103 of the HDD failure probability calculation process S100 to obtain the failure probability, it becomes possible to perform an accurate diagnosis that is suitable for the target storage system.

[0100] Figure 14 is an example block diagram showing the configuration of a first modified RAID failure probability calculation system in this embodiment. The HDD information recording table 200 in Figure 1 is replaced with the HDD information table 200' of the first modified version, and the HDD failure probability information table 300 is replaced with the HDD status information table 300'. Details of the HDD information table 200' and HDD status information table 300' of the first modified version will be described later.

[0101] Figure 15 is an example flowchart illustrating the RAID failure probability calculation process of a first modified embodiment of the present invention. S10' enclosed by the dashed line in Figure 15 corresponds to S10 in Figure 3, and S100' enclosed by the dashed line corresponds to S100 in Figure 3. Note that S20, S30, S40, S200, and S300 are the same processes as in the embodiment shown in Figure 3, so their explanation is omitted, and only S10' and S100' will be explained.

[0102] In the first modified example, the RAID failure probability calculation process pre-stores an HDD failure information table and creates an HDD failure probability prediction model trained on the data in the HDD failure information table (S10').

[0103] Figure 16 shows an example of the HDD information recording table 200' of a first modified example in an embodiment of the present invention.

[0104] The HDD information recording table 200' of the first modified example has an HDD identifier 201 for each HDD, HDD information 202' (202'a, 202'b, 202'c) which contains attribute information for each HDD, and HDD status information 203' (203'a, 203'b, ...) which contains status information for each HDD. The HDD information recording table 200' of the first modified example has the HDD information recording table 200 shown in Figure 5 with the addition of lot information for each HDD, and is the same as the HDD information recording table 200 except for the lot information 202'd in HDD information 202'.

[0105] Lot information 202'd contains the lot number of the HDD or similar information. The information that can be used to identify an HDD varies depending on the manufacturer and model, but for HDDs that retain a lot number, the lot number is stored in lot information 202'd. For HDDs that do not retain a lot number, the serial number, manufacturing plant number, manufacturing date, or a combination of these information is stored in lot information 202'd. This information allows it to be determined whether or not the HDDs belong to the same lot, or whether or not they were manufactured around the same time.

[0106] Figure 17 shows an example of the HDD status information table 300' in an embodiment of the present invention.

[0107] The HDD status information table 300' is created based on past failure cases. The HDD status information table 300' contains HDD status information 301' (301'a, 301'b), HDD information 302' (302'a, 302'b), and failure information 303' (303'a, 303'b). The HDD status information 301' is information that holds the error items of errors that occur in the HDD and the number of errors for each item.

[0108] The error count stores, for example, the cumulative number of errors since the start of operation, or the number of errors over a certain period (e.g., one day or one week). HDD information 302' stores the HDD model 302'a and lot information 302'b of the HDD that showed the error indicated in HDD status information 301'. Failure information 303' records whether the HDD that showed the error indicated in HDD status information 301' failed within one week and two weeks thereafter. In the example in Figure 17, 1 is recorded if it failed, and 0 if it did not fail.

[0109] The HDD failure probability calculation process uses the HDD status information 301' and HDD information 302' from the HDD status information table 300' as explanatory variables, and the failure information 303'a and 303'b as target variables to train a model. The model only needs to take the HDD information and HDD status information as input and output the failure probability.

[0110] For example, a probability calculation model using logistic regression or a method that calculates the failure rate by determining the proportion of models that output 1 in ensemble learning, such as random forests, would also be acceptable. By using HDD state information 301' as an explanatory variable, the model can learn the relationship between HDD errors, represented by the error count, and HDD failures.

[0111] Furthermore, by using HDD model 302'a as an explanatory variable, it is expected that the failure trends for each HDD model can be learned. Additionally, by using lot information 302'd as an explanatory variable, it is expected that failure trends dependent on the HDD lot, manufacturing period, and manufacturing environment can be learned. There are cases where the same failures tend to occur at the same time in HDDs manufactured from the same lot, at the same time, and in the same environment (factory, etc.).

[0112] Figure 18 is an example flowchart illustrating the HDD failure probability calculation process S100' of a first modified embodiment of the present invention.

[0113] Steps S110 and S111, enclosed by dashed lines in Figure 18, correspond to steps S102 and S103 in Figure 7. Steps S101 and S104 are the same processes as those in the embodiment shown in Figure 7, so their explanation is omitted. Steps S110 and S111 will be explained.

[0114] In S110, the HDD failure probability calculation unit obtains the HDD model 202'a and lot information 202'd from the HDD information recording table 200'.

[0115] Next, in S111, the HDD failure probability calculation unit inputs the HDD information and HDD status information of each HDD in the monitored system, acquired in S101, into the HDD failure probability prediction model created in S10', and obtains the failure probability of each HDD that is output.

[0116] Since the subsequent processing is the same as in this embodiment, this concludes the description of the first modified example of this embodiment.

[0117] The first modification involves inputting error information from the monitored HDD into a model trained on past HDD error data and failure status information, and then calculating the probability of HDD failure. By using a machine learning model, the calculation of failure probability and the creation of tables can be replaced by the model training process. Therefore, when there are many types of errors, the conditions for how errors occur corresponding to the failure probability also become enormous, making the creation of a model effective.

[0118] Furthermore, since it is not necessary to recalculate the failure probability recorded in the table to reflect new information and update the information, or to add information when a new type of error is confirmed, updating the HDD failure probability calculation process to reflect new failure information is easy.

[0119] Furthermore, if an error pattern that has never occurred before is detected, it is possible to predict the probability of HDD failure. An unprecedented pattern would be, for example, if two errors, Error A and Error B, have occurred individually in the past but never simultaneously, and then both errors occur at the same time on the monitored HDD.

[0120] [Second modified example of this embodiment] As a second modification of this embodiment, a model may be trained with HDD information and I / O transitions as explanatory variables and RAID reconstruction time as the dependent variable, and the HDD information and predicted I / O transition values ​​of the HDDs included in the monitored target 120 are input to the model to obtain the RAID reconstruction time.

[0121] Figure 19 is an example of a block diagram showing the configuration of a second modified RAID failure probability calculation system in this embodiment.

[0122] This configuration replaces the RAID reconstruction time table 400 in Figure 1 with RAID reconstruction time table 400', and further adds the past I / O information table 800 and the predicted I / O information table 900. Details of the RAID reconstruction time table 400', past I / O information table 800, and predicted I / O information table 900 will be described later.

[0123] Figure 20 is an example flowchart illustrating the RAID failure probability calculation process for a second modified example in an embodiment of the present invention.

[0124] In Figure 20, steps S21 and S22, enclosed by dashed lines, correspond to step S20 in Figure 3, and step S200, enclosed by dashed lines, corresponds to step S200' in Figure 3. Note that steps S10, S30, S40, S100, and S300 are the same processes as in the embodiment shown in Figure 3, so their explanation is omitted, and only steps S21, S22, and S200' will be explained.

[0125] In the second modified example, the RAID failure probability calculation process pre-stores a past I / O information table 800 and creates an I / O volume trend prediction model trained on the data in the past I / O information table 800 (S21).

[0126] Next, in the second modified example, the RAID failure probability calculation process pre-stores a RAID reconstruction time table 400' and creates a RAID reconstruction time prediction model trained on the data in the RAID reconstruction time table 400' (S22).

[0127] Figure 21 shows an example of a past I / O information table 800 in an embodiment of the present invention.

[0128] The past I / O information table 800 contains an HDD identifier 801 and an I / O volume trend table 802. The past I / O volume trend table 802 holds the total I / O volume for each HDD indicated by the HDD identifier 801. Table 8020 records the timestamp for each date and time at arbitrary time intervals (5-minute intervals in the diagram), and table 8021 records the I / O volume (MB) of each HDD observed at each date and time.

[0129] The RAID failure probability calculation process trains a model for each HDD, using the timestamp of the past I / O volume trend 8020 in the past I / O information table 800 as the explanatory variable, and the I / O volume 8021 at each time indicated by the timestamp as the target variable.

[0130] The timestamp information used as an explanatory variable may further include attributes such as year, month, day, day of the week, week number in the month, or whether it is a weekday or public holiday. The model only needs to take a timestamp as input and output a predicted value of the I / O volume at the time indicated by the input. Alternatively, an I / O volume trend prediction model may be created for each HDD, or an I / O volume trend prediction model may be created for each HDD model.

[0131] Figure 22 shows an example of a RAID reconstruction time table 400' in an embodiment of the present invention.

[0132] The RAID Reconfiguration Time Table 400' is created based on past maintenance and replacement cases. The RAID Reconfiguration Time Table 400' is a record of past maintenance and replacements and contains information on the replaced HDD, including the HDD model 401', capacity 402', RAID level 403' of the RAID group it constitutes, I / O volume trend 404', and RAID reconfiguration time 405'. The past I / O volume trend 404' is a record of the I / O volume trend over a certain period from the start of the replacement at arbitrary time intervals (5-minute intervals in the figure). In the example in Figure 22, the past I / O volume trend 404' records the I / O volume trend over 24 hours from the start of the replacement at 5-minute intervals.

[0133] The RAID failure probability calculation process trains a model using the HDD model 401', capacity 402', RAID level 403', and I / O volume trend 404' from the RAID reconstruction time table 400' as explanatory variables, and the RAID reconstruction time 405' as the dependent variable. Note that the HDD model 401' information may be converted to an appropriate variable and used as an explanatory variable in the model.

[0134] In this example, the I / O volume trend 404' uses a sequence of I / O volumes as one of the explanatory variables in the model, but a simpler method could be used instead, such as using the average I / O volume over a certain period, or just the I / O volume immediately before maintenance replacement, as one of the explanatory variables. Alternatively, a RAID reconstruction prediction model could be created by training it with information from the RAID reconstruction time table, or a RAID reconstruction time prediction model could be created for each HDD model.

[0135] Figure 23 is an example flowchart illustrating the RAID reconstruction time calculation process S200' of a second modified example in an embodiment of the present invention.

[0136] The steps S210 to S213 enclosed by the dashed lines in Figure 22 correspond to steps S202 and S203 in Figure 7. Steps S201 and S204 are the same processes as in the embodiment shown in Figure 9, so their explanation is omitted, and steps S210 to S213 will be explained.

[0137] The RAID reconstruction time calculation unit stores information about the period for which the I / O volume is predicted in the predicted I / O information table (S210). The prediction period is a set of timestamps, which are timestamps within a certain period starting from the start time of the RAID failure probability calculation process or a predetermined timing.

[0138] The RAID reconstruction time calculation unit obtains predicted values ​​for the amount of I / O at each time point in the prediction period by inputting the timestamps of the prediction period recorded in 9021 of the prediction I / O information table 900 into the I / O transition prediction model created in S21 (S211).

[0139] Next, the RAID reconstruction time calculation unit stores the predicted I / O volume at each time point in the prediction period, obtained in S211, in 9021 of the prediction I / O information table 900 (S212).

[0140] Next, the RAID reconstruction time calculation unit inputs the HDD model 202a and capacity 202b stored in the HDD information recording table 200, and the predicted I / O amount stored in the predicted I / O information table 900 into the RAID reconstruction time calculation model created in S22, and obtains the output RAID reconstruction time (S213).

[0141] Furthermore, if the information for HDD model 401' is converted into an appropriate variable in S22 and used as an explanatory variable for the model, the information for HDD model 202a entered in S212 will be converted in the same way and entered into the model. Also, if the average value of the I / O volume over a certain period or only the I / O volume immediately before maintenance replacement is used as one of the explanatory variables in S22 instead of the I / O volume, the predicted I / O volume entered in S212 will be processed in the same way and entered into the model.

[0142] Figure 24 shows an example of a predicted I / O information table 900 in an embodiment of the present invention.

[0143] The predicted I / O information table 900 contains an HDD identifier 901 and an I / O volume trend 902. The past I / O volume trend 902 holds the predicted I / O volume for each HDD indicated by the HDD identifier 901. In Figure 24, the timestamp 9020 for the prediction period stores timestamps for 24 hours from the starting point at 5-minute intervals. In addition, 9021 records the predicted value of the I / O volume for each HDD at each time point during the prediction period, as described in S212 below.

[0144] The present invention is not limited to the embodiments described above, and can be implemented using any components without departing from its spirit. The embodiments and modifications described above are merely examples, and the present invention is not limited to these as long as the features of the invention are not impaired. Furthermore, although various embodiments and modifications have been described above, the present invention is not limited to these. Other embodiments conceivable within the scope of the technical idea of ​​the present invention are also included within the scope of the present invention.

[0145] For example, some of the hardware components of each device in this embodiment may be provided in other devices.

[0146] Furthermore, each part of the RAID failure probability calculation device may be installed in another device. The second modification involves training a model that uses HDD information (model, capacity) and I / O trends obtained from past HDD maintenance and replacement work as explanatory variables, and RAID reconstruction time information as the dependent variable. The HDD information (model, capacity) and predicted I / O volume of the monitored HDD are then input, and the RAID reconstruction time is predicted. The predicted I / O volume is a prediction of future I / O volume trends, obtained by another model that has learned past I / O volume trends.

[0147] Instead of using the previous value or the most recent average as the I / O volume, calculating the RAID reconstruction time based on the temporal changes in the I / O volume of the trained model allows for a more accurate RAID reconstruction time that reflects the actual I / O volume. [Explanation of Symbols]

[0148] 1 RAID failure probability calculation system, 100 Management computer, 101 Processing unit, 102 Memory, 103 Storage device, 104 HDD failure probability calculation unit, 105 HDD information collection unit, 106 RAID reconstruction time calculation unit, 107 RAID failure probability calculation unit, 108 Input / output device, 109 HDD error information collection unit, 120 Monitored system, 130, 140 RAID group, 131, 132, 141, 142 HDD, 200 HDD information recording table, 300 HDD failure probability information table, 400 RAID reconstruction time table, 500 RAID configuration information recording table, 600 HDD failure probability recording table, 700 RAID failure probability recording table

Claims

1. An HDD error information collection unit that collects errors from HDDs that make up a RAID group, An HDD failure probability calculation unit calculates the probability of HDD failure based on the HDD errors collected by the HDD error information collection unit, A RAID configuration information record table that stores RAID configuration information for the RAID group composed of the HDDs, A RAID failure probability calculation unit calculates the failure probability of the RAID group based on the HDD failure probability calculated by the HDD failure probability calculation unit and the RAID configuration information obtained from the RAID configuration information record table. A RAID failure probability calculation device comprising an output unit that outputs the failure probability of the aforementioned RAID group.

2. The system includes a RAID reconstruction time table that stores the RAID reconstruction time, which is the time required for RAID reconstruction when the HDDs constituting the RAID group are replaced. The RAID failure probability calculation device according to claim 1, wherein the RAID failure probability calculation unit calculates the failure probability of the RAID group from the HDD failure probability obtained by the HDD failure probability calculation unit, the RAID configuration information stored in the RAID configuration information recording table, and the RAID reconstruction time stored in the RAID reconstruction time table.

3. The RAID failure probability calculation unit includes a storage unit that stores error information in an HDD information recording table that records error information that occurred on the HDD, and an HDD failure probability information table that stores the correspondence between the number of errors that occurred on the HDD and the failure probability. The RAID failure probability calculation device according to claim 2, which calculates the failure probability of the RAID group by referring to the error information in the HDD information recording table and the failure probability corresponding to the number of errors of the HDD, which is determined by the HDD failure probability calculation unit.

4. The RAID failure probability calculation device according to claim 3, wherein the RAID failure probability calculation unit includes a storage unit that stores a model in which the errors of the HDD are input values ​​and the failure probability is output values, and the failure probability is calculated based on the errors of the HDD.

5. The RAID failure probability calculation unit calculates the error stored in the HDD failure probability information table. The RAID failure probability calculation device according to claim 3, which calculates the failure probability of the HDD by accumulating the probabilities of sudden failures in which the HDD fails and stops without occurring in the HDD.

6. The RAID failure probability calculation device according to claim 3, wherein the RAID configuration information stored in the RAID configuration information recording table is stored for each attribute of the HDD based on the record of the time required for RAID reconstruction work when the HDD is replaced.

7. The RAID failure probability calculation unit determines, from the failure probabilities of the HDDs constituting the RAID group, the probability of a multiple failure occurring in which a number of HDDs in the RAID group fail that prevent data recovery during the same period. The RAID failure probability calculation device according to claim 2, which calculates the failure probability of the RAID group obtained from the HDD failure probability, the RAID configuration information, and the RAID reconstruction time, by adding the probability of multiple failures occurring to obtain the failure probability of the RAID group.

8. The RAID failure probability calculation device according to claim 2, wherein, during RAID reconstruction performed when replacing the HDDs constituting the RAID group, the probability of another HDD constituting the RAID group failing is determined, and this probability is added to the failure probability of the RAID group to obtain the failure probability of the RAID group.

9. The RAID failure probability calculation device according to claim 3, wherein the RAID configuration information stored in the RAID configuration information recording table is stored for each attribute of the HDD based on a record of the time required for RAID reconstruction work obtained from the I / O amount of the HDD.

10. The RAID failure probability calculation device according to claim 2, comprising a storage unit that stores attribute information including information about the HDD model and a model that takes the predicted I / O amount as an input value and the RAID reconstruction time as an output value, and receiving the attribute information and the predicted I / O amount, and determining the RAID reconstruction time.

11. The HDD error information collection unit collects errors from the HDDs that make up the RAID group. The HDD failure probability calculation unit calculates the failure probability of the HDD based on the HDD errors collected from the HDD error information collection unit. The RAID failure probability calculation unit calculates the failure probability of the RAID group based on the HDD failure probability calculation unit's calculation of the HDD failure probability and the RAID configuration information obtained from the RAID configuration information record table that stores the RAID configuration information of the RAID group composed of the HDDs. A method for calculating the probability of failure of a RAID group, wherein the output unit outputs the probability of failure of the RAID group.

12. A computer, Error collection process to collect errors from HDDs that make up a RAID group, A failure probability calculation process that determines the probability of failure of the HDD based on the collected errors of the HDD, A RAID failure probability calculation process that determines the failure probability of the RAID group based on the failure probability of the HDD and the RAID configuration information obtained from a RAID configuration information record table that stores the RAID configuration information of the RAID group composed of the HDDs, A RAID failure probability calculation program that executes an output process to output the failure probability of the aforementioned RAID group.

Images