A storage drive failure prediction method and storage cluster

Abstract

The application discloses a storage drive fault prediction method and a storage cluster. The method comprises the following steps: obtaining fault quantity information of a storage group in a storage cluster, each storage group comprising a plurality of storage drives with the same attribute in the storage cluster, the fault quantity information representing the number of storage drives in the corresponding storage group that have failed, and the storage cluster comprising a plurality of storage drives; and outputting fault prediction information of the storage cluster in the case that the fault quantity information of any storage group meets a pre-warning condition, the fault prediction information representing that a plurality of storage drives belonging to a target storage group in the storage cluster are at risk of failure, and the target storage group being the storage group whose fault quantity information meets the pre-warning condition.

Description

Technical Field

[0001] This application relates to the field of fault prediction, and in particular to a method for predicting storage drive faults and a storage cluster. Background Technology

[0002] Currently, storage clusters composed of multiple storage drives are widely used in various fields that require large-scale data storage. In order for a storage cluster to function properly, each storage drive typically has a built-in fault detection module to detect whether the corresponding storage drive has failed, so that timely countermeasures can be taken when a failure occurs, such as migrating the data stored in the failed storage drive.

[0003] However, storage drives in a storage cluster can experience large-scale failures, meaning multiple storage drives in the cluster fail simultaneously. In such cases, the sheer number of failed drives makes it difficult to migrate the data stored on these drives in a timely manner, leading to data loss. Currently, modules that detect individual storage drive failures are ill-suited to predicting large-scale failures across the entire storage cluster.

[0004] Therefore, how to predict large-scale failures involving multiple storage drives in a storage cluster has become an urgent problem to be solved. Summary of the Invention

[0005] Therefore, this application discloses the following technical solution:

[0006] The first aspect of this application provides a method for predicting storage drive failures, including:

[0007] Obtain the failure count information of storage groups in the storage cluster. Each storage group includes multiple storage drives with the same attributes in the storage cluster. The failure count information represents the number of storage drives that have failed in the corresponding storage group. The storage cluster includes multiple storage drives.

[0008] If the number of failures in any storage group meets the warning conditions, the fault prediction information of the storage cluster is output. The fault prediction information indicates that multiple storage drives belonging to the target storage group in the storage cluster are at risk of failure. The target storage group refers to the storage group whose number of failures meets the warning conditions.

[0009] Optionally, the storage group includes at least one of the following:

[0010] Storage groups are divided by manufacturer, and each storage group divided by manufacturer includes multiple storage drives in the storage cluster that are manufactured by the same manufacturer.

[0011] Storage groups are divided by model number, and each storage group includes multiple storage drives of the same model number in the storage cluster.

[0012] Storage groups are divided by firmware, and each storage group includes multiple storage drives in the storage cluster that have the same version of firmware installed.

[0013] Storage groups are divided by production batch, and each storage group includes multiple storage drives belonging to the same production batch in the storage cluster.

[0014] Optionally, the fault count information of the storage group includes the immediate fault count and the cumulative fault count;

[0015] The instantaneous failure count represents the number of storage drives that have failed in the corresponding storage group at the current moment;

[0016] The cumulative number of failures represents the number of storage drives that have failed within the cumulative time period up to the current moment in the corresponding storage group;

[0017] The number of faults in the storage group meets the early warning conditions, including:

[0018] The number of immediate failures of the storage group is greater than a first threshold, and / or the cumulative number of failures of the storage group is greater than a second threshold, wherein the first threshold is less than the second threshold.

[0019] Optionally, obtaining the number of failures in storage groups within the storage cluster includes:

[0020] For any storage group in the storage cluster, determine whether the failure of the storage driver in the storage group is a target failure or a non-target failure, so as to obtain the failure quantity information of the storage group based on the number of storage drivers in the storage group that have experienced target failures;

[0021] The target faults include faults caused by the hardware and / or firmware of the storage drive itself, and the non-target faults include faults caused by associated modules that support the operation of the storage drive.

[0022] Optional, also includes:

[0023] Based on the attributes of the storage drives that have failed in the target storage group and the attributes of the storage drives that have not failed in the target storage group, the predicted cause of failure for the target storage group is determined.

[0024] Output the fault handling suggestions corresponding to the predicted fault causes.

[0025] Optional, also includes:

[0026] Based on the attributes of the storage drives that have failed in the target storage group, a fault knowledge base related to the failed storage drives is obtained. The fault knowledge base includes relevant information about other storage drives that have the same attributes as the failed storage drives.

[0027] The predicted fault causes corresponding to the target storage group are determined based on the fault knowledge base.

[0028] Output the fault handling suggestions corresponding to the predicted fault causes.

[0029] Optional, also includes:

[0030] Based on the number range to which the number of faults in the target storage group belongs, a fault handling strategy corresponding to the number range is executed on the storage drives contained in the target storage group, wherein different fault handling strategies correspond to different number ranges.

[0031] Optionally, the fault quantity information includes the cumulative fault quantity, and the execution of a fault handling strategy corresponding to the quantity range to which the storage drives contained in the target storage group belong includes at least one of the following:

[0032] If the cumulative number of faults is greater than the second threshold and less than or equal to the third threshold, the hot data stored in the storage drive of the target storage group whose access frequency meets the first condition will be migrated to the storage drive of the non-target storage group.

[0033] If the cumulative number of faults is greater than the third threshold and less than or equal to the fourth threshold, reduce the frequency of storing data to the storage driver of the target storage group;

[0034] If the cumulative number of failures exceeds the fourth threshold, the storage driver for the target storage group is removed from the storage cluster;

[0035] Wherein, the cumulative number of failures represents the number of storage drives that have failed within the cumulative time period up to the current moment in the corresponding storage group, and the second threshold, the third threshold, and the fourth threshold increase sequentially.

[0036] Optionally, removing the storage drive of the target storage group from the storage cluster includes at least one of the following:

[0037] Based on the total amount of data stored in the storage drives of the target storage group and the available capacity of the storage cluster, determine the maximum number of drives to be removed, and remove the maximum number of storage drives in the target storage group.

[0038] Remove a portion of the storage drives from the target storage group, and remove a portion of the storage drives from the target storage group again if a storage drive in the target storage group fails;

[0039] The available capacity of the storage cluster refers to the amount of data that other storage drives in the storage cluster, excluding the target storage group, can store.

[0040] A second aspect of this application provides a storage cluster, including a storage controller and multiple storage drives;

[0041] The plurality of storage drives are used to store data;

[0042] The storage controller is used to perform:

[0043] Obtain the failure count information of storage groups in the storage cluster. Each storage group includes multiple storage drives with the same attributes in the storage cluster. The failure count information represents the number of storage drives that have failed in the corresponding storage group. The storage cluster includes multiple storage drives.

[0044] If the number of failures in any storage group meets the warning conditions, the failure prediction information of the storage cluster is output. The failure prediction information indicates that the storage driver in the storage cluster belonging to the target storage group has a failure risk. The target storage group refers to the storage group whose failure number information meets the warning conditions. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0046] Figure 1 This is a flowchart of a storage drive failure prediction method provided in an embodiment of this application;

[0047] Figure 2 This is a flowchart of another storage drive failure prediction method provided in the embodiments of this application;

[0048] Figure 3 This is a schematic diagram illustrating an application scenario of a storage drive failure prediction method provided in an embodiment of this application;

[0049] Figure 4 This is a flowchart of another storage drive failure prediction method provided in the embodiments of this application;

[0050] Figure 5 This is a schematic diagram of a storage cluster structure provided in an embodiment of this application. Detailed Implementation

[0051] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0052] This application provides a method for predicting storage drive failures. Please refer to [link to relevant documentation]. Figure 1 The method may include the following steps.

[0053] S101, obtain the failure number information of storage groups in the storage cluster. Each storage group includes multiple storage drives with the same attributes in the storage cluster. The failure number information represents the number of storage drives that have failed in the corresponding storage group. The storage cluster includes multiple storage drives.

[0054] S102, if the number of faults in any storage group meets the warning conditions, output the fault prediction information of the storage cluster. The fault prediction information indicates that multiple storage drives belonging to the target storage group in the storage cluster are at risk of failure. The target storage group refers to the storage group whose number of faults meets the warning conditions.

[0055] The form of fault prediction information is not limited. For example, all storage drives that do not currently fail and belong to the target storage group can be marked as high-risk storage drives, and the identifiers and high-risk labels of these high-risk storage drives can be output as fault prediction information.

[0056] Alternatively, text description information can be output as fault prediction information. For example, if the target storage group is a group consisting of all storage drives manufactured by manufacturer A, then the fault prediction information can be output as: All storage drives manufactured by manufacturer A have a high risk of failure. Please pay attention to this.

[0057] The beneficial effects of this embodiment are as follows:

[0058] The storage drives of the storage cluster are divided into multiple storage groups based on the same attributes, and the failure number information of each storage group is collected. Then, based on the failure number information of each storage group, it is predicted whether there is a risk of multiple storage drives in the corresponding storage group failing at the same time, and failure prediction information is output when there is a corresponding risk.

[0059] For example, if storage drives within a storage group belonging to the same manufacturer have failed multiple times in a row, then it can be assumed that other storage drives produced by this manufacturer that have not yet failed also have a high risk of failure and will also fail at some point in the near future. Therefore, failure prediction information can be output for this storage group to indicate that these storage drives are at risk of failure.

[0060] Therefore, this solution can effectively predict whether a large-scale failure involving multiple storage drives will occur in the storage cluster, thereby taking countermeasures in advance when a large-scale failure may occur to avoid data loss.

[0061] A storage cluster consists of one or more server devices, each equipped with multiple storage drives, each capable of storing a certain amount of data. The method in this embodiment can be executed by the storage controller of the storage cluster. Each server device can be equipped with a detection module for detecting storage drive failures. The storage controller can collect the failures detected by these detection modules to determine the number of failures in each storage group. Specific methods for the detection modules to detect failures can be found in relevant technologies in the field of storage drive failure detection. The storage controller can be an independent control device separate from the server devices, or it can be a control module integrated into the server devices.

[0062] Multiple storage drives that share commonalities in the manufacturing process often have similar safety hazards. For example, if a manufacturer uses hard drive materials that are generally defective, then multiple storage drives produced by that manufacturer may simultaneously experience failures related to those hard drive materials within a short period of time (e.g., within a single day).

[0063] Therefore, the same attributes in step S101 may include manufacturing attributes related to the production process of the storage drive. As some examples, having the same attributes may include any one or more of the following: belonging to the same manufacturer, having the same model, having the same firmware version, and belonging to the same production batch.

[0064] Correspondingly, the method of dividing storage groups in S101 may include at least one of the following:

[0065] Storage groups are divided by manufacturer, with each storage group corresponding to one manufacturer. Each storage group divided by manufacturer includes multiple storage drives in the storage cluster that are manufactured by the same manufacturer.

[0066] Storage groups are divided by model number, with each storage group corresponding to a model number. Each storage group divided by model number includes multiple storage drives with the same model number in the storage cluster.

[0067] Storage groups are divided by firmware, with each storage group corresponding to a firmware version. Each storage group divided by firmware includes multiple storage drives in the storage cluster that have the same firmware version installed.

[0068] Storage is divided into production batches, with each storage group corresponding to a production batch. Each storage group divided into production batches includes multiple storage drives belonging to the same production batch in the storage cluster.

[0069] Optionally, storage groups can be divided by combining multiple attributes, such as grouping multiple storage drives belonging to the same manufacturer and having the same firmware version into one storage group.

[0070] There can be overlap between different storage groups, meaning that a storage drive can belong to multiple storage groups at the same time. For example, storage drives 1, 2, and 3 belong to the storage group of manufacturer A, and all storage drives in this group are manufactured by manufacturer A. At the same time, storage drives 2 and 3 belong to the storage group with firmware version 1.0, and all storage drives in this group have firmware version 1.0 installed.

[0071] The division of storage groups can be configured as needed by users of the storage cluster, without any limitations. For example, it can be configured to divide storage into multiple storage groups only based on the same manufacturer, or it can be configured to divide storage into multiple storage groups based on the same manufacturer, and further divide the storage drives produced by one or more manufacturers into multiple smaller storage groups based on the same model or firmware version.

[0072] Dividing multiple storage drives into storage groups based on whether they share the same manufacturing attributes can make the potential failures and timing of failures within each group somewhat similar. Therefore, by analyzing the failure count within each storage group, it's possible to more accurately predict whether other drives with the same manufacturing attributes within that group are likely to fail in the near future, thus obtaining more accurate failure prediction information. Manufacturing attributes include, but are not limited to, manufacturer, firmware version, manufacturing process, model, batch, and materials used.

[0073] Optionally, the failure count information for the storage group includes the immediate failure count and the cumulative failure count;

[0074] The instantaneous failure count represents the number of storage drives that have failed in the corresponding storage group at the current moment;

[0075] The cumulative number of failures represents the number of storage drives that have failed within the cumulative time period up to the current moment in the corresponding storage group;

[0076] The number of storage group failures meets the warning criteria, including:

[0077] The number of immediate failures of a storage group is greater than a first threshold, and / or the cumulative number of failures of a storage group is greater than a second threshold, where the first threshold is less than the second threshold.

[0078] In other words, as long as either the number of immediate failures of a storage group is greater than the first threshold or the cumulative number of failures of a storage group is greater than the second threshold, the warning condition can be determined to be met. If neither of the two conditions is met, the warning condition is not met.

[0079] Optionally, based on the division of storage groups, a first threshold and a second threshold applicable to each storage group can be pre-configured. Different storage groups may have different thresholds, that is, two different storage groups may have different first thresholds and / or different second thresholds.

[0080] The first threshold for a storage group, also known as the immediate threshold, can be determined based on the fault tolerance capability of the storage cluster. Fault tolerance can be represented by the maximum number of storage drives that can fail simultaneously without causing data loss. The larger the value of N, the stronger the fault tolerance. Fault tolerance is related to the number of replicas of the same data in the storage cluster and the way each replica is stored. As examples, the first threshold for any storage group can be set equal to the corresponding N value of the storage cluster, or it can be set slightly less than the N value.

[0081] The second threshold for a storage group, also known as the cumulative threshold, can be determined based on the past failure patterns of storage drives with similar attributes. For example, based on past failure patterns, if 12 storage drives manufactured by Manufacturer A have failed cumulatively, there is a high probability that subsequent storage drives from Manufacturer A will experience a large-scale failure simultaneously. Therefore, the second threshold for the storage group corresponding to Manufacturer A can be set to 12. Alternatively, the second threshold can also be determined based on a preset second ratio and the number of storage drives in the storage group. For example, if the second ratio is 10% and the storage group contains 100 storage drives, then the second threshold is set to 10.

[0082] When a failure (or a faulty state) is detected in any storage drive in the storage cluster, the storage controller can prompt the relevant personnel responsible for maintenance to take action. The actions include, but are not limited to, repairing the faulty storage drive to eliminate the fault, or deactivating the faulty storage drive (i.e., removing it from the storage cluster).

[0083] For a storage group, the number of immediate failures is equal to the number of storage drives in that storage group that have failed (or are in a failed state) and have not yet been removed at the current moment. For a storage group, the cumulative number of failures is equal to the total number of storage drives that have failed within that storage group during the cumulative time period up to the present, regardless of whether these failed storage drives have been repaired or removed.

[0084] The specific length of the cumulative time period can be set as needed and is not limited. For example, the cumulative time period can be the most recent 15 days, the most recent 30 days, or the most recent 90 days.

[0085] Taking the cumulative time period of the last 30 days as an example, if a storage group has 20 storage drives that have failed in the last 30 days, and as of now, 3 of these 20 storage drives have been removed from the storage cluster, and 6 have been repaired and their corresponding faults have been resolved, then the immediate number of failures for this storage group is equal to 11, meaning that there are currently 11 storage drives in this storage group that are in a faulty state and have not been removed. The cumulative number of failures for this storage group is equal to 20, meaning that a total of 20 storage drives have failed in the last 30 days.

[0086] The advantages of calculating the number of immediate and cumulative failures separately using the above methods, and then issuing warnings based on different thresholds, are:

[0087] By counting the number of immediate failures and issuing an alert when they exceed a first threshold, we can effectively identify and predict failures occurring simultaneously in multiple storage drives within a short period of time. By counting the number of cumulative failures and issuing an alert when they exceed a second threshold, we can identify large-scale failures caused by inherent quality defects in multiple storage drives within a storage group. Therefore, issuing alerts for both immediate and cumulative failures can effectively predict failures of different causes, improving the accuracy and scope of failure prediction.

[0088] Optionally, methods for obtaining information on the number of failures in storage groups within a storage cluster may include:

[0089] For any storage group in the storage cluster, determine whether the failure of the storage driver in the storage group is a target failure or a non-target failure, so as to obtain the failure quantity information of the storage group based on the number of storage drivers in the storage group that have experienced target failures;

[0090] Targeted failures include failures caused by the storage drive's own hardware and / or firmware, while non-targeted failures include failures caused by associated modules that support the operation of the storage drive.

[0091] In this embodiment, when the storage controller detects a failure in any storage drive, it can obtain relevant information about the storage drive through the detection module. This information includes, but is not limited to, the status information of each register before and after the storage drive failure, and the status information of each associated module connected to the storage drive to support its operation. The associated modules may, for example, include the power supply of the storage drive. Methods for collecting this information can be found in relevant technologies in the field of storage drive failure detection.

[0092] Based on the above information, the storage controller can determine whether the faulty storage drive is caused by its own reasons (such as damage to internal storage chips, circuit failure, firmware failure, etc.) or by a related module failure, such as a failure due to power supply failure. If it is the former, the fault of this storage drive is determined to be the target fault; if it is the latter, it is determined to be a non-target fault.

[0093] If the failure of the storage drive is a target failure, the storage controller updates the failure count information of the storage group to which the storage drive belongs, for example, by incrementing the immediate failure count of the storage group and the cumulative failure count by 1. If the failure of the storage drive is a non-target failure, the storage controller ignores the failure and does not update the failure count information of the storage group to which the storage drive belongs, that is, it does not count the non-target failure in the failure count information.

[0094] The advantage of obtaining fault quantity information using the above method is that sometimes the failure of a storage drive may not be caused by the storage drive itself, but by the failure of related modules.

[0095] For example, when the power supply to the storage drive fails, the storage drive will stop and become unusable due to the lack of power input. At this time, the detection module will identify that the storage drive has failed, but this failure is caused by the power supply failure, rather than by the storage drive itself.

[0096] Obviously, the failure of storage drives affected by such associated modules will not lead to large-scale failures of other storage drives in the same storage group at the same time. Including such failures in the failure count information will reduce the accuracy of failure prediction information. Therefore, by distinguishing between target failures and non-target failures, and only counting target failures when obtaining failure count information, we can exclude those non-target failures that cause interference, thereby obtaining more accurate failure prediction information.

[0097] Optional, please see Figure 2 Here is a flowchart of another fault prediction method in this embodiment, which includes... Figure 1The steps S101 and S102 shown are included, and the following steps are also included after step S102.

[0098] S201, based on the attributes of the storage drives that have failed in the target storage group and the attributes of the storage drives that have not failed in the target storage group, determine the predicted cause of failure for the target storage group.

[0099] S202 outputs fault handling suggestions corresponding to the predicted fault causes.

[0100] The storage drives that fail in step S201 include storage drives in the target storage group that are currently in a faulty state and have not been removed, as well as storage drives in the target storage group that have previously failed and have been repaired or removed.

[0101] In step S201, the attributes of all failed storage drives in the target storage group can be statistically analyzed. For example, failed storage drives all use the same firmware version, failed storage drives all use the same manufacturing process, and failed storage drives all use the same material (such as storage chips from the same supplier).

[0102] Furthermore, the attributes of these failed storage drives can be compared with the attributes of the non-failed storage drives in the target storage group. If a failed storage drive has an attribute that is different from that of a non-failed storage drive in the target storage group, then this attribute can be identified as a predicted cause of failure.

[0103] In S202, after determining the predicted cause of the failure, the preset failure handling suggestions corresponding to the predicted cause can be read from the pre-configured failure handling suggestion library, and the predicted cause of the failure and the failure handling suggestions can be output together to the relevant personnel responsible for maintaining the storage cluster.

[0104] The above attributes all refer to manufacturing attributes related to the production and manufacturing process. Manufacturing attributes include, but are not limited to, manufacturer, firmware version, production process, model, batch, and materials used.

[0105] For example, if the faulty storage drives all use firmware version 1.1, and the non-faulty storage drives all use firmware version 1.2, then it can be determined that firmware version 1.1 is the predicted cause of the fault. The corresponding fault handling suggestion "update firmware version" is read, and the output information is: the predicted cause of the fault is the use of firmware version 1.1, and it is recommended to update the firmware version.

[0106] For example, if all the storage drives that did not fail use a certain firmware version, such as version 1.3, while the other storage drives that failed use different firmware versions, such as version 1.2 or version 1.1, then the output message would be: The predicted cause of the failure is that the firmware version 1.3 is not being used, and it is recommended to update the firmware version.

[0107] Optionally, in the above methods for determining the cause of predicted failures, the specific attributes to be analyzed can be set by the storage cluster user as needed. For example, the user can set the firmware version, manufacturing process, and materials to be compared sequentially when determining the cause of predicted failures, or the user can set the firmware version not to be compared, and instead compare the manufacturer, manufacturing process, and materials sequentially.

[0108] Optionally, after outputting the predicted fault information, the method of this embodiment can also obtain the predicted fault cause and fault handling suggestions in the following manner: It further includes:

[0109] Based on the attributes of the failed storage drives in the target storage group, obtain a fault knowledge base related to the failed storage drives. The fault knowledge base includes relevant information about other storage drives with the same attributes as the failed storage drives.

[0110] Determine the predicted causes of failures for the target storage group based on the fault knowledge base;

[0111] Output the predicted causes of the fault and corresponding fault handling suggestions.

[0112] Optionally, information about other storage drives with the same attributes as the failed storage drive may include product information released by the manufacturer of the failed storage drive; past failure handling cases released by the manufacturer; and product information and / or failure handling cases of other storage drives that use the same firmware version and the same materials (such as the same type of storage chip) as the failed storage drive.

[0113] In this embodiment, the storage controller can collect the aforementioned fault knowledge base only after outputting the predicted fault information, or it can collect and store the corresponding fault knowledge base for various storage drives in the storage cluster in advance, and directly call the corresponding fault knowledge base when outputting the predicted fault information.

[0114] After obtaining a fault knowledge base related to the failed storage drive, the failed storage drive and the obtained fault knowledge base can be input into a pre-built fault reasoning model. The fault reasoning model is then used to generate a predicted cause of the fault, and the corresponding fault handling suggestions are obtained according to the method described in step S202 above. The fault reasoning model can be a pre-trained neural network model with reasoning capabilities (e.g., a large language model).

[0115] The above methods can more accurately determine and predict the causes of failures by introducing a fault knowledge base.

[0116] Based on the two methods of determining the predicted causes of failures and providing fault handling suggestions, this embodiment can simultaneously output the predicted causes of failures and fault handling suggestions for relevant personnel to refer to when outputting the predicted failure information, thereby improving the efficiency of maintaining the storage cluster.

[0117] Optionally, after outputting fault prediction information, the storage controller can also automatically execute fault handling strategies to improve the security of the storage cluster.

[0118] The fault handling strategy can vary depending on the number of faults; that is, the fault handling strategy executed when there are many faults can be different from the fault handling strategy executed when there are few faults.

[0119] In other words, after outputting fault prediction information, the method of this embodiment may further include:

[0120] Based on the number range to which the number of failures in the target storage group belongs, the fault handling strategy corresponding to the number range is executed on the storage drives contained in the target storage group. The fault handling strategies are different for different number ranges.

[0121] To accommodate the differences in the number of storage drives between different storage groups, the above-mentioned range can be expressed as a percentage. For example, the range may include: 10% to 20% of the total number of storage drives; 20% to 30% of the total number of storage drives; 30% to 50% of the total number of storage drives; and more than 50% of the total number of storage drives.

[0122] Fault handling strategies corresponding to different quantity ranges can be pre-configured by storage cluster users as needed, without limitation. As some examples, fault handling strategies may include, but are not limited to: increasing the number of data copies, performing system upgrades, data migration and transformation, marking and removing storage drives, etc.

[0123] The advantage of applying the above embodiments is that, in the event that a large number of storage drives in the target storage group fail, the storage controller can automatically implement the corresponding fault handling strategy, thereby preventing data loss and other problems caused by the failure of relevant personnel to maintain the data in a timely manner.

[0124] As examples, the aforementioned failure count information may include the cumulative failure count of the target storage group. Accordingly, a failure handling strategy corresponding to the corresponding failure range is implemented for the storage drives contained in the target storage group, including at least one of the following:

[0125] Strategy 1: If the cumulative number of failures is greater than the second threshold and less than or equal to the third threshold, migrate the hot data stored in the storage drive of the target storage group whose access frequency meets the first condition to the storage drive of the non-target storage group.

[0126] Strategy 2: If the cumulative number of failures is greater than the third threshold but less than or equal to the fourth threshold, reduce the frequency of storing data to the storage drivers of the target storage group.

[0127] Strategy 3: If the cumulative number of failures exceeds the fourth threshold, remove the storage driver of the target storage group from the storage cluster.

[0128] The cumulative number of failures represents the number of storage drives that have failed within the cumulative time period up to the current moment in the corresponding storage group, with the second, third, and fourth thresholds increasing sequentially.

[0129] The second threshold could be, for example, 10% of the total number of storage drives, and the third threshold could be 30% of the total number of storage drives. The access frequency meets the first condition, specifically, the access frequency can be greater than a preset hot data frequency threshold, such as an access frequency of more than 70 times per minute or more than 100 times per minute.

[0130] In other words, in Strategy 1, storage drives that belong to the target storage group and have not yet failed can be marked as high-risk storage drives, while storage drives that do not belong to the target storage group can be marked as non-high-risk storage drives. Then, all hot data stored on high-risk storage drives can be migrated to non-high-risk storage drives.

[0131] The fourth threshold could be, for example, 50% of the total number of storage drives. In Strategy 2, storage drives belonging to the target storage group can be marked as high-risk storage drives, and storage drives not belonging to the target storage group can be marked as non-high-risk storage drives. Then, whenever the storage controller obtains data that needs to be stored, it will preferentially store the data in non-high-risk storage drives, and only store data in high-risk storage drives when the used capacity of all non-high-risk storage drives is close to the capacity limit.

[0132] By applying strategy two, the amount of data stored in high-risk storage drives can be gradually reduced to avoid data loss due to failure of high-risk storage drives.

[0133] In Strategy 3, high-risk storage drives can be removed using at least one of the following removal methods:

[0134] Method 1: Determine the maximum number of drives to be removed based on the total amount of data stored in the storage drives of the target storage group and the available capacity of the storage cluster, and remove the maximum number of storage drives to be removed from the target storage group.

[0135] Method 2 involves removing some storage drives from the target storage group, and then removing some storage drives from the target storage group again if a storage drive in the target storage group fails.

[0136] The available capacity of a storage cluster represents the amount of data that other storage drives in the storage cluster, excluding the target storage group, can store.

[0137] In removal method one, the total amount of data currently stored by all high-risk storage drives in the storage cluster can be calculated, as well as the total available capacity of all non-high-risk storage drives in the storage cluster, i.e., how much data can be stored in total by all non-high-risk storage drives.

[0138] Based on the total amount of data and available capacity mentioned above, we can determine the maximum number of high-risk storage drives that can be removed, assuming that as much data as possible from high-risk storage drives is migrated to non-high-risk storage drives.

[0139] For example, if the total data volume is 20 gigabytes (GB) and there is 15GB of free space, then it can be determined that after migrating as much data as possible from high-risk storage drives to non-high-risk storage drives, there will still be 5GB of data that needs to be stored on high-risk storage drives. That is, at least one high-risk storage drive needs to be retained. Assuming there are currently 6 high-risk storage drives, the maximum number of drives that can be removed can be determined to be 5.

[0140] Once the maximum number of high-risk storage drives to be removed is determined, data from high-risk storage drives can be migrated as much as possible to non-high-risk storage drives. Necessary high-risk storage drives can be retained to store excess data that cannot be migrated. The remaining high-risk storage drives that have already been cleared and are within the maximum removal limit can then be removed from the storage cluster. For example, all five high-risk storage drives mentioned above can be removed.

[0141] Optionally, when migrating data in the manner described above, hot data can be migrated to non-high-risk storage drives first.

[0142] In this embodiment, removing the storage driver from the storage cluster can be understood as powering down the storage driver and simultaneously marking the storage cluster as unavailable in the system.

[0143] If a high-risk storage drive is removed using removal method one, the removed high-risk storage drive can be selectively recovered using the following recovery methods:

[0144] When the amount of data to be stored in the storage cluster increases and the remaining storage drives are insufficient to store the data, the high-risk storage drives that were removed can be restored to the storage cluster one by one, and data can be stored on these high-risk storage drives again. When storing data, priority should be given to storing non-hot data with low access frequency on high-risk storage drives.

[0145] To reduce the risk of data loss, the number of high-risk storage drives that need to be restored each time should be kept to a minimum, such as only one at a time.

[0146] The advantage of removing data using removal method one is that it can minimize the amount of data stored on high-risk storage drives while meeting storage requirements, thereby maximizing data security.

[0147] In removal method two, after outputting the fault prediction information, a first preset proportion of high-risk storage drives can be removed first. Then, each time a high-risk storage drive failure is detected, a second preset proportion of high-risk storage drives can be removed, until all high-risk storage drives are removed.

[0148] The first and second preset ratios can be set as needed, with the former being larger than the latter. For example, the first preset ratio could be 40% and the second preset ratio could be 20%.

[0149] If removed using removal method two, the removed high-risk storage drive can be restored to the storage cluster as follows:

[0150] When the amount of data to be stored increases and more storage space is required, high-risk storage drives that have been removed but are currently functioning normally (having completed fault repair or not yet failed) will be gradually restored to the storage cluster according to a preset ratio. The preset ratio can be 20%, 30%, or other ratios, and is not limited to one.

[0151] The advantage of removing via method two is that it reduces the number of drives removed each time, avoiding unnecessary power consumption caused by frequently restoring access to high-risk storage drives when more data needs to be stored.

[0152] Optionally, each time a predicted fault is output, the storage controller can simultaneously notify the system administrator who manages and maintains the storage cluster to take necessary measures, such as contacting the supplier or after-sales service.

[0153] Furthermore, each time a predicted fault information is output, the storage controller can output the predicted fault information in the form of an alarm event, and can also automatically generate after-sales service orders for each high-risk storage drive in the target storage group for system administrators to call.

[0154] Optionally, other quantity ranges and corresponding fault handling strategies may also be included.

[0155] For example, if the cumulative number of failures in a target storage group exceeds 20% of its total number of storage drives, and the predicted cause of the failure is determined to be firmware-related in the manner described above, a failure handling strategy of upgrading the firmware version can be implemented.

[0156] If multiple storage drive failures are still detected in the target storage group after upgrading the firmware version, it can be assumed that the cause of the failure is not the firmware version, and other fault handling strategies can be adopted.

[0157] For example, if the cumulative number of failures in a target storage group exceeds 50% of its total number of storage drives, an upgraded data fault tolerance fault handling strategy can be implemented for all storage drives in the entire storage cluster. This fault handling strategy may include any one or more of the following:

[0158] Increase the number of data copies in the storage drive, that is, make more copies of the same data and store them separately;

[0159] Data is stored using erasure coding strategies. For details on how erasure coding strategies are applied, please refer to the relevant technologies.

[0160] At the same time, the number of data copies is increased and erasure coding strategy is used to store the data.

[0161] The total number of storage drives mentioned in the above range refers to the total number of storage drives within a storage group (such as the target storage group).

[0162] The following is combined Figure 3 The following are application scenario examples to illustrate the implementation process of the storage drive failure prediction method in this embodiment.

[0163] Figure 3 Here is an example of a storage cluster where the storage controller is connected to three server devices (server devices 1 to 3) in the storage cluster, and each server device is equipped with multiple storage drives.

[0164] The storage controller monitors each storage drive for failure in real time using the method described above, and manages the storage space of each storage drive. The storage controller groups all storage drives by model, into a Y-model storage group and an X-model storage group. The former contains all Y-model storage drives, and the latter contains all X-model storage drives.

[0165] At a certain moment, the storage controller detects that two Y-type storage drives in a Y-type storage group have failed simultaneously, meaning the immediate number of failures in the Y-type storage group is 2. In addition, one Y-type storage drive had already failed and been removed within the previous cumulative time period, meaning the cumulative number of failures in the Y-type storage group is 3. Assuming that the number of immediate failures has exceeded the aforementioned first threshold, an alarm event containing predicted failure information can be output to the system administrator. The predicted failure information can indicate that all Y-type storage drives are high-risk storage drives.

[0166] After an alarm event is output, the storage controller can selectively execute the corresponding fault handling strategy according to the aforementioned preset fault handling strategies and corresponding execution conditions.

[0167] In conjunction with the above embodiments, this embodiment also provides a storage drive failure prediction method, please refer to [link to relevant documentation]. Figure 4 The method may include the following steps.

[0168] S401 divides storage drives with the same attributes in the storage cluster into the same storage group, forming multiple storage groups.

[0169] The method for dividing storage groups can be found in the relevant content of step S101 in the aforementioned embodiments.

[0170] S402 monitors each storage drive for failures to count the instantaneous number of failures for each storage group and the cumulative number of failures over a cumulative time period.

[0171] For the statistical methods of instantaneous failure count and cumulative failure count, please refer to the definitions of the two mentioned above.

[0172] S403 outputs fault prediction information corresponding to the target storage group when the early warning conditions are met.

[0173] S404, determine if the predicted fault cause is the firmware version.

[0174] If the predicted cause of the failure is the firmware version, for example, if firmware version 1.1 is determined to be the predicted cause of the failure in the aforementioned embodiment, then step S405 is executed; if the predicted cause of the failure is not the firmware version, then step S406 is executed. The implementation of S404 is described in the aforementioned embodiment regarding the determination of the predicted cause of the failure.

[0175] S405, upgrade the firmware version of high-risk storage drives belonging to the target storage group.

[0176] For methods on upgrading firmware versions, please refer to the relevant technologies in the field of firmware upgrades for storage drives, which will not be elaborated here.

[0177] Optionally, if a storage drive failure is detected again in the target storage group some time after the firmware version is upgraded, S406 can be executed.

[0178] S406 executes preset fault handling strategies as needed.

[0179] For details on which fault handling strategies to implement, please refer to the description of fault handling strategies in the aforementioned embodiments.

[0180] This application also provides a storage cluster, see [link to relevant documentation] Figure 5 This includes multiple storage drives 501 and a storage controller 502;

[0181] Multiple storage drives 501 are used to store data;

[0182] Storage controller 502 is used to perform:

[0183] Obtain the failure count information of storage groups in the storage cluster. Each storage group includes multiple storage drives with the same attributes in the storage cluster. The failure count information represents the number of storage drives that have failed in the corresponding storage group. The storage cluster includes multiple storage drives.

[0184] If the number of failures in any storage group meets the warning conditions, the failure prediction information of the storage cluster is output. The failure prediction information indicates that the storage driver in the storage cluster belonging to the target storage group has a failure risk. The target storage group refers to the storage group whose failure number information meets the warning conditions.

[0185] The working principle of the storage cluster in this embodiment can be found in the relevant steps of the storage driver failure prediction method in the foregoing embodiments, and will not be repeated here.

[0186] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0187] For ease of description, the above systems or devices are described separately as various modules or units based on their functions. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware components.

[0188] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0189] Finally, it should be noted that in this document, relational terms such as first, second, third, and fourth are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0190] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for predicting storage drive failures, comprising: Obtain the failure count information of storage groups in the storage cluster. Each storage group includes multiple storage drives with the same attributes in the storage cluster. The failure count information represents the number of storage drives that have failed in the corresponding storage group. The storage cluster includes multiple storage drives. If the number of failures in any storage group meets the warning conditions, the fault prediction information of the storage cluster is output. The fault prediction information indicates that multiple storage drives belonging to the target storage group in the storage cluster are at risk of failure. The target storage group refers to the storage group whose number of failures meets the warning conditions.

2. The method according to claim 1, wherein the storage group comprises at least one of the following: Storage groups are divided by manufacturer, and each storage group divided by manufacturer includes multiple storage drives in the storage cluster that are manufactured by the same manufacturer. Storage groups are divided by model number, and each storage group includes multiple storage drives of the same model number in the storage cluster. Storage groups are divided by firmware, and each storage group includes multiple storage drives in the storage cluster that have the same version of firmware installed. Storage groups are divided by production batch, and each storage group includes multiple storage drives belonging to the same production batch in the storage cluster.

3. The method according to claim 1, wherein the fault quantity information of the storage group includes the immediate fault quantity and the cumulative fault quantity; The instantaneous failure count represents the number of storage drives that have failed in the corresponding storage group at the current moment; The cumulative number of failures represents the number of storage drives that have failed within the cumulative time period up to the current moment in the corresponding storage group; The number of faults in the storage group meets the early warning conditions, including: The number of immediate failures of the storage group is greater than a first threshold, and / or the cumulative number of failures of the storage group is greater than a second threshold, wherein the first threshold is less than the second threshold.

4. The method according to claim 1, wherein obtaining the number of failures of storage groups in the storage cluster includes: For any storage group in the storage cluster, determine whether the failure of the storage driver in the storage group is a target failure or a non-target failure, so as to obtain the failure quantity information of the storage group based on the number of storage drivers in the storage group that have experienced target failures; The target faults include faults caused by the hardware and / or firmware of the storage drive itself, and the non-target faults include faults caused by associated modules that support the operation of the storage drive.

5. The method according to claim 1, further comprising: Based on the attributes of the storage drives that have failed in the target storage group and the attributes of the storage drives that have not failed in the target storage group, the predicted cause of failure for the target storage group is determined. Output the fault handling suggestions corresponding to the predicted fault causes.

6. The method according to claim 1, further comprising: Based on the attributes of the failed storage drives in the target storage group, a fault knowledge base related to the failed storage drives is obtained, and the fault knowledge base includes relevant information of other storage drives with the same attributes as the failed storage drives. The predicted fault causes corresponding to the target storage group are determined based on the fault knowledge base. Output the fault handling suggestions corresponding to the predicted fault causes.

7. The method according to claim 1, further comprising: Based on the number range to which the number of faults in the target storage group belongs, a fault handling strategy corresponding to the number range is executed on the storage drives contained in the target storage group, wherein different fault handling strategies correspond to different number ranges.

8. The method according to claim 7, wherein the fault quantity information includes a cumulative fault quantity, and the execution of a fault handling strategy corresponding to the number range to which the storage drives included in the target storage group belong includes at least one of the following: If the cumulative number of faults is greater than the second threshold and less than or equal to the third threshold, the hot data stored in the storage drive of the target storage group whose access frequency meets the first condition will be migrated to the storage drive of the non-target storage group. If the cumulative number of faults is greater than the third threshold and less than or equal to the fourth threshold, reduce the frequency of storing data to the storage driver of the target storage group; If the cumulative number of failures exceeds the fourth threshold, the storage driver for the target storage group is removed from the storage cluster; The cumulative number of failures refers to the number of storage drives that have failed within the cumulative time period up to the current moment in the corresponding storage group, and the second threshold, the third threshold, and the fourth threshold increase sequentially.

9. The method of claim 8, wherein removing the storage drive of the target storage group from the storage cluster comprises at least one of the following: Based on the total amount of data stored in the storage drives of the target storage group and the available capacity of the storage cluster, determine the maximum number of drives to be removed, and remove the maximum number of storage drives in the target storage group. Remove a portion of the storage drives from the target storage group, and remove a portion of the storage drives from the target storage group again if a storage drive in the target storage group fails; in, The available capacity of the storage cluster represents the amount of data that other storage drives in the storage cluster, excluding the target storage group, can store.

10. A storage cluster, comprising a storage controller and multiple storage drives; The plurality of storage drives are used to store data; The storage controller is used to perform: Obtain the failure count information of storage groups in the storage cluster. Each storage group includes multiple storage drives with the same attributes in the storage cluster. The failure count information represents the number of storage drives that have failed in the corresponding storage group. The storage cluster includes multiple storage drives. If the number of failures in any storage group meets the warning conditions, the failure prediction information of the storage cluster is output. The failure prediction information indicates that the storage driver in the storage cluster belonging to the target storage group has a failure risk. The target storage group refers to the storage group whose failure number information meets the warning conditions.

Images