Method, electronic device and computer program product for storing failure predictions

Abstract

Embodiments of the present disclosure relate to a method, an electronic device and a computer program product for storing failure prediction. In the method for storing failure prediction provided by embodiments of the present disclosure, an accuracy rate of failure predictions in a past period of time is obtained, each failure prediction is predicted based on a comparison of a prediction confidence with a confidence threshold, and the accuracy rate indicates an amount of work to be performed for reconstruction and diagnosis of a predicted failure storage after the failure prediction; the confidence threshold is adjusted in response to the accuracy rate being less than an accuracy rate threshold; a detection rate of failure predictions in the past period of time is obtained; and the confidence threshold is inversely adjusted in response to the detection rate being less than a detection rate threshold. In this way, the reliability of storage in the node is ensured while reducing unnecessary reconstruction and diagnosis.

Description

Technical Field

[0001] Embodiments of this disclosure relate to the field of data storage, and more specifically, to methods, electronic devices, and computer program products for storing fault prediction. Background Technology

[0002] With the continuous development of computer technology, people are increasingly reliant on server-side storage capabilities. If any storage component in a server fails, it can lead to data loss or service interruption, causing significant disruption to users. Machine learning-based storage failure prediction is known to exist, implemented on a single node in a clustered computing system. This node is responsible for collecting storage health information from other nodes. Based on the collected information, this node predicts storage failures on other nodes and returns the prediction results to each node. Summary of the Invention

[0003] In a first aspect of this disclosure, a method for predicting storage failures is provided. The method includes obtaining the accuracy of failure predictions over a past period, each failure prediction being made based on a prediction confidence level compared to a confidence threshold, the accuracy indicating the workload required to refactor and diagnose the predicted faulty storage after the failure prediction. The method further includes adjusting a confidence threshold in response to an accuracy level falling below an accuracy threshold. The method also includes obtaining the detection rate of failure predictions over a past period. The method further includes reversibly adjusting the confidence threshold in response to a detection rate falling below a detection rate threshold.

[0004] In a second aspect of this disclosure, an electronic device is provided. The electronic device includes a processor and a memory coupled to the processor, the memory having instructions stored therein, the instructions causing the device to perform actions when executed by the processor. The actions include acquiring the accuracy of fault predictions over a past period, each fault prediction being predicted based on a comparison of a prediction confidence level with a confidence threshold, the accuracy indicating the workload required to reconstruct and diagnose predicted faulty memory after fault prediction. The actions also include adjusting a confidence threshold in response to an accuracy level falling below an accuracy threshold. The actions further include acquiring the detection rate of fault predictions over a past period. The actions also include reversibly adjusting the confidence threshold in response to a detection rate falling below a detection rate threshold.

[0005] In a third aspect of this disclosure, a computer program product is provided, which is tangibly stored on a computer-readable medium and includes machine-executable instructions that, when executed, cause a machine to perform the method according to the first aspect.

[0006] The summary section is provided to present the chosen concepts in a simplified form, which will be further described in the detailed description below. The summary section is not intended to identify key or principal features of this disclosure, nor is it intended to limit the scope of this disclosure. Attached Figure Description

[0007] The above and other objects, features, and advantages of this disclosure will become more apparent from the more detailed description of exemplary embodiments thereof, taken in conjunction with the accompanying drawings, in which like reference numerals generally denote like parts. In the drawings:

[0008] Figure 1 A schematic diagram of an example environment in which embodiments of the present disclosure may be implemented is shown;

[0009] Figure 2 A flowchart illustrating an example method for storing fault prediction according to embodiments of the present disclosure is shown;

[0010] Figure 3 This is a schematic diagram illustrating the relationship between accuracy and confidence threshold, and the relationship between detection rate and confidence threshold, according to some embodiments of the present disclosure; and

[0011] Figure 4 A block diagram of an example device that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0012] The principles of this disclosure will now be described with reference to several exemplary embodiments illustrated in the accompanying drawings. While preferred embodiments of this disclosure are shown in the drawings, it should be understood that these embodiments are described only to enable those skilled in the art to better understand and implement this disclosure, and are not intended to limit the scope of this disclosure in any way.

[0013] The term "comprising" and its variations as used herein signify open inclusion, i.e., "including but not limited to". Unless otherwise stated, the term "or" means "and / or". The term "based on" means "at least partially based on". The terms "one example embodiment" and "one embodiment" mean "at least one example embodiment". The term "another embodiment" means "at least one additional embodiment". The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0014] Generally, each node in a cluster is equipped with Self-Monitoring Analysis and Reporting Technology (SMART) to monitor and report various storage reliability metrics. One node in the cluster is responsible for collecting SMART data from all nodes in the cluster for predicting storage failures on those nodes.

[0015] Furthermore, to verify whether the predicted suspicious storage is indeed faulty, read-write tests are needed to diagnose it. In practice, if the suspicious storage is diagnosed first and then rebuilt, only read tests can be run, not write tests. This is because write operations would disrupt the data on the suspicious storage. Therefore, after predicting a suspected faulty storage, a rebuild should be performed first to migrate the data from the suspicious storage. Only then should the diagnosis be run.

[0016] The higher the accuracy of fault prediction, the fewer refactoring and diagnostic runs are required. Typically, fault prediction is based on comparing a predicted value to a preset threshold. This preset threshold is fixed. However, over time or with storage maintenance, the number of failed storage units may increase or decrease. Therefore, using a fixed threshold for storage fault prediction will affect both the accuracy and detection rate.

[0017] Embodiments of this disclosure propose a scheme for storage fault prediction in which the confidence threshold for fault prediction varies with a predetermined accuracy threshold and a predetermined detection rate threshold. This achieves storage fault prediction based on an adaptive confidence threshold.

[0018] According to various embodiments of this disclosure, the accuracy of fault prediction for storage over a past period is obtained, and the accuracy indicates the workload required to reconstruct and diagnose the predicted faulty storage after fault prediction. A confidence threshold for fault prediction is adjusted in response to an accuracy rate lower than the accuracy threshold. Furthermore, the detection rate of fault prediction over a past period is also obtained. In response to a detection rate lower than the detection rate threshold, the aforementioned confidence threshold is adjusted in reverse.

[0019] According to the embodiments described herein, by adjusting the confidence threshold, the accuracy of fault prediction can reach the user's desired accuracy. This reduces unnecessary refactoring and diagnostics, thus saving system resources and reducing computational costs. Furthermore, by adjusting the confidence threshold inversely, the detection rate of fault prediction can also reach the user's desired threshold, thereby ensuring the reliability of storage in the node.

[0020] The basic principles and several exemplary implementations of this disclosure will be described below with reference to the accompanying drawings. It should be understood that these exemplary embodiments are given only to enable those skilled in the art to better understand and implement the embodiments of this disclosure, and are not intended to limit the scope of this disclosure in any way.

[0021] Figure 1 A schematic diagram of an example environment in which embodiments of this disclosure may be implemented is shown. For example... Figure 1 As shown, environment 100 is a cluster computing environment, including nodes 110-1, 110-2, 110-3, and 110-N (which can also be collectively referred to as "node 110" or individually) that are interconnected and communicate with each other, where N is a natural number. Node 110 can be a server that provides storage services to users.

[0022] Node 110-1 includes storage 120-1, node 110-2 includes storage 120-2, node 110-3 includes storage 120-3, and node 110-N includes storage 120-M. Storage 120-1, storage 120-2, storage 120-3, and storage 120-M can also be collectively referred to as "Storage 120" or individually, where M is a natural number. Storage 120 can be a hard disk drive, a removable disk drive, or other types of disk drives. Furthermore, storage 120 can be a single disk or a Redundant Array of Independent Disks (RAID) that combines multiple disks.

[0023] Each storage unit 120 is equipped with Self-Monitoring, Analysis, and Reporting Technology (SMART) to monitor and report various storage reliability metrics. Each node 110 can include or deploy a fault predictor. The fault predictor can utilize SMART data to predict storage failures. Therefore, any node in the 110 can act as a proxy node in the cluster, collecting SMART data from the storage units 120 across the nodes 110 and performing fault prediction for each storage unit 120. Figure 1 Only the fault predictor 130 located in node 110-1 is shown. Node 110-1 acts as a proxy node in environment 100.

[0024] For example, fault predictor 130 can predict the failure of storage 120-2 based on SMART data collected by node 110-1 from node 110-2. The output of fault predictor 130 is a prediction confidence score. This prediction confidence score ranges from [0, 1]. This prediction confidence score needs to be compared with a confidence threshold to determine whether storage 120-2 is predicted to be faulty. If storage 120-2 is predicted to be faulty, it needs to be refactored. Then, known fault diagnosis techniques can be used to diagnose whether storage 120-2 is actually faulty.

[0025] It should be understood that, in addition to fault predictors that utilize SMART data for prediction, other fault prediction tools can also be used to predict storage failures. The scope of this disclosure is not limited in this respect as long as the prediction tool used outputs the prediction results in the form of confidence levels. Furthermore, the prediction results can also be expressed as percentage values ​​(i.e., prediction confidence levels), for example, [0%, 100%].

[0026] Furthermore, it should be understood that the structure and function of environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure. For example, embodiments of this disclosure can also be applied to environments different from environment 100. Additionally, although Figure 1 Only four nodes are shown, but it is not limited to this; there can be more or fewer nodes. Although Figure 1 The image only shows one storage unit per node, but it is not limited to this; each node can also include more storage units.

[0027] Figure 2 A flowchart of an example method 200 for storing fault prediction according to an embodiment of the present disclosure is shown. Method 200 can be implemented, for example, in a cluster computing environment (e.g., Figure 1 The method is executed in an agent node (e.g., node 110-1) within the environment 100 shown. It should be understood that method 200 may also include additional actions not shown and / or the actions shown may be omitted; the scope of this disclosure is not limited in this respect. The following is in conjunction with… Figure 1 Let me describe method 200 in detail.

[0028] At box 210, node 110-1 obtains the accuracy of fault prediction over a past period. For example, node 110-1 may obtain the accuracy of fault predictions made by fault predictor 130 for multiple storage units 120 over the past week. This period can also be any suitable time frame, such as a day, several days, several weeks, a month, or several months, and the scope of this disclosure is not limited in this respect. As previously stated, each fault prediction made by fault predictor 130 for the multiple storage units 120 is based on a comparison of prediction confidence with a confidence threshold. If fault predictor 130 is configured to predict storage unit 120 as faulty in response to a prediction confidence exceeding the confidence threshold, then the relationship between accuracy and the confidence threshold can be as follows: Figure 3 As shown.

[0029] Figure 3 This is a schematic diagram illustrating the relationship between accuracy and confidence threshold curve 310 and the relationship between detection rate and confidence threshold curve 320 according to some embodiments of the present disclosure. Figure 3 The horizontal axis represents the confidence threshold, and the vertical axis represents the accuracy and detection rate as percentage values.

[0030] exist Figure 3 In this context, accuracy increases with increasing confidence threshold, while detection rate decreases with increasing confidence threshold. It should be understood that if fault predictor 130 is configured to predict storage 120 as faulty in response to a prediction confidence level not exceeding a confidence threshold, then accuracy will decrease with increasing confidence threshold, and detection rate will increase with increasing confidence threshold.

[0031] Because refactoring and diagnostics are required for storage 120 predicted to fail to determine whether it is truly faulty, accuracy indicates the workload of refactoring and diagnosing the predicted faulty storage after fault prediction. Figure 3 In the illustrated embodiment, the higher the confidence threshold, the higher the accuracy, the lower the detection rate, and the less work is required for reconstruction and diagnosis after fault prediction.

[0032] In embodiments of this disclosure, the user can set and / or adjust the accuracy threshold and the detection rate threshold as needed. In response to an accuracy rate being less than the accuracy threshold over a past period (e.g., within a week), the confidence threshold can be adjusted, such as... Figure 2 As shown in box 220.

[0033] For example, assuming the current confidence threshold is 0.7, the accuracy of Fault Predictor 130 over the past week was 76%. Figure 3In the illustrated embodiment, the accuracy threshold is set to 80%, which is greater than the accuracy over the past week. Therefore, the confidence threshold needs to be increased. The 80% accuracy threshold corresponds to point 301 on curve 310. The x-coordinate of point 301 (i.e., the confidence threshold) is approximately 0.74. Therefore, it is sufficient to adjust the confidence threshold to be greater than or equal to 0.74.

[0034] return Figure 2 In box 230, the detection rate of fault prediction over a past period (e.g., one week) can also be obtained. Then, in box 240, the confidence threshold is adjusted in reverse in response to the detection rate being less than the detection rate threshold.

[0035] For example, assuming the reliability threshold is adjusted to 0.9 at box 220, the detection rate over the past week is 55%. Figure 3 In the illustrated embodiment, the detection rate threshold is set to 60%, which is greater than the detection rate over the past week. Therefore, the confidence threshold needs to be lowered. The 60% detection rate threshold corresponds to point 302 on curve 320. The x-coordinate of point 302 (i.e., the confidence threshold) is approximately 0.87. Therefore, in Figure 3 In the illustrated embodiment, the confidence threshold can be adjusted to any value greater than or equal to 0.74 and less than or equal to 0.87.

[0036] It should be understood that the values ​​of the accuracy threshold and the detection rate threshold are merely exemplary and are not intended to limit the scope of this disclosure. In some other embodiments, other suitable values ​​may be used as the accuracy threshold and the detection rate threshold. Similarly, the accuracy and detection rate values ​​described above are merely exemplary and are not intended to limit the scope of this disclosure. In some other embodiments, the obtained accuracy and detection rate may be other suitable values.

[0037] Similarly, the confidence threshold values ​​described above are merely exemplary and are not intended to limit the scope of this disclosure. The confidence threshold of this disclosure can range from (0, 1). In some embodiments, the default value of the confidence threshold can be set to 0.5, and the value range is within (0.5, 1).

[0038] When the fault predictor 130 is configured to predict storage 120 as faulty in response to a prediction confidence level not exceeding a confidence threshold, the confidence threshold can be lowered in response to an accuracy level less than an accuracy threshold; and the confidence threshold can be raised in response to a detection rate less than a detection rate threshold.

[0039] In this way, even if the amount of storage failure increases or decreases over time or due to storage maintenance, the confidence threshold can adaptively adjust with changes in the accuracy and / or detection rate thresholds, thus ensuring that the accuracy and / or detection rate meet the user's expectations. This reduces unnecessary refactoring and diagnostics, thereby saving system resources and reducing computational costs while ensuring storage reliability.

[0040] In some embodiments, obtaining the detection rate may include: obtaining a first number of fault stores identified by diagnosis over a past period of time (e.g., one week); obtaining a second number of fault stores reported by processes other than fault prediction over a past period of time (e.g., one week); and calculating the detection rate by dividing the first number by the sum of the first number and the second number.

[0041] The detection rate indicates the proportion of faulty memory that is predicted out of all faulty memory. However, the total number of faulty memory is difficult to know. Therefore, the total number of faulty memory can be calculated by summing the first number of memory identified as faulty over a period of time with the second number of memory faults (e.g., I / O errors) detected by other programs or processes during that period. The proportion of the first number to the total number is the detection rate.

[0042] Figure 4 A schematic block diagram of an example device 400 that can be used to implement embodiments of the present disclosure is shown. For example, such as Figure 1 The cloud server 110 shown can be implemented by device 400. For example... Figure 4 As shown, device 400 includes a central processing unit (CPU) 401, which can perform various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) 402 or loaded from storage unit 408 into random access memory (RAM) 403. RAM 403 may also store various programs and data required for the operation of device 400. CPU 401, ROM 402, and RAM 403 are interconnected via bus 404. Input / output (I / O) interface 405 is also connected to bus 404.

[0043] Multiple components in device 400 are connected to I / O interface 405, including: input unit 406, such as keyboard, mouse, etc.; output unit 407, such as various types of monitors, speakers, etc.; storage unit 408, such as disk, optical disk, etc.; and communication unit 409, such as network card, modem, wireless transceiver, etc. Communication unit 409 allows device 400 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0044] The various processes and handling described above, such as method 200, can be executed by processing unit 401. For example, in some embodiments, method 200 can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 408. In some embodiments, part or all of the computer program can be loaded and / or installed on device 400 via ROM 402 and / or communication unit 409. When the computer program is loaded into RAM 403 and executed by CPU 401, one or more actions of method 200 described above can be performed.

[0045] This disclosure can be a method, apparatus, system, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of this disclosure.

[0046] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0047] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0048] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0049] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0050] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0051] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0052] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0053] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for predicting computer storage failures, comprising: Multiple fault predictions stored in the computer over a period of time are generated using reliability metrics stored in the computer. Each fault prediction is based on a comparison between the prediction confidence level and a confidence level threshold. Determine the accuracy of the multiple fault predictions over the time period, the accuracy indicating the workload of reconstructing and diagnosing the predicted faulty computer storage after the fault prediction; The confidence threshold is adjusted in response to the accuracy being less than the accuracy threshold; Determine the detection rate of the fault prediction within the specified time period; In response to the detection rate being less than the detection rate threshold, the confidence threshold is adjusted in reverse; and In response to the multiple fault predictions for the computer storage, data is migrated out of the computer storage.

2. The method according to claim 1, wherein determining the detection rate comprises: Obtain the first number of faulty computers identified by diagnosis within the specified time period; Obtain a second number of faulty computers stored in the process report other than the fault prediction during the aforementioned period; The detection rate is calculated by dividing the first number by the sum of the first number and the second number.

3. The method according to claim 1, wherein the default value of the confidence threshold is 0.

5.

4. The method according to claim 3, wherein the confidence threshold is not less than 0.5 and less than 1.

5. An electronic device, comprising: processor; as well as A memory coupled to the processor, the memory having instructions stored therein, the instructions causing the device to perform actions when executed by the processor, the actions including: Multiple fault predictions stored in the computer over a period of time are generated using reliability metrics stored in the computer. Each fault prediction is based on a comparison between the prediction confidence level and a confidence level threshold. Determine the accuracy of the multiple fault predictions over the time period, the accuracy indicating the workload of reconstructing and diagnosing the predicted faulty computer storage after the fault prediction; The confidence threshold is adjusted in response to the accuracy being less than the accuracy threshold; Determine the detection rate of the fault prediction within the specified time period; In response to the detection rate being less than the detection rate threshold, the confidence threshold is adjusted in reverse; and In response to the multiple fault predictions for the computer storage, data is migrated out of the computer storage.

6. The device according to claim 5, wherein determining the detection rate comprises: Obtain the first number of faulty computers identified by diagnosis within the specified time period; Obtain a second number of faulty computers stored in the process report other than the fault prediction during the aforementioned period; The detection rate is calculated by dividing the first number by the sum of the first number and the second number.

7. The device according to claim 5, wherein the default value of the confidence threshold is 0.

5.

8. The device according to claim 7, wherein the confidence threshold is not less than 0.5 and less than 1.

9. A computer program product tangibly stored on a computer-readable medium and comprising machine-executable instructions that, when executed, cause a machine to perform the method according to any one of claims 1 to 4.

Images