Detecting and predicting electronic storage device data anomalies

Abstract

A computer-implemented method, system, and computer program product for device fault detection are disclosed. In the method, stage-based prediction may be performed for a plurality of storage devices to determine a plurality of sampling ranges and corresponding sampling rates. A respective sampling range may include at least one storage device of the plurality of storage devices. A sampled data set may be obtained by selecting a set of storage devices from respective sampling ranges with corresponding sampling rates. A device failure of the set of storage devices may be detected based on the sampled data set.

Description

BACKGROUND

[0001] The present invention relates to data processing, and more specifically, to detecting anomalies in electronic storage devices.

[0002] For enterprises / organizations, data is an asset and grows at an exponential rate. Data is typically stored in storage devices such as hard disk drives (HDDs), solid state drives (SSDs), tapes on-site or in the cloud, etc. However, failure of storage devices results in many negative impacts such as data loss, unavailability of services, additional operational costs, economic losses, etc. SUMMARY

[0003] This Summary is provided to introduce some concepts in a simplified form that are further described below in the DETAILED DESCRIPTION. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used in limiting the scope of the claimed subject matter.

[0004] According to one embodiment of the present invention, a computer-implemented method for device failure detection is provided. In the method, a stage-based prediction can be performed for a plurality of storage devices to determine a plurality of sampling ranges and corresponding sampling rates. A respective sampling range can include at least one storage device of the plurality of storage devices. A sampled data set can be obtained by selecting a set of storage devices from the respective sampling range with the corresponding sampling rate. A device failure of the set of storage devices can be detected based on the sampled data set.

[0005] Accordingly, an effective balance of accuracy, performance, and cost of failure detection of storage devices can be provided.

[0006] In some embodiments, performing the stage-based prediction for the plurality of storage devices can include performing the stage-based prediction based on real-time monitoring data associated with the plurality of storage devices and a plurality of models. The respective models can be trained with benchmark data and historical monitoring data associated with the plurality of storage devices. Accordingly, historical monitoring data and benchmark data, which includes open data for specific manufacturers, models, and batches via supervised or unsupervised algorithms, can be employed to facilitate detection.

[0007] In some embodiments, the stage-based prediction includes at least two prediction stages. The prediction of a next stage is performed based on a result of the prediction of a previous stage. Accordingly, an appropriate feature can be utilized to reduce a prediction range and cost.

[0008] In some embodiments, performing stage-based prediction for the plurality of storage devices further comprises the following stages. In a first stage, environmental anomalies can be predicted for the plurality of storage devices to determine a first sampling range with a first sampling rate. The first sampling range comprises storage devices in a normal environment. In a second stage, performance anomalies of storage devices in an abnormal environment can be predicted to determine a second sampling range with a second sampling rate. The second sampling range comprises storage devices that are in an abnormal environment and perform normally. In a third stage, device monitoring data anomalies are predicted for storage devices that are in an abnormal environment and perform abnormally, determining a third sampling range with a third sampling rate and a fourth sampling range with a fourth sampling rate. The third range comprises storage devices that are in an abnormal environment, perform abnormally, and have normal device monitoring data. The fourth range comprises storage devices that are in an abnormal environment, perform abnormally, and have abnormal device monitoring data. Further, the first sampling rate is lower than the second sampling rate, the second sampling rate is lower than the third sampling rate, and the third sampling rate is lower than the fourth sampling rate. Thus, different sampling ranges can be assigned different sampling rates to filter high-risk devices for more detailed device failure prediction.

[0009] In some embodiments, the steps of performing, obtaining, and detecting can be implemented multiple times, wherein the step of performing is scheduled based on a scheduling policy. Thus, the prediction can be performed on-demand based on a dynamic scheduling policy and recent detection results.

[0010] In some embodiments, a failure library can be generated based on the detected device failures. Thus, a failure library can be established and maintained to save detected anomalies for proactive sampling.

[0011] In some embodiments, a scheduling requirement can be evaluated based on the failure library, benchmark data, and historical monitoring data. A scheduling policy can be selected based on the scheduling requirement. Thus, the prediction can be dynamically adjusted based on actual needs.

[0012] In some embodiments, the respective models are scheduled for update based on the scheduling requirement. Thus, the individual models can be dynamically updated based on actual needs.

[0013] In some embodiments, the step of detecting device failures for the set of storage devices based on the sampling dataset can comprise detecting device failures based on real-time monitoring data associated with the set of storage devices and a plurality of device failure prediction models. The respective device failure prediction models are trained with benchmark data and historical monitoring data associated with the plurality of storage devices. Thus, historical monitoring data and benchmark data can be employed to facilitate detection, the benchmark data comprising open data for specific manufacturers, models, and batches via supervised or unsupervised algorithms.

[0014] According to another embodiment of the present disclosure, a system for device failure detection is provided. The system can include one or more processors, a memory coupled to at least one of the one or more processors, and a set of computer program instructions stored in the memory. The set of computer program instructions can be executable by at least one of the one or more processors to perform the above-described method.

[0015] According to another embodiment of the present disclosure, a computer program product for device failure detection is provided. The computer program product can include a computer readable storage medium having program instructions embodied therin. The program instructions, which can be executable by one or more processors, cause the one or more processors to perform the above-described method.

[0016] In addition to the exemplary aspects and embodiments described above, others will become apparent to those of ordinary skill in the art by reading the foregoing description, by inspecting the accompanying drawings, and by practicing the disclosure. There can be many alterations made to carry out the disclosure in BRIEF DESCRIPTION OF DRAWINGS

[0017] The above and other objects, features and advantages of the present disclosure will become more apparent from the following description when taken in conjunction with the accompanying drawings, in which like reference characters refer to like elements throughout the figures, and in which:

[0018] Figure 1 is an example computing environment suitable for implementing embodiments of the present disclosure.

[0019] Figure 2 is an example device failure detection system according to embodiments of the present disclosure.

[0020] Figure 3 is an example process for device failure detection according to embodiments of the present disclosure.

[0021] Figure 4 shows an example process for stage-based prediction according to embodiments of the present disclosure.

[0022] Figure 5 shows an example block diagram of a scheduler module according to embodiments of the present disclosure.

[0023] Figure 6 shows an example flowchart of a computer-implemented method for device failure detection according to embodiments of the present disclosure. DETAILED DESCRIPTION

[0024] Various aspects of the present disclosure are described by way of illustrative text, flowcharts, block diagrams of computer systems, and / or block diagrams of machine logic included in computer program product (CPP) embodiments. With respect to any flowcharts, depending on the technology involved, the operations can be performed in a different sequence than shown in the given flowchart. For example, again depending on the technology involved, two operations shown in consecutive flowchart blocks can be performed in reverse order, as single integrated steps, concurrently, or in at least partial temporal overlap.

[0025] A computer program product embodiment (“CPP embodiment” or “CPP”) is a term used in the present disclosure to describe any collection of one or more storage media (also referred to as “media”) collectively including machine-readable code corresponding to instructions and / or data used to perform computer operations specified in a given CPP claim, collectively included in a set of one or more storage devices. A “storage device” is any tangible device that can hold and store instructions used by a computer processor. Without limitation, computer-readable storage media can be an electronic storage media, a magnetic storage media, an optical storage media, an electromagnetic storage media, a semiconductor storage media, a mechanical storage media, or any suitable combination of the foregoing. Some known types of storage devices that include these media include: a magnetic disk, a hard drive, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), a memory stick, a floppy disk, a mechanical encoding device such as punch cards or punch holes / platforms formed in the main surface of a disk, or any suitable combination of the foregoing. Computer-readable storage media, as the term is used in the present disclosure, is not to be construed as being memory in the form of a transitory signal per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide, light pulses through an optical cable, electrical signals through a wire, and / or other transmission media. As will be appreciated by one of skill in the art, data is typically moved at some incidental point in time during normal operation of a storage device, such as during access, defragmentation, or garbage collection, but this does not make the storage device transitory, as the data is not transitory while it is stored.

[0026] The computing environment 100 includes an example of an environment for executing at least some computer code involved in performing the methods of the present invention, such as the equipment failure detection system 200. In addition to the block 200, the computing environment 100 includes, for example, a computer 101, a wide area network (WAN) 102, an end user device (EUD) 103, a remote server 104, a public cloud 105, and a private cloud 106. In this embodiment, the computer 101 includes a processor group 110 (including a processing circuit 120 and a cache 121), a communication fabric 111, a volatile memory 112, a persistent storage 113 (including an operating system 122 and the block 200, as described above), a peripheral device group 114 (including a user interface (UI), a device group 123, a storage device 124, and an Internet of Things (IoT) sensor set 125), and a network module 115. The remote server 104 includes a remote database 130. The public cloud 105 includes a gateway 140, a cloud coordination module 141, a host physical machine group 142, a virtual machine group 143, and a container group 144.

[0027] The computer 101 can take the form of a desktop computer, a laptop computer, a tablet computer, a smart phone, a smart watch or other wearable computer, a mainframe computer, a quantum computer, or any other form of computer or mobile device now known or hereafter developed that is capable of running a program, accessing a network, or querying a database such as the remote database 130. As is well known in the computer arts, and depending on the technology, the performance of the computer-implemented methods can be distributed among multiple computers and / or among multiple locations. On the other hand, in this presentation of the computing environment 100, the detailed discussion is focused on a single computer, particularly the computer 101, to keep the presentation as simple as possible. Even in the case where the performance of the computer-implemented methods is distributed among multiple computers and / or among multiple locations, the computer 101 can be one of the computers involved in the distributed performance. Figure 1 The computer 101 can be located in the cloud, although not shown in the cloud in FIG. 1. On the other hand, the computer 101 need not be in the cloud, unless it can be affirmatively indicated to that extent.

[0028] The processor group 110 includes one or more computer processors of any type now known or hereafter developed. The processing circuit 120 can be distributed over multiple packages, for example, multiple cooperating integrated circuit chips. The processing circuit 120 can implement multiple processor threads and / or multiple processor cores. The cache 121 is memory located in the processor chip package and is typically used for data or code that should be quickly accessible to threads or cores running on the processor group 110. The cache memory is typically organized into multiple levels according to relative proximity to the processing circuit. Alternatively, some or all of the cache in the processor group can be located “off-chip.” In some computing environments, the processor group 110 can be designed to work with qubits and perform quantum computations.

[0029] Computer readable program instructions generally be loaded onto the computer 101 to cause the processor complex 110 of the computer 101 to perform a series of operational steps to implement the computer-implemented method such that the instructions so executed will instantiate the method specified in the flow charts and / or the narrative description of the computer-implemented method included in this document (collectively, the "inventive method"). These computer readable program instructions are stored in various types of computer readable storage media such as the cache 121 and other storage media discussed below. The program instructions and related data are accessed by the processor complex 110 to control and guide the execution of the inventive method. In the computing environment 100, at least some of the instructions for performing the inventive method can be stored in the persistent storage 113 in block 200.

[0030] The communication fabric 111 is a signal carrying pathway that allows the various components of the computer 101 to communicate with each other. Typically, this fabric is composed of switches and conductive pathways such as those that make up a bus, a bridge, a physical input / output port, and the like. Other types of signal communication pathways can be used, for example, fiber optic communication pathways and / or wireless communication pathways.

[0031] The volatile memory 112 is any type of volatile memory now known or developed in the future. Examples include dynamic random access memory (RAM) or static RAM. Typically, volatile memory is characterized by random access, but this is not required unless affirmatively indicated. In the computer 101, the volatile memory 112 is located in a single package and is internal to the computer 101, but, alternatively or additionally, the volatile memory can be distributed in multiple packages and / or located externally with respect to the computer 101.

[0032] The persistent storage 113 is any form of non-volatile storage for a computer now known or developed in the future. The non-volatile nature of this storage means that the stored data is retained regardless of whether power is supplied to the computer 101 and / or directly to the persistent storage 113. The persistent storage 113 can be read only memory (ROM), but typically at least a portion of the persistent storage allows for writing of data, deletion of data, and re-writing of data. Some common forms of persistent storage include magnetic disks and solid state storage devices. The operating system 122 can take several forms such as various known proprietary operating systems or an open source portable operating system interface type of operating system employing a kernel. The code included in block 200 typically includes at least some of the computer code involved in performing the inventive method.

[0033] The peripheral device group 114 includes a group of peripheral devices of the computer 101. Data communication connections between the peripheral devices and other components of the computer 101 can be achieved by various means, such as Bluetooth connections, near-field communication (NFC) connections, connections made by cables such as universal serial bus (USB) type cables, plug-in type connections (e.g., secure digital (SD) cards), connections through local area communication networks, and even connections through wide area networks such as the Internet. In various embodiments, the UI device set 123 can include components such as display screens, speakers, microphones, wearable devices such as goggles and smart watches, keyboards, mice, printers, touch pads, game controllers, and haptic devices. The storage device 124 is an external storage device, for example, an external hard drive, or is a pluggable storage device, for example, an SD card. The storage device 124 can be permanent and / or volatile. In some embodiments, the storage device 124 can take the form of a quantum computing storage device for storing data in the form of qubits. In embodiments where the computer 101 needs to have a large amount of storage (e.g., in cases where the computer 101 stores and manages a large database locally), then the storage device can be provided by a peripheral storage device designed for storing very large amounts of data, such as a storage area network (SAN) shared by multiple geographically distributed computers. The IoT sensor group 125 is made up of sensors that can be used in Internet of Things applications. For example, one sensor can be a thermometer, while another sensor can be a motion detector.

[0034] The network module 115 is a collection of computer software, hardware, and firmware that allows the computer 101 to communicate with other computers over the WAN 102. The network module 115 can include hardware such as a modem or Wi-Fi signal transceiver, software for packetizing and / or depacketizing data for transmission over a communication network, and / or web browser software for transmitting data over the Internet. In some embodiments, the network control functions and network forwarding functions of the network module 115 are performed on the same physical hardware device. In other embodiments (e.g., embodiments that utilize software defined networking (SDN)), the control functions and forwarding functions of the network module 115 are performed on physically separate devices, such that the control functions manage several different network hardware devices. Computer readable program instructions for performing the methods of the present application can generally be downloaded to a computer 101 from an external computer or external storage device through a network adaptation card or network interface included in the network module 115.

[0035] WAN 102 is any wide area network (e.g., the Internet) capable of transmitting computer data over non-local distances through any technology now known or later developed for transmitting computer data. In some embodiments, the WAN can be replaced and / or supplemented by a local area network (LAN), which is designed to transmit data between devices located in a local area, such as a Wi-Fi network. WANs and / or LANs typically include computer hardware such as copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers and edge servers.

[0036] End user device (EUD) 103 is any computer system used and controlled by an end user (e.g., a customer of the business operating computer 101), and can take any form discussed above in connection with computer 101. EUD 103 typically receives helpful and useful data from the operation of computer 101. For example, in the hypothetical case where computer 101 is designed to provide recommendations to end users, the recommendations would typically be transmitted from network module 115 of computer 101 to EUD 103 over WAN 102. In this manner, EUD 103 can display or otherwise present the recommendations to the end user. In some embodiments, EUD 103 can be a client device, such as a thin client, a thick client, a mainframe computer, a desktop computer, etc.

[0037] Remote server 104 is any computer system that provides at least some data and / or functionality to computer 101. Remote server 104 can be controlled and used by the same entity operating computer 101. Remote server 104 represents a machine that collects and stores helpful and useful data for use by other computers, such as computer 101. For example, in the hypothetical case where computer 101 is designed and programmed to provide recommendations based on historical data, then that historical data can be provided to computer 101 from remote database 130 of remote server 104.

[0038] The public cloud 105 is any computer system that is usable by multiple entities, which provides on-demand availability of computer system resources and / or other computer capabilities, particularly data storage (cloud storage) and computing capabilities, without direct active management by the user. Cloud computing generally utilizes sharing of resources to achieve consistency of scale and economy. Direct and active management of the computing resources of the public cloud 105 is performed by computer hardware and / or software of the cloud coordination module 141. The computing resources provided by the public cloud 105 are generally implemented by virtual computing environments running on various computers that make up the set of host physical machines 142, which is the universe of physical computers in and / or available to the public cloud 105. The virtual computing environments (VCEs) generally take the form of virtual machines from the set of virtual machines 143 and / or containers from the set of containers 144. It will be appreciated that these VCEs can be stored as images and can be transferred as images or between various physical machine hosts after instantiation of the VCE. The cloud coordination module 141 manages the transfer and storage of the images, deploys new instantiations of VCEs, and manages the active instantiations of VCE deployments. The gateway 140 is a collection of computer software, hardware, and firmware that allows the public cloud 105 to communicate over the WAN 102.

[0039] Some further explanation of virtualized computing environments (VCEs) will now be provided. VCEs can be stored as “images.” New active instances of a VCE can be instantiated from the image. Two common types of VCEs are virtual machines and containers. Containers are VCEs that use operating system-level virtualization. This refers to an operating system feature in which the kernel allows multiple isolated user space instances, called containers, to exist. From the perspective of the programs running in them, these isolated user space instances generally behave as actual computers. Computer programs running on a normal operating system can utilize all of the resources of that computer, such as connected devices, files and folders, network shares, CPU capabilities, and quantifiable hardware capabilities. However, programs running within a container can only use the contents of the container and the devices allocated to the container, which is a feature known as containerization.

[0040] The private cloud 106 is similar to the public cloud 105 except that the computing resources are available only to a single enterprise. Although the private cloud 106 is depicted in communication with the WAN 102, in other embodiments, the private cloud can be completely disconnected from the Internet and only accessible through a local / private network. A hybrid cloud is a combination of multiple clouds of different types (e.g., private, community, or public cloud types) that are typically implemented by different vendors respectively. Each of the multiple clouds remains as a separate and discrete entity, but the larger hybrid cloud architecture is bound together through standardized or proprietary technologies that enable coordination, management, and / or data / application portability between the multiple constituent clouds. In this embodiment, the public cloud 105 and the private cloud 106 are both part of a larger hybrid cloud.

[0041] It is to be understood that Figure 1 The computing environment 100 in FIG. 1 is provided for purposes of illustration and description only and is not intended to imply any limitation of the embodiments of the application, e.g., at least a portion of the program code involved in performing a method of the application can be loaded into the cache 121, volatile memory 112, or other storage means (e.g., storage device 124) of the computer 101, or at least a portion of the program code involved in performing a method of the application can be stored in other local or / and remote computing environments and loaded as needed. As another example, the peripheral devices 114 can also be implemented by separate peripheral devices connected to the computer 101 through an interface. For a further example, the WAN can be replaced and / or supplemented by any other connection to an external computer (e.g., through the Internet using an Internet service provider).

[0042] Generally, due to large-scale monitoring data and large-scale deployment in production, failure detection (or anomaly prediction) of a large number of storage devices is expensive and time-consuming. Methods for predicting device failure can include threshold-based methods (which set a threshold based on selected metrics), statistical-based methods (which construct a statistical model based on selected metrics), learning-based methods (which construct a machine learning or deep learning model based on given features to predict anomalies or lifetimes). However, most existing methods focus more on the accuracy of the method. Few studies address the real challenge of a large number of storage devices with a balance of accuracy, performance, and cost.

[0043] Embodiments of the present disclosure provide a device failure detection system for detecting / predicting anomalies / failures of large-scale storage devices. Based on the embodiments, an effective balance of accuracy, performance, and cost of storage device failure detection can be achieved. The detection range and resources allocated for the detection module can be reduced. In addition, more time and cost for detecting abnormal storage devices can be saved.

[0044] Reference is now made to Figure 2A block diagram showing an exemplary device failure detection system 200 according to some embodiments of the present disclosure is provided.

[0045] It can be noted that the processing of the device failure detection system 200 according to embodiments of the present disclosure can be implemented in a computing environment. Figure 1

[0046] As shown in Figure 2 , in some embodiments, the device failure detection system 200 can include a prediction module 210, an obtaining module 220, a detection module 230. In further embodiments, the device failure detection system 200 can also include a model generation / update module 240, a scheduler module 250, etc. All or some of the modules can be configured to communicate with each other (e.g., via a communication structure 111 as depicted in Figure 1 , such as a bus, shared memory, a switch, or a network). Any one or more of these modules can be implemented using the processing circuitry 120 in Figure 1 (e.g., by configuring the processing circuitry 120 to perform the functions described for that module). It can be noted that the addition, removal, and / or modification of one or more modules can be configured based on actual needs.

[0047] Figure 3 An exemplary process 300 for device failure detection according to embodiments of the present disclosure is depicted. The process 300 can be implemented with the device failure detection system 200 and will be described below in conjunction with Figure 2 .

[0048] At block 310, the prediction module 210 can perform stage-based prediction for a plurality of storage devices to determine a plurality of sampling ranges and corresponding sampling rates. A respective sampling range can include at least one storage device of the plurality of storage devices.

[0049] In some embodiments, the storage device can be at least one of an HDD, an SSD, a memory card, a floppy disk, an optical disk drive (compact disk (CD), digital versatile disk (DCD, Blu-ray DVD)), a RAM, a ROM, etc. Further, the respective storage device can be associated with a set of monitoring data, such as environmental data, performance data, device monitoring data, metadata, and maintenance data, etc.

[0050] ​As an example, the environmental data can include metrics of the operating environment (e.g., server room, row, rack, etc.) of the storage device, such as temperature, humidity, air quality, etc. The performance data can reflect the performance of the applications deployed on the storage device. For example, the performance data can include Input / Output Operations Per Second (IOPS), Mean Time between Failures (MTBF), Mean Time to repair (MTTR), read / write speed, etc. The device monitoring data can include Self-Monitoring Analysis and Reporting Technology (SMART) data, which indicates the metrics of the attributes of the HDDs and SSDs, such as read error rate, start / stop count, drive calibration retry count, etc. For other types of storage devices, other types of device monitoring data known in the art can also be included. Further, the metadata can include the vendor, type, size, age, etc. of the storage device. The maintenance data can include the maintenance logs related to the storage device. It can be appreciated that any other appropriate monitoring data associated with the storage device can also be obtained based on actual needs.

[0051] Accordingly, the prediction module 210 can receive large-scale monitoring data associated with the plurality of storage devices in real-time. The received monitoring data can also be referred to as real-time monitoring data 305. Accordingly, the phase-based prediction can be performed based on the real-time monitoring data 305. Generally, the real-time monitoring data 305 is highly helpful for predicting the storage device failure, especially the SMART data. However, the volume of this real-time monitoring data 305 is too large to determine an effective prediction. In an embodiment, the phase-based prediction can help filter the most useful data for further prediction / detection.

[0052] In some embodiments, the prediction module 210 can perform the phase-based prediction based on the real-time monitoring data by means of a plurality of anomaly prediction models. For example, the anomaly prediction models can include an environmental anomaly prediction model, a performance anomaly prediction model, a device monitoring data anomaly prediction model, etc. Further, the respective anomaly prediction models can be at least one of a classification model, a regression model, a clustering model, a heuristic model. It can be appreciated that any other appropriate model known in the art can also be implemented based on actual needs.

[0053] In an example, the environmental anomaly prediction model can be configured to predict whether the storage device is deployed in an abnormal environment, such as abnormal temperature, abnormal humidity, corrosive gas, etc. For example, thresholds of temperature, humidity, and / or gas amount can be predefined in advance. In another example, the performance anomaly prediction model can be configured to predict whether the storage device performs abnormally, or whether the applications distributed on the storage device have abnormal performance. For example, the performance anomaly of the storage device can be predicted if the read / write speed is relatively slow, if the IOPS is sharply reduced, or if the MTTR is relatively high. For example, corresponding thresholds can be predefined with respect to the read / write speed, IOPS, or MTTR. In yet another example, the device monitoring data anomaly prediction model (e.g., the SMART data anomaly prediction model) can be configured to predict abnormal attributes of the storage device.

[0054] In some embodiments, the anomaly prediction models as described above can be pre-trained and stored in the knowledge base 301. Alternatively, at block 350, the model generation / update module 240 can generate the anomaly prediction models from external benchmark databases and historical monitoring databases of a plurality of storage devices. Specifically, the external benchmark databases can store open source data for storage devices of specific manufacturers, models, and batches via supervised or unsupervised algorithms. Further, the historical monitoring databases can store historical monitoring data 307 associated with the storage devices, such as environmental data, performance data, device monitoring data, metadata, and maintenance data, etc. The historical monitoring data 307 can be similar to the real-time monitoring data, but collected from historical data sampling. Repetitive description of the historical monitoring data can be omitted here. Then, the generated anomaly prediction models can be stored in the knowledge base 301.

[0055] Therefore, the prediction module 210 can access the pre-trained or generated anomaly prediction models from the knowledge base 301 to determine corresponding anomaly predictions.

[0056] Further, the stage-based prediction can also be performed based on a scheduling policy. For example, the scheduling policy can indicate, for a specific storage device, a prediction timing (also referred to as a sampling timing), a prediction frequency (also referred to as a sampling frequency), a proper model to be adopted, an update timing of the corresponding model, etc. For example, the prediction frequency can be predefined as a higher frequency in the case that the storage device to be detected is critical, the associated SLA requirement is at a higher level, etc. Further, a proper model can be selected from the stored anomaly prediction models according to the scheduling policy based on actual needs.

[0057] In some embodiments, the scheduling policy can be predefined and stored in the knowledge base 301. For example, the prediction module 210 can receive user input from a user specifying a scheduling policy based on actual experience. Accordingly, at block 320, the scheduler module 250 can access the predefined scheduling policy from the knowledge base 301 to schedule the corresponding anomaly prediction.

[0058] In addition, the scheduler module 250 can also schedule the update to the stored anomaly prediction model based on the scheduling policy, which will be described below.

[0059] Through the prediction processing, the storage devices with different failure likelihoods can be determined and thus can be classified into several sampling ranges. Each sampling range can include a range of storage devices among the plurality of storage devices, for example, storage devices at the same location (such as a server room, a rack, a row, etc.), storage devices with similar performance, or storage devices with similar device monitoring data. For example, in the case where the temperature of a first server room is abnormally high while the temperature of a second server room is normal, it can be determined that a sampling range including storage devices in the first server room has a higher failure likelihood than another sampling range including storage devices in the second server room.

[0060] Then, a corresponding sampling rate can be assigned to the respective sampling range. The sampling rate can indicate a percentage of the storage devices in the sampling range to be sampled. In some embodiments, a sampling range with high potential failure devices can be assigned a higher sampling rate than a sampling range with low potential failure devices. For the above example, the sampling range including storage devices in the first server room can be assigned a higher sampling rate than the sampling range including storage devices in the second room.

[0061] In some embodiments, the stage-based prediction can include two or more stages of anomaly prediction performed sequentially. For example, a next stage prediction (e.g., device monitoring data anomaly prediction) can be implemented based on the results of a previous stage prediction (e.g., environment anomaly prediction). Accordingly, high-risk devices can be filtered for more detailed storage device failure prediction / detection.

[0062] Figure 4 A flowchart depicting an exemplary process 400 for stage-based prediction according to some embodiments of the present disclosure is depicted.

[0063] At the first stage, at block 410, environmental anomaly prediction can be performed for the plurality of storage devices via an environmental anomaly prediction model, e.g., based on environmental data and metadata as well as maintenance data. It can be noted that different environmental anomaly prediction models can be used for different kinds of storage devices at this stage. Accordingly, at block 415, it can be determined whether the corresponding storage device is deployed in an abnormal environment or a normal environment. If the corresponding storage device is deployed in a normal environment, then at block 420, a first sampling range comprising the storage devices in the normal environment can be determined. The first sampling range can be assigned a first sampling rate, e.g., a normal sampling rate, such as 5%. In addition, if the corresponding storage device is deployed in an abnormal environment, then in the next stage of prediction (i.e., the second stage), the storage devices in the abnormal environment can be further processed.

[0064] At the second stage, at block 425, performance anomaly prediction can be performed for the storage devices in the abnormal environment via a performance anomaly prediction model, e.g., based on performance data and metadata as well as maintenance data. It can be noted that different performance anomaly prediction models can be used for different kinds of storage devices at this stage. Accordingly, at block 430, it can be determined whether the corresponding storage device is performing normally or abnormally. If the corresponding storage device is performing normally, then at block 435, a second sampling range comprising the storage devices in the abnormal environment and with normal performance can be determined. The second sampling range can be assigned a second sampling rate, which can be higher than the first sampling rate, e.g., 20%. In addition, if the corresponding storage device is performing abnormally, then in the next stage of prediction (i.e., the third stage), the storage devices in the abnormal environment and with abnormal performance can be further processed.

[0065] At the third stage, at block 440, device monitoring data anomaly prediction can be performed for the storage devices in the abnormal environment and with abnormal performance via a device monitoring data anomaly prediction model, e.g., based on device monitoring data and metadata as well as maintenance data. It can be noted that different device monitoring data anomaly prediction models can be used for different kinds of storage devices at this stage. Accordingly, at block 445, it can be determined whether the device monitoring data of the corresponding storage device is abnormal or normal. If the device monitoring data of the corresponding storage device is normal, then at block 450, a third sampling range comprising the storage devices in the abnormal environment with abnormal performance and normal device monitoring data can be determined. The third sampling range can be assigned a third sampling rate, which can be higher than the second sampling rate, e.g., 50%. In addition, at block 455, a fourth sampling range comprising the storage devices in the abnormal environment with abnormal performance and abnormal device monitoring data can be determined. The fourth sampling range can be assigned a fourth sampling rate, which can still be higher than the third sampling rate, e.g., 100%.

[0066] Therefore, the four sampling ranges and the corresponding sampling rates determined by the prediction module 210 can be output to the obtaining module 220. Based on the above example, in the process 400, the fourth sampling range can represent high-risk storage devices, and more attention should be given when performing device failure detection, while the first sampling range can represent low-risk storage devices, and less attention can be given to save resources, reduce costs and improve efficiency.

[0067] As can be appreciated, the process 400 is described for illustrative purposes only, and other appropriate details (including addition, removal, modification of one or more blocks) can also be implemented in some other embodiments of the present disclosure. For example, the process 400 can include two or more stages. According to actual needs, the sampling range can be further divided based on the monitoring data.

[0068] Returning to Figure 3 After the stage-based prediction process, at block 330, the obtaining module 220 can obtain a sampling data set by selecting a set of storage devices from the corresponding sampling range with the corresponding sampling rate.

[0069] In some embodiments, for each sampling range, the storage devices can be randomly selected based on the corresponding sampling rate. For example, if a sampling range is assigned a sampling rate of 20%, 20% of the storage devices in the sampling range can be randomly selected. In this way, the set of storage devices can include storage devices selected from each sampling range. It can be appreciated that the set can include more high-risk storage devices and fewer low-risk storage devices.

[0070] With respect to the example process 400 as discussed above, 5% of the storage devices in the first sampling range, 20% of the storage devices in the second sampling range, 50% of the storage devices in the third sampling range, and 100% of the storage devices in the fourth sampling range can be selected to form a set of storage devices to be further detected.

[0071] Therefore, the sampling data set can include real-time monitoring data associated with the set of storage devices. Therefore, the detection range can be narrowed from a plurality of storage devices to a set of storage devices within the plurality of storage devices, and the processing data can be reduced from large-scale monitoring data to sufficient features, i.e., monitoring data associated with the set of storage devices. Since the monitoring data required for prediction / detection is significantly reduced, the detection cost can be reduced, and the detection efficiency can be improved. The process for generating the sampling data set can be referred to as an active sampling process.

[0072] Then, at block 340, the detection module 230 can detect the device failure of the set of storage devices based on the sampled dataset. In some embodiments, the detection module 230 can apply a device failure prediction model to the sampled dataset. For example, the device failure prediction model can be at least one of a classification model, a regression model, a clustering model, a heuristic model.

[0073] Similar to the anomaly prediction model discussed above, the device failure prediction model can be pre-trained and stored in the knowledge base 301. Alternatively, at block 350, the model generation / update module 240 can generate the device failure prediction model based on the external benchmark database and the historical monitoring database of the plurality of storage devices. The respective model is trained with the benchmark data 308 and the historical monitoring data 307 associated with the plurality of storage devices. The repeated description can be omitted here. Then, the generated device failure prediction model can be stored in the knowledge base 301. Accordingly, the detection module 230 can access the pre-trained or generated failure prediction model from the knowledge base 301 to determine the failure detection.

[0074] Further, the device failure detection can also be performed based on a scheduling policy. For example, the scheduling policy can indicate, for a particular storage device, the appropriate device failure prediction model to be employed, the update timing of the respective model, etc. The scheduling policy can be predefined and stored in the knowledge base 301. Accordingly, the scheduler module 250 can access the predefined scheduling policy from the knowledge base 301 to determine the appropriate device failure prediction model for the storage device based on the scheduling policy. Further, the scheduler module 250 can also schedule the update of the stored device failure prediction model based on the scheduling policy, which will be described below.

[0075] Accordingly, the detection result 345 can be output for further operations, such as repairing the failed device, replacing the failed device with a new device, etc.

[0076] In some embodiments, the detected device failure can be stored in the failure base 302. Further, the stored device failure can be labeled with the corresponding failure type. For example, the detected device failure can be classified into different failure types, e.g., missing (it cannot establish a working I / O path to the disk), failed (the disk responds but does not work), failed (the disk reports a SMART trip or an uncorrected read error rate is too high), read-only (the disk can read but cannot write to certain sectors), slow (the disk performance is low compared to its peers, and the system will only read from the disk if necessary to avoid data loss), etc.

[0077] Accordingly, the failure base 302 can be established and maintained to save the detected failure with the corresponding failure type for further processing, e.g., updating the corresponding model and policy.

[0078] In some embodiments, at block 350, the model generation / update module 240 can update the respective model stored in the knowledge base 301 based on the fault library 302, the historical monitoring database, the external benchmark database. The updated model can then be used for the next process of device fault detection. In some embodiments, the model update can be scheduled by the scheduler module 250 based on the scheduling policy.

[0079] Figure 5 An exemplary block diagram of the scheduler module 250 according to an embodiment of the present disclosure is depicted.

[0080] As Figure 5 shown, in some embodiments, the scheduler module 250 can include an evaluation sub-module 510, a policy determination sub-module 520, a scheduler sub-module 530, and / or the like.

[0081] In some embodiments, the evaluation sub-module 510 can evaluate the scheduling requirement based on the fault library 302, the historical monitoring database 502, the external benchmark database 501. For example, the evaluation sub-module 510 can determine the normal failure rate of a certain kind of storage device according to the external benchmark database 501 and the historical monitoring database 502. The evaluation sub-module 510 can also determine the detected failure rate of the certain kind of storage device from the fault library 302. Then, the evaluation sub-module 510 can compare the normal failure rate with the detected failure rate to obtain the difference therebetween. If the difference is higher than a preset threshold, the evaluation sub-module 510 can determine that there is a scheduling requirement of adjusting the prediction frequency (or sampling frequency), updating the employed model (anomaly prediction model, device failure prediction model), and / or the like.

[0082] For example, if the normal failure rate is 4% and the detected failure rate is 9%, the difference is 5%, which is higher than a preset threshold, e.g., 3%. Therefore, the scheduling requirement can be determined as an increased prediction frequency, e.g., from once a day to twice a day. In addition, the evaluation sub-module 510 can determine that the employed model should be updated.

[0083] In some embodiments, the policy determination sub-module 520 can select a scheduling policy from the scheduling policies stored in the knowledge base 301 based on the scheduling requirement. For example, the scheduling policy can indicate the prediction timing (sampling timing), the prediction frequency (sampling frequency), the anomaly prediction model / device failure prediction model to be employed, the update timing of the respective model, and / or the like.

[0084] Further, the scheduler sub-module 530 can schedule the prediction process based on the determined strategy, e.g., adjust the prediction frequency (or sampling frequency). In addition, the scheduler sub-module 530 can schedule the update of the respective model, e.g., trigger the model generation / update module 240 to update the respective model based on the detected device failures stored in the failure library 302, the historical monitoring database 502 and the external benchmark database 501.

[0085] Thus, the respective models can be updated / re-trained with new training data, e.g., detected device failures, new inputs from the historical monitoring database and the external benchmark database, etc. Accordingly, the prediction / detection results can be dynamically adjusted. The detection efficiency can be improved, thereby facilitating the reduction of cost and avoiding the waste of resources.

[0086] Figure 6 An exemplary flowchart of a method 600 for device failure detection according to embodiments of the present disclosure is depicted. The process can be implemented by a computing device, such as the computer 101 shown. Figure 1

[0087] At block 610, the computing device can perform a stage-based prediction on the plurality of storage devices to determine a plurality of sampling ranges and corresponding sampling rates. The respective sampling range can include at least one storage device of the plurality of storage devices.

[0088] Thus, an effective balance of accuracy, performance and cost of the failure detection of the storage devices can be provided.

[0089] In some embodiments, performing the stage-based prediction on the plurality of storage devices can include performing the stage-based prediction based on real-time monitoring data associated with the plurality of storage devices and a plurality of models. The respective models are trained with benchmark data and historical monitoring data associated with the plurality of storage devices. Thus, the historical monitoring data and the benchmark data, which includes open data for specific manufacturers, models and batches via supervised or unsupervised algorithms, can be employed to facilitate the detection.

[0090] In some embodiments, the benchmark models can include at least two of the following models: an environmental anomaly prediction model, a performance anomaly prediction model and a SMART anomaly prediction model.

[0091] In some embodiments, the stage-based prediction includes at least two prediction stages. The prediction of the next stage is performed based on the results of the prediction of the previous stage. Thus, sufficient features can be utilized to reduce the prediction range and cost.

[0092] ​In some embodiments, performing stage-based prediction for the plurality of storage devices further comprises the following stages. In a first stage, environmental anomalies can be predicted for the plurality of storage devices to determine a first sampling range with a first sampling rate. The first sampling range comprises storage devices in a normal environment. In a second stage, performance anomalies of storage devices in an abnormal environment can be predicted to determine a second sampling range with a second sampling rate. The second sampling range comprises storage devices that are in an abnormal environment and perform normally. In a third stage, device monitoring data anomalies are predicted for storage devices that are in an abnormal environment and perform abnormally, determining a third sampling range with a third sampling rate and a fourth sampling range with a fourth sampling rate. The third range comprises storage devices that are in an abnormal environment, perform abnormally, and have normal device monitoring data. The fourth range comprises storage devices that are in an abnormal environment, perform abnormally, and have abnormal device monitoring data. Further, the first sampling rate is lower than the second sampling rate, the second sampling rate is lower than the third sampling rate, and the third sampling rate is lower than the fourth sampling rate. Thus, different sampling ranges can be assigned different sampling rates to filter high-risk devices for more detailed device failure prediction.

[0093] At block 620, the computing device can obtain a sampling dataset by selecting a set of storage devices from the respective sampling ranges with the corresponding sampling rates.

[0094] At block 630, the computing device can detect device failures of the set of storage devices based on the sampling dataset.

[0095] In some embodiments, the steps of performing, obtaining, and detecting can be implemented multiple times, and the step of performing is scheduled based on a scheduling policy. Thus, the prediction can be performed on-demand based on a dynamic scheduling policy and recent detection results.

[0096] In some embodiments, a failure library can be generated based on the detected device failures. Thus, the failure library can be established and maintained to save anomalies detected for the active sampling.

[0097] In some embodiments, a scheduling requirement can be evaluated based on the failure library, benchmark data, and historical monitoring data. A scheduling policy can be selected based on the scheduling requirement. Thus, the prediction can be dynamically adjusted based on actual needs.

[0098] In some embodiments, the respective models are scheduled for updates based on the scheduling requirement. Thus, the individual models can be dynamically updated based on actual needs.

[0099] In some embodiments, the step of detecting device failures of the set of storage devices based on the sampled dataset can include detecting device failures based on real-time monitoring data associated with the set of storage devices and a plurality of device failure prediction models. The respective device failure prediction models are trained with benchmark data and historical monitoring data associated with the plurality of storage devices. Thus, historical monitoring data and benchmark data, which includes open data for specific manufacturers, models, and batches via supervised or unsupervised algorithms, can be employed to facilitate detection.

[0100] It can be noted that the order of the blocks described in the above embodiments is for illustrative purposes only. Any other suitable sequence (including addition, deletion, and / or modification of at least one block) can also be implemented to determine the respective embodiments.

[0101] Additionally, in some embodiments of the present disclosure, a system for device failure detection can be provided. The system can include one or more processors, a memory coupled to at least one of the one or more processors, and a set of computer program instructions stored in the memory. The set of computer program instructions can be executable by at least one of the one or more processors to perform the above-described method.

[0102] In some other embodiments of the present disclosure, a computer program product for device failure detection can be provided. The computer program product can include a computer readable storage medium having program instructions. The program instructions executable by one or more processors to cause the one or more processors to perform the above-described method.

[0103] The present application can be a system, a method, and / or a computer program product at any possible technical detail level of integration. The computer program product can include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present application.

[0104] The description of the various embodiments of the present application has been presented for purposes of illustration, but is not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope of the described embodiments. The terminology used is for the purpose of describing the embodiments and is not intended to limit the scope of the present application.

Claims

1. A computer-implemented method, comprising: One or more processors perform phase-based predictions for multiple storage devices to determine multiple sampling ranges and corresponding sampling rates, wherein the corresponding sampling ranges include at least one of the multiple storage devices; The one or more processors obtain the sampled dataset by selecting a set of storage devices from the corresponding sampling range using the corresponding sampling rate; and The one or more processors detect device failures in the set of storage devices based on the sampled dataset.

2. The computer-implemented method according to claim 1, wherein, Performing the phase-based prediction for the plurality of storage devices further includes: The phase-based prediction is performed by one or more processors based on real-time monitoring data associated with the plurality of storage devices and multiple models, wherein the corresponding models are trained using benchmark data and historical monitoring data associated with the plurality of storage devices.

3. The computer-implemented method according to any one of the preceding claims, wherein, The phase-based prediction includes predictions for at least two phases; In this process, the predictions for the next stage are based on the results of the predictions in the previous stage.

4. The computer-implemented method according to any one of the preceding claims, wherein, Performing the phase-based prediction for the plurality of storage devices further includes: In the first phase, the one or more processors predict environmental anomalies for the plurality of storage devices to determine a first sampling range with a first sampling rate, wherein the first sampling range includes storage devices in a normal environment; In the second phase, the one or more processors predict performance anomalies in storage devices under abnormal environments to determine a second sampling range with a second sampling rate, wherein the second sampling range includes storage devices that perform normally under abnormal environments; and In the third stage, the one or more processors predict device monitoring data anomalies for storage devices that are performing abnormally in an abnormal environment, in order to determine a third sampling range with a third sampling rate and a fourth sampling range with a fourth sampling rate, wherein the third range includes storage devices that are performing abnormally in an abnormal environment and have normal device monitoring data, and wherein the fourth range includes storage devices that are performing abnormally in an abnormal environment and have abnormal device monitoring data. Wherein, the first sampling rate is lower than the second sampling rate, the second sampling rate is lower than the third sampling rate, and the third sampling rate is lower than the fourth sampling rate.

5. The computer-implemented method according to any one of the preceding claims further includes: The execution, acquisition, and detection steps are performed multiple times by the one or more processors, wherein the execution steps are arranged based on a scheduling policy.

6. The computer-implemented method according to any one of the preceding claims further includes: The one or more processors generate a fault library based on detected device faults.

7. The computer-implemented method according to claim 6, further comprising: The scheduling requirements are assessed by the one or more processors based on the fault database, the baseline data, and the historical monitoring data; The scheduling strategy is selected based on the scheduling requirements.

8. The computer-implemented method according to claim 7, wherein, The corresponding model schedules updates based on the aforementioned scheduling requirements.

9. The computer-implemented method according to any one of the preceding claims, wherein, Detecting device faults in the set of storage devices based on the sampled dataset includes: The one or more processors detect device failures based on real-time monitoring data associated with the set of storage devices and multiple device failure prediction models, wherein the corresponding device failure prediction models are trained using benchmark data and historical monitoring data associated with the multiple storage devices.

10. A computer system, comprising: One or more computer processors; A memory coupled to at least one of the processors; as well as A set of computer program instructions, stored in the memory and executed by at least one of the one or more computer processors, to perform the following actions: Stage-based prediction is performed for multiple storage devices to determine multiple sampling ranges and corresponding sampling rates, wherein the corresponding sampling ranges include at least one of the multiple storage devices; The sampled dataset is obtained by selecting a set of storage devices from the corresponding sampling range using the corresponding sampling rate; and Device failures of the set of storage devices are detected based on the sampled dataset.

11. The computer system according to claim 10, wherein, Performing the phase-based prediction for the plurality of storage devices further includes: The phase-based prediction is performed based on real-time monitoring data associated with the plurality of storage devices and multiple models, wherein the corresponding models are trained using benchmark data and historical monitoring data associated with the plurality of storage devices.

12. The system according to any one of claims 10 to 11, wherein, The phase-based prediction includes predictions in at least two phases, wherein the prediction in the next phase is performed based on the results of the prediction in the previous phase.

13. The computer system according to any one of claims 10 to 12, wherein, Performing the phase-based prediction for the plurality of storage devices further includes: In the first phase, environmental anomalies are predicted for the plurality of storage devices to determine a first sampling range with a first sampling rate, wherein the first sampling range includes storage devices in a normal environment; In the second phase, performance anomalies are predicted for storage devices in abnormal environments to determine a second sampling range with a second sampling rate, wherein the second sampling range includes storage devices that perform normally in abnormal environments; and In the third stage, for storage devices that perform abnormally in an abnormal environment, predict device monitoring data anomalies to determine a third sampling range with a third sampling rate and a fourth sampling range with a fourth sampling rate, wherein the third range includes storage devices that perform abnormally in an abnormal environment and have normal device monitoring data, and wherein the fourth range includes storage devices that perform abnormally in an abnormal environment and have abnormal device monitoring data. Wherein, the first sampling rate is lower than the second sampling rate, the second sampling rate is lower than the third sampling rate, and the third sampling rate is lower than the fourth sampling rate.

14. The computer system according to any one of claims 10 to 13, wherein, The action also includes: The execution, acquisition, and detection steps are performed multiple times, wherein the execution steps are arranged based on a scheduling strategy.

15. The computer system according to any one of claims 10 to 14, wherein, The action also includes: A fault database is generated based on the detected equipment faults.

16. The computer system of claim 15, wherein the action further comprises: The scheduling requirements are assessed based on the fault database, the baseline data, and the historical monitoring data. The scheduling strategy is selected based on the scheduling requirements; and The corresponding model arranges updates based on the aforementioned scheduling requirements.

17. The computer system according to any one of claims 10 to 16, wherein, Detecting device faults in the set of storage devices based on the sampled dataset includes: The device faults are detected based on real-time monitoring data associated with the set of storage devices and multiple device fault prediction models, wherein the corresponding device fault prediction models are trained using benchmark data and historical monitoring data associated with the multiple storage devices.

18. A computer program product comprising a computer-readable storage medium having program instructions embodied therein, the program instructions being executable by one or more processors to cause the one or more processors to perform the following actions: Stage-based prediction is performed for multiple storage devices to determine multiple sampling ranges and corresponding sampling rates, where, The corresponding sampling range includes at least one of the plurality of storage devices; The sampled dataset is obtained by selecting a set of storage devices from the corresponding sampling range using the corresponding sampling rate. as well as Device failures of the set of storage devices are detected based on the sampled dataset.

19. The computer program product according to claim 18, wherein, Performing the phase-based prediction for the plurality of storage devices includes: The phase-based prediction is performed based on real-time monitoring data associated with the plurality of storage devices and multiple models. The corresponding models are trained using benchmark data and historical monitoring data associated with the multiple storage devices.

20. The computer program product according to any one of claims 18 to 19, wherein, The phase-based prediction includes predictions for at least two phases; In this process, the predictions for the next stage are based on the results of the predictions in the previous stage.

Images