Detection and prediction of data anomalies in electronic storage devices

Abstract

Computer implementations, systems, and computer program products for device failure detection are disclosed. In the method, phase-based predictions may be performed on multiple storage devices to determine multiple sampling ranges and corresponding sampling ratios. Each sampling range may include at least one storage device from the multiple storage devices. A sampling dataset may be obtained by selecting a group of storage devices from each sampling range having a corresponding sampling ratio. Device failures may be detected for the group of storage devices based on the sampling dataset.

Description

Technical Field

[0001] The present invention relates to data processing, and more particularly to the detection of anomalies in electronic storage devices.

Background Art

[0002] For enterprises / organizations, data is property and is growing at an exponential rate. Data is usually stored in storage devices such as hard disk drives (HDDs), solid state drives (SSDs), or tapes on-premises or in the cloud. However, failures of storage devices can cause many adverse effects such as data loss, service unavailability, additional operating costs, economic losses, etc.

Summary of the Invention

[0003] This summary is provided to introduce a selected concept in a simplified form that will be further described below in the mode for carrying out the invention. This summary of the invention is not intended to identify the key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0004] According to one embodiment of the present invention, a computer-implemented method for device failure detection is provided. In this method, phase-based prediction can be performed on multiple storage devices to determine multiple sampling ranges and corresponding sampling ratios. Each sampling range may include at least one storage device among the multiple storage devices. A sampling dataset can be obtained by selecting a group of storage devices from each sampling range having a corresponding sampling ratio. Device failures can be detected for the group of storage devices based on the sampling dataset.

[0005] Therefore, an efficient balance of accuracy, performance, and cost can be provided for fault detection in storage devices.

[0006] In some embodiments, the step of performing phase-based predictions for multiple storage devices may include: performing phase-based predictions based on real-time monitoring data associated with multiple storage devices and multiple models. Each model is trained on benchmark data and historical monitoring data associated with multiple storage devices. Therefore, historical monitoring data and benchmark data, including open data about specific manufacturers, models, and batches, may be used to facilitate detection via supervised or unsupervised algorithms.

[0007] In some embodiments, phase-based forecasting includes at least two forecasting phases. Later forecasting phases are performed based on the results of earlier forecasting phases. Therefore, the forecasting scope and cost can be reduced by using appropriate features.

[0008] In some embodiments, the step of performing phase-based prediction for multiple storage devices further includes the following phases: In a first phase, environmental anomalies on multiple storage devices may be predicted in order to determine a first sampling range having a first sampling ratio. The first sampling range includes storage devices in a normal environment. In a second phase, operational anomalies on storage devices in an abnormal environment may be predicted in order to determine a second sampling range having a second sampling ratio. The second sampling range includes storage devices that are operating normally in an abnormal environment. In a third phase, device monitoring data anomalies on storage devices that are operating abnormally in an abnormal environment may be predicted in order to determine a third sampling range having a third sampling ratio and a fourth sampling range having a fourth sampling ratio. The third range includes storage devices that are operating abnormally in an abnormal environment and have normal device monitoring data. The fourth range includes storage devices that are operating abnormally in an abnormal environment and have abnormal device monitoring data. Furthermore, the first sampling ratio is lower than the second sampling ratio, which is lower than the third sampling ratio, and which is lower than the fourth sampling ratio. Therefore, different sampling ranges can be assigned to different sampling ratios in order to filter out high-risk devices for predicting device failures, which will be described in more detail later.

[0009] In some embodiments, the execution, acquisition, and detection steps may be implemented multiple times, with the execution step being scheduled based on a scheduling policy. Thus, predictions may be performed on demand based on a dynamic scheduling policy and recent detection results.

[0010] In some embodiments, a fault base can be generated based on detected device failures. Therefore, for active sampling, the fault base can be built and maintained to store detected anomalies.

[0011] In some embodiments, scheduling needs can be assessed based on failure-based, benchmark data, and historical monitoring data. A scheduling policy can be selected based on these scheduling needs. Therefore, forecasts can be dynamically adjusted based on actual needs.

[0012] In some embodiments, each model is scheduled to be updated based on scheduling needs. Therefore, each model can be dynamically updated based on actual needs.

[0013] In some embodiments, the step of detecting device failures for a group of storage devices based on a sampled dataset may include the step of detecting device failures based on real-time monitoring data associated with the group of storage devices and multiple device failure prediction models. Each device failure prediction model is trained on benchmark data and historical monitoring data associated with multiple storage devices. Therefore, historical monitoring data and benchmark data, including open data for specific manufacturers, models, and batches, may be used to facilitate detection via supervised or unsupervised algorithms.

[0014] According to another embodiment of the present invention, a system for detecting device failures is provided. The system may comprise one or more processors, a memory linked to at least one of the one or more processors, and a set of computer program instructions stored in the memory. The set of computer program instructions may be executed by at least one of the one or more processors to perform the above method.

[0015] According to another embodiment of the present invention, a computer program product for device failure detection is provided. The computer program product may comprise a computer-readable storage medium having program instructions embodied therein. Such program instructions are executable by one or more processors to cause one or more processors to perform the above method.

[0016] In addition to the exemplary embodiments and models described above, further embodiments and models will become apparent by referring to the drawings and by studying the following description. [Brief explanation of the drawing]

[0017] Through a more detailed description of some embodiments of the present disclosure in the accompanying drawings, the above and other objects, features, and advantages of the present disclosure will become more apparent, and the same reference numerals herein generally refer to the same components in the embodiments of the present disclosure.

[0018] [Figure 1] This is an exemplary computing environment applicable to implementing embodiments of the present disclosure.

[0019] [Figure 2] This is an exemplary device failure detection system according to an embodiment of the present disclosure.

[0020] [Figure 3] This is an exemplary process for device failure detection according to embodiments of the present disclosure.

[0021] [Figure 4] This disclosure illustrates an exemplary process for phase-based prediction according to embodiments of this disclosure.

[0022] [Figure 5] An exemplary block diagram of a scheduler module according to an embodiment of the present disclosure is shown.

[0023] [Figure 6] An exemplary flowchart of a computer-implemented method for device fault detection according to an embodiment of the present disclosure is shown.

Best Mode for Carrying Out the Invention

[0024] Various aspects of the present disclosure are described by illustrative text, flowcharts, block diagrams of computer systems, and / or block diagrams of machine logic included in embodiments of a computer program product (CPP). For any flowchart, depending on the technology involved, operations may be performed in an order different from that shown in a given flowchart. For example, again depending on the technology involved, two operations shown in consecutive flowchart blocks may be performed in reverse order, as a single integrated step, simultaneously, or at least partially overlapping in time.

[0025] Embodiments of a computer program product ("CPP Embodiment" or "CPP") are terms used in this disclosure to describe any set of one or more storage media (also called "mediums") that collectively comprise a set of one or more storage devices that collectively comprise machine-readable code corresponding to instructions and / or data for performing computer operations specified in a given CPP claim. A "storage device" is any tangible device capable of holding and storing instructions for use by a computer processor. Computer-readable storage media may, but are not limited to, electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, mechanical storage media, or any preferred combination thereof. Some known types of storage devices, including these media, include diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), compact disc read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanically encoded devices (such as pits / lands formed on the main surface of a punch card or disk), or any suitable combination of the foregoing. When the term "computer-readable storage medium" is used in this disclosure, it shall not be interpreted as storage in the form of a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides, optical pulses passing through optical fiber cables, electrical signals communicated through wires, and / or other transmission media.As will be understood by those skilled in the art, data is typically moved at various irregular times during the normal operation of a storage device, such as during access, defragmentation, or garbage collection, but since the data is not transient while it is stored, the above means that the storage device is not considered to be transient.

[0026] Computing environment 100 includes an example of an environment for the execution of at least some of the computer code involved in the execution of the method of the present invention, such as device failure detection system 200. In addition to block 200, computing environment 100 includes, for example, computer 101, wide area network (WAN) 102, end user device (EUD) 103, remote server 104, public cloud 105, and private cloud 106. In this embodiment, computer 101 includes a processor set 110 (including processing circuit 120 and cache 121), communication fabric 111, volatile memory 112, persistent storage 113 (including operating system 122 and block 200 as specified above), a set of peripheral devices 114 (including a set of user interface (UI) devices 123, storage 124, and a set of Internet of Things (IoT) sensors 125), and network module 115. Remote server 104 includes remote database 130. Public cloud 105 includes gateway 140, cloud orchestration module 141, host physical machine set 142, virtual machine set 143, and container set 144.

[0027] Computer 101 may take the form of a desktop computer, laptop computer, tablet computer, smartphone, smartwatch or other wearable computer, mainframe computer, quantum computer, or any other form of computer or mobile device currently known or to be developed in the future that is capable of running programs, accessing networks, or querying databases such as remote database 130. As is well understood in the field of computer technology, and depending on the technology, the execution of a computer implementation method can be distributed among multiple computers and / or multiple locations. On the other hand, in this presentation of computing environment 100, in order to keep the presentation as concise as possible, the detailed discussion focuses on a single computer, specifically computer 101. Computer 101 is not shown in the cloud in Figure 1, but it may be located in the cloud. On the other hand, computer 101 does not need to be in the cloud unless explicitly stated otherwise.

[0028] The processor set 110 includes one or more computer processors of any type currently known or to be developed in the future. The processing circuitry 120 may be distributed across multiple packages, for example, multiple coordinated integrated circuit chips. The processing circuitry 120 may implement multiple processor threads and / or multiple processor cores. The cache 121 is memory located within the processor chip package and is typically used for data or code that should be available for high-speed access by threads or cores running on the processor set 110. The cache memory is typically organized into multiple levels depending on its relative proximity to the processing circuitry. Alternatively, some or all of the cache for the processor set may be located "off-chip". In some computing environments, the processor set 110 may operate using qubits and be designed to perform quantum computing.

[0029] Computer-readable program instructions typically cause the processor set 110 of computer 101 to execute a series of operational steps, thereby loading them onto computer 101 to implement a computer implementation method. As a result, and therefore the instructions thus executed, the instructions instantiate the methods specified in the flowcharts and / or descriptions of the computer implementation methods contained herein (collectively referred to as the “Methods of the Invention”). These computer-readable program instructions are stored in various types of computer-readable storage media, such as a cache 121 and other storage media discussed below. The program instructions and associated data are accessed by the processor set 110 to control and direct the execution of the Methods of the Invention. In the computing environment 100, at least some of the instructions for executing the Methods of the Invention may be stored in blocks 200 of persistent storage 113.

[0030] The communication fabric 111 is a signal-conducting path that enables various components of the computer 101 to communicate with one another. Typically, this fabric is made up of switches and conductive paths, such as buses, bridges, physical input / output ports, and similar switches and conductive paths. Other types of signal-conducting paths, such as optical fiber communication paths and / or wireless communication paths, may be used.

[0031] The volatile memory 112 is any type of volatile memory currently known or to be developed in the future. Examples include dynamic random-access memory (RAM) or static RAM. Typically, volatile memory is characterized by random access, but this is not required unless explicitly stated. In computer 101, the volatile memory 112 is located in a single package and resides inside computer 101, but alternatively or in addition, the volatile memory may be distributed across multiple packages and / or located externally to computer 101.

[0032] The persistent storage 113 is any form of non-volatile storage for a computer, currently known or to be developed in the future. The non-volatility of this storage means that the stored data is maintained whether or not power is directly supplied to the computer 101 and / or the persistent storage 113. The persistent storage 113 may be read-only memory (ROM), but typically at least a portion of the persistent storage allows for writing, deleting, and rewriting of data. Some well-known forms of persistent storage include magnetic disks and solid-state storage devices. The operating system 122 may take multiple forms, such as various known proprietary operating systems or open-source Portable Operating System Interface type operating systems, which utilize a kernel. The code contained in block 200 typically includes at least a portion of computer code involved in performing the method of the present invention.

[0033] The peripheral device set 114 includes a set of peripheral devices for the computer 101. Data communication connections between the peripheral devices and other components of the computer 101 may be implemented in various ways, such as Bluetooth, near-field communication (NFC), cable (such as a Universal Serial Bus (USB) type cable), insertable connections (such as a Secure Digital (SD) card), connections via a local area communication network, and connections via a wide area network such as the Internet. In various embodiments, the UI device set 123 may include multiple components, such as a display screen, speaker, microphone, wearable devices (such as goggles and smartwatches), keyboard, mouse, printer, touchpad, game controller, and haptic device. Storage 124 is external storage such as an external hard drive, or insertable storage such as an SD card. Storage 124 may be persistent and / or volatile. In some embodiments, storage 124 may take the form of a quantum computing storage device for storing data in the form of qubits. In embodiments where computer 101 is required to have a large amount of storage (for example, computer 101 locally stores and manages a large database), this storage may be provided by peripheral storage devices designed to store very large amounts of data, such as a storage area network (SAN) shared by multiple geographically distributed computers. The IoT sensor set 125 consists of sensors that can be used in Internet of Things applications. For example, one sensor may be a thermometer and another may be a motion detector.

[0034] The network module 115 is a collection of computer software, hardware, and firmware that enables computer 101 to communicate with other computers via the WAN 102. The network module 115 may include hardware such as a modem or Wi-Fi signal transceiver, software for packetizing and / or depacketizing data for transmission over a communication network, and / or web browser software for communicating data over the internet. In some embodiments, the network control and network forwarding functions of the network module 115 are performed on the same physical hardware device. In other embodiments (e.g., embodiments utilizing software-defined networking (SDN)), the control and forwarding functions of the network module 115 are performed on physically separate devices, such that the control function manages several different network hardware devices. Computer-readable program instructions for performing the methods of the present invention can typically be downloaded from an external computer or external storage device to computer 101 through a network adapter card or network interface included in the network module 115.

[0035] WAN102 is any wide area network (e.g., the Internet) capable of transmitting computer data over non-local distances by any currently known or future-developed technology for transmitting computer data. In some embodiments, the WAN may be replaced and / or complemented by a local area network (LAN), such as a Wi-Fi network, designed to exchange data between devices located in a local area. The WAN and / or LAN typically include computer hardware such as copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and edge servers.

[0036] The end-user device (EUD) 103 is any computer system used and controlled by an end-user (e.g., a customer of the company operating computer 101), and can take any of the forms discussed above in relation to computer 101. Typically, EUD 103 receives useful and valuable data from the operation of computer 101. For example, in a hypothetical case where computer 101 is designed to provide recommendations to the end-user, these recommendations would typically be communicated from the network module 115 of computer 101 to EUD 103 via WAN 102. In this way, EUD 103 can display or otherwise present the recommendations to the end-user. In some embodiments, EUD 103 may be a client device such as a thin client, heavy client, mainframe computer, or desktop computer.

[0037] The remote server 104 is any computer system that provides at least some data and / or functionality to computer 101. The remote server 104 may be controlled and used by the same entity that operates computer 101. The remote server 104 represents a machine that collects and stores useful and valuable data for use by other computers, such as computer 101. For example, in a hypothetical case where computer 101 is designed and programmed to provide recommendations based on historical data, this historical data may be provided to computer 101 from the remote database 130 of the remote server 104.

[0038] The public cloud 105 is any computer system available for use by multiple entities, providing on-demand availability of computer system resources and / or other computer functions, particularly data storage (cloud storage) and computing power, without direct active management by the user. Cloud computing typically leverages resource sharing to achieve coherence and economies of scale. Direct active management of the computing resources of the public cloud 105 is performed by the computer hardware and / or software of the cloud orchestration module 141. The computing resources provided by the public cloud 105 are typically implemented by virtual computing environments running on various computers that make up the computers of the host physical machine set 142, which is the universe of physical computers available in and / or to the public cloud 105. The virtual computing environment (VCE) typically takes the form of virtual machines from the virtual machine set 143 and / or containers from the container set 144. These VCEs may be stored as images and are understood to be transportable either as images or after instantiation of the VCEs, among and between various physical machine hosts. The cloud orchestration module 141 manages the transfer and storage of images, deploys new instances of VCE, and manages active instances of VCE deployments. The gateway 140 is a collection of computer software, hardware, and firmware that enables the public cloud 105 to communicate over the WAN 102.

[0039] Some further explanation of virtualized computing environments (VCEs) is provided here. A VCE can be stored as an "image." From this image, a new active instance of the VCE can be instantiated. Two well-known types of VCEs are virtual machines and containers. A container is a VCE that uses operating system-level virtualization. This refers to an operating system feature where the kernel allows for the existence of multiple isolated user-space instances called containers. These isolated user-space instances typically behave like actual computers in terms of the programs running within them. Computer programs running on a normal operating system can utilize all of that computer's resources, including connected devices, files and folders, network shares, CPU power, and quantifiable hardware capabilities. However, programs running inside a container can only use the contents of the container and the devices allocated to the container; this feature is known as containerization.

[0040] The private cloud 106 is similar to the public cloud 105, except that its computing resources are available only for use by a single enterprise. While the private cloud 106 is shown as communicating with the WAN 102, in other embodiments, the private cloud may be completely isolated from the internet and accessible only via a local / private network. A hybrid cloud is a combination of multiple clouds of different types (e.g., private, community, or public cloud types), often implemented by different vendors. Each of the multiple clouds remains a separate discrete entity, but the larger hybrid cloud architecture is bound together by standardized or proprietary technologies that enable orchestration, management, and / or data / application portability between the multiple configuration clouds. In this embodiment, both the public cloud 105 and the private cloud 106 are part of a larger hybrid cloud.

[0041] The computing environment 100 in Figure 1 is provided for illustrative purposes only, without implying any limitation to any embodiment of the present invention, and it is understood that, for example, at least a portion of the program code involved in performing the method of the present invention may be loaded into a cache 121, volatile memory 112, or stored in other storage of the computer 101 (e.g., storage 124), or at least a portion of the program code involved in performing the method of the present invention may be stored in other local and / or remote computing environments and loaded when needed. As another example, peripheral device 114 may also be implemented by independent peripheral devices connected to the computer 101 through an interface. As a further example, the WAN may be replaced and / or complemented by any other connection made to an external computer (e.g., via the Internet using an Internet service provider).

[0042] Generally, failure detection (or anomaly prediction) for high-capacity storage devices is costly and time-consuming due to the large amount of monitoring data and the large-scale deployment in manufacturing. Methods for predicting device failures can include threshold-based methods (setting thresholds based on selected metrics), statistics-based methods (building statistical models based on selected metrics), and learning-based methods (building machine learning or deep learning models based on given features to predict anomalies or lifespan). However, most existing methods focus more on the accuracy of the method. Little effort has been made to address the real-world challenges of balancing accuracy, performance, and cost for high-capacity storage devices.

[0043] Embodiments of this disclosure provide a device failure detection system for detecting and predicting anomalies / failures in large-scale storage devices. Based on the embodiments, an efficient balance of accuracy, performance, and cost can be achieved in storage device failure detection. The detection range and the resources allocated to detection modules can be reduced. In addition, more time and cost can be saved in detecting anomaly storage devices.

[0044] Referring now to Figure 2, a block diagram is provided showing an exemplary device failure detection system 200 according to several embodiments of the present disclosure.

[0045] It should be noted that the process of the device failure detection system 200 according to the embodiments of this disclosure can be implemented in the computing environment shown in Figure 1.

[0046] As shown in Figure 2, in some embodiments, the device failure detection system 200 may comprise a prediction module 210, an acquisition module 220, and a detection module 230. In further embodiments, the device failure detection system 200 may also comprise a model generation / update module 240 and / or a scheduler module 250, etc. All or some of the modules may be configured to communicate with one another (e.g., via a communication fabric 111 as shown in Figure 1, such as a bus, shared memory, switches, or a network). One or more of these modules may be implemented using the processing circuit 120 in Figure 1 (e.g., by configuring the processing circuit 120 to perform the functions described for that module). It should be noted that the addition, removal, and / or modification of one or more modules may be configured on a case-by-case basis.

[0047] Figure 3 shows an exemplary process 300 for device failure detection according to an embodiment of the present disclosure. Process 300 may be implemented using a device failure detection system 200, which will be described below in reference to Figure 2.

[0048] In block 310, the prediction module 210 may perform phase-based predictions on multiple storage devices to determine multiple sampling ranges and corresponding sampling ratios. Each sampling range may include at least one of the multiple storage devices.

[0049] In some embodiments, the storage device may be at least one of the following: HDD, SSD, memory card, floppy disk, optical disk drive (Compact Disk (CD), Digital Versatile Disc (DCD, Blu-ray DVD)), RAM, and / or ROM. Furthermore, each storage device may be associated with a set of monitoring data, such as environmental data, performance data, device monitoring data, metadata, and maintenance data.

[0050] For example, environmental data may include metrics about the operating environment of the storage device (e.g., server room, column, rack, etc.), such as temperature, humidity, and / or air quality. Performance data may reflect the performance of applications deployed on the storage device. For example, performance data may include Input / Output Operations Per Second (IOPS), Mean Time between Failures (MTBF), Mean Time to Repair (MTTR), and / or read / write speeds. Device monitoring data may include Self-Monitoring Analysis and Reporting Technology (SMART) data showing metrics of HDD and SSD attributes, such as read error rate, start / stop count, and drive calibration retry count. Other types of device monitoring data known in the art may also be included for other types of storage devices. Furthermore, metadata may include the vendor, type, size, and / or age of the storage device. Maintenance data may include maintenance logs related to the storage device. As can be understood, any other appropriate monitoring data associated with the storage device may also be acquired based on actual needs.

[0051] Therefore, the prediction module 210 can receive large amounts of monitoring data associated with multiple storage devices in real time. The received monitoring data may also be referred to as real-time monitoring data 305. Thus, phase-based prediction can be performed based on the real-time monitoring data 305. In general, real-time monitoring data 305 is a great help in predicting storage device failures, and SMART data in particular is. However, the volume of such real-time monitoring data 305 is too large to determine an efficient prediction. In embodiments, phase-based prediction can help filter the data to be most useful for further prediction / detection.

[0052] In some embodiments, the prediction module 210 may perform phase-based predictions based on real-time monitoring data using multiple anomaly prediction models. For example, the anomaly prediction models may include an environmental anomaly prediction model, an operational anomaly prediction model, and / or a device monitoring data anomaly prediction model. Furthermore, each anomaly prediction model may be at least one of a classification model, a regression model, a clustering model, or a heuristic model. As can be understood, any other suitable model known in the art may also be implemented based on actual needs.

[0053] In one instance, an environmental anomaly prediction model may be configured to predict whether a storage device is in an abnormal environment, such as abnormal temperature, abnormal humidity, and / or corrosive gas. For example, thresholds for temperature, humidity, and / or gas levels may be predefined. In another instance, an operational anomaly prediction model may be configured to predict whether a storage device is operating abnormally or whether the operation of applications distributed on the storage device is abnormal. For example, anomalies in the operation of a storage device may be predicted if the read / write speed is relatively slow, the IOPS is dramatically reduced, or the MTTR is relatively high. For example, the respective thresholds for read / write speed, IOPS, or MTTR may be predefined. In a further instance, a device monitoring data anomaly prediction model (e.g., a SMART data anomaly prediction model) may be configured to predict abnormal attributes of a storage device.

[0054] In some embodiments, the anomaly prediction model discussed above may be pre-trained and stored in the knowledge base 301. Alternatively, in block 350, the model generation / update module 240 may generate an anomaly prediction model from an external benchmark database and a historical monitoring database of multiple storage devices. Specifically, the external benchmark database may store open-source data about storage devices for specific manufacturers, models, and batches using supervised or unsupervised algorithms. Furthermore, the historical monitoring database may store historical monitoring data 307 associated with the storage devices, such as environmental data, performance data, device monitoring data, metadata, and / or maintenance data. The historical monitoring data 307 may be similar to real-time monitoring data but is collected from sampling of historical data. A repeated explanation of historical monitoring data may be omitted herein. The generated anomaly prediction model may then be stored in the knowledge base 301.

[0055] Therefore, the prediction module 210 can access pre-trained or generated anomaly prediction models from the knowledge base 301 to determine the corresponding anomaly prediction.

[0056] Furthermore, phase-based predictions can be further performed based on scheduling policies. For example, scheduling policies may specify, for a particular storage device, the predicted timing (also called sampling timing), predicted frequency (also called sampling frequency), the appropriate model to employ, and / or the update timing for each model. For example, if the storage device to be detected is critical and / or the associated SLA requirements are at a higher level, the predicted frequency may be predefined as a higher frequency. Also, an appropriate model may be selected from stored anomaly detection models according to the scheduling policy based on actual needs.

[0057] In some embodiments, scheduling policies may be predefined and stored in the knowledge base 301. For example, the prediction module 210 may receive user input from a user specifying a scheduling policy based on actual experience. Thus, the scheduler module 250 may access the predefined scheduling policy from the knowledge base 301 in block 320 to schedule the corresponding anomaly prediction.

[0058] Furthermore, the scheduler module 250 can further schedule updates of the stored anomaly detection models based on the scheduling policy, which will be explained below.

[0059] Predictive processing can determine storage devices with different failure probabilities and classify them into several sampling ranges. Each sampling range may include a range of storage devices among multiple storage devices, for example, storage devices in the same location (e.g., server room, rack, column, etc.), storage devices with similar performance, or storage devices with similar device monitoring data. For example, if the temperature in a first server room is abnormally high, while the temperature in a second server room is normal, a sampling range may be determined that includes storage devices in the first server room that are more likely to fail than another sampling range that includes storage devices in the second room.

[0060] Next, a sampling ratio may be assigned to each sampling range. The sampling ratio may represent the proportion of storage devices to be sampled within the sampling range. In some embodiments, a sampling range containing devices that are likely to fail may be assigned a higher sampling ratio than a sampling range containing devices that are less likely to fail. In the example above, a sampling range containing storage devices in the first server room may be assigned a higher sampling ratio than a sampling range containing storage devices in the second room.

[0061] In some embodiments, phase-based prediction may include two or more phases of anomaly prediction performed sequentially. For example, predictions in later phases (e.g., device monitoring data anomaly prediction) may be implemented based on the results of predictions in earlier phases (e.g., environmental anomaly prediction). Thus, high-risk devices may be filtered for more detailed storage device failure prediction / detection.

[0062] Figure 4 shows a flowchart of an exemplary process 400 for phase-based prediction according to some embodiments of the present disclosure.

[0063] In the first phase, in block 410, environmental anomaly prediction may be performed for multiple storage devices via an environmental anomaly prediction model, based on, for example, environmental data, metadata, and maintenance data. Note that different environmental anomaly prediction models may be used in this phase for different types of storage devices. In this way, in block 415, it may be determined whether each storage device is in an abnormal or normal environment. If each storage device is in a normal environment, in block 420, a first sampling range may be determined that includes the storage devices in the normal environment. A first sampling ratio, such as a normal sampling ratio of 5%, may be assigned to the first sampling range. Otherwise, each storage device is in an abnormal environment, and storage devices in an abnormal environment may be further processed in the next phase of prediction, i.e., the second phase.

[0064] In the second phase, in block 425, anomaly prediction may be performed for storage devices in an abnormal environment via an anomaly prediction model, based on, for example, performance data, metadata, and maintenance data. Note that different anomaly prediction models may be used in this phase for different types of storage devices. In this way, in block 430, it may be determined whether each storage device is operating abnormally or normally. If each storage device is operating normally, in block 435, a second sampling range may be determined that includes storage devices in an abnormal environment that are performing normally. The second sampling range may be assigned a second sampling ratio higher than the first sampling ratio, for example, 20%. If, instead, each storage device is operating abnormally, storage devices operating abnormally in an abnormal environment may be further processed in the next phase of prediction, i.e., the third phase.

[0065] In the third phase, in block 440, device monitoring data anomaly prediction may be performed for storage devices that are malfunctioning in an abnormal environment, based on, for example, device monitoring data, metadata, and maintenance data, via a device monitoring data anomaly prediction model. Note that different device monitoring data anomaly prediction models may be used in this phase for different types of storage devices. In this way, in block 445, it may be determined whether the device monitoring data for each storage device is abnormal or normal. If the device monitoring data for each storage device is normal, in block 450, a third sampling range may be determined that includes storage devices showing an abnormal environment, malfunction, and normal device monitoring data. The third sampling range may be assigned a third sampling ratio higher than the second sampling ratio, for example, 50%. Also, in block 455, a fourth sampling range may be determined that includes storage devices showing an abnormal environment, malfunction, and abnormal device monitoring data. The fourth sampling range may be assigned a fourth sampling ratio still higher than the third sampling ratio, for example, 100%.

[0066] Therefore, the four sampling ranges and corresponding sampling ratios determined by the prediction module 210 can be output to the acquisition module 220. Based on the above example, in process 400, the fourth sampling range may represent high-risk storage devices, which should be given more attention when performing device failure detection, while the first sampling range may represent low-risk storage devices, which can receive less attention in order to save resources, reduce costs, and improve efficiency.

[0067] So as can be understood, process 400 is described for illustrative purposes only, and other suitable details (including the addition, removal, and modification of one or more blocks) may also be achieved in some other embodiments of this disclosure. For example, process 400 may comprise two or more phases. The sampling range may be further subdivided based on monitoring data, depending on the actual need.

[0068] Returning to Figure 3, after the phase-based prediction process, in block 330, the acquisition module 220 may acquire a sampled dataset by selecting a group of storage devices from each sampling range at the corresponding sampling ratio.

[0069] In some embodiments, for each sampling range, storage devices may be randomly selected based on the corresponding sampling ratio. For example, if a sampling range is assigned a sampling ratio of 20%, then 20% of the storage devices in the sampling range may be randomly selected. In this way, a group of storage devices may include the storage devices selected from each sampling range. As can be understood, the group may contain more high-risk storage devices and fewer low-risk storage devices.

[0070] With respect to the exemplary process 400 discussed above, storage devices representing 5% of the first sampling range, 20% of the second sampling range, 50% of the third sampling range, and 100% of the fourth sampling range may be selected to form a group of storage devices to be further detected.

[0071] Therefore, the sampling dataset may include real-time monitoring data associated with a group of storage devices. This allows the detection range to be narrowed from multiple storage devices to a group of storage devices within that group, and the processing data to be reduced from large-scale monitoring data to appropriate features, i.e., monitoring data associated with a group of storage devices. Because the monitoring data required for prediction / detection is significantly reduced, detection costs may be lowered, while detection efficiency may be improved. The process for generating the sampling dataset may be referred to as an active sampling process.

[0072] Next, in block 340, the detection module 230 may detect device failures for a group of storage devices based on the sampled dataset. In some embodiments, the detection module 230 may apply a device failure prediction model to the sampled dataset. For example, the device failure prediction model may be at least one of a classification model, a regression model, a clustering model, or a heuristic model.

[0073] Similar to the anomaly prediction model discussed above, the device failure prediction model can be pretrained and stored in the knowledge base 301. Alternatively, the model generation / update module 240 may generate a device failure prediction model in block 350 based on an external benchmark database and a historical monitoring database of multiple storage devices. Each model is trained with benchmark data 308 and historical monitoring data 307 associated with multiple storage devices. Repeated explanations may be omitted in this specification. The generated device failure prediction model can then be stored in the knowledge base 301. Thus, the detection module 230 can access the pretrained or generated failure prediction model from the knowledge base 301 to determine failure detection.

[0074] Furthermore, device failure detection can also be performed based on a scheduling policy. For example, a scheduling policy may indicate the appropriate device failure prediction model to be adopted for a particular storage device and / or the timing of updates for each model. The scheduling policy may be predefined and stored in the knowledge base 301. Thus, the scheduler module 250 can access the predefined scheduling policy from the knowledge base 301 and determine an appropriate device failure prediction model for a storage device based on the scheduling policy. Furthermore, the scheduler module 250 may further schedule updates of the stored device failure prediction model based on the scheduling policy, which will be described below.

[0075] Therefore, detection result 345 may be output for further action, such as repairing a faulty device and / or replacing the faulty device with a new one.

[0076] In some embodiments, detected device failures may be stored in a failure base 302. Stored device failures may also be labeled with a corresponding failure type. For example, detected device failures may be classified into different failure types, such as Unrecognized (unable to establish a normal I / O path to the disk), Stopped (the disk is responsive but not functioning), Failed (the disk has reported a SMART trip or has an excessively high rate of uncorrected read errors), Read-only (the disk is readable but unable to write to certain sectors), and / or Slow (the disk performance is low compared to its peers, and the system only reads from the disk when necessary to avoid data loss).

[0077] Therefore, the detected faults, along with their corresponding fault types, can be stored, and a fault base 302 can be built and maintained for further processing, such as updating the corresponding models and policies.

[0078] In some embodiments, in block 350, the model generation / update module 240 may update each model stored in the knowledge base 301 based on the failure base 302, the history monitoring database, and the external benchmark database. The updated models can then be used for the next process of device failure detection. In some embodiments, model updates can be scheduled by the scheduler module 250 based on a scheduling policy.

[0079] Figure 5 shows an exemplary block diagram of a scheduler module 250 according to an embodiment of the present disclosure.

[0080] As shown in Figure 5, in some embodiments, the scheduler module 250 may include an evaluation submodule 510, a policy determination submodule 520, and / or a scheduler submodule 530, etc.

[0081] In some embodiments, the evaluation submodule 510 may assess the need for scheduling based on the failure base 302, the history monitoring database 502, and the external benchmark database 501. For example, the evaluation submodule 510 may determine the normal failure rate of a particular type of storage device from the external benchmark database 501 and the history monitoring database 502. The evaluation submodule 510 may also determine the detected failure rate of a particular type of storage device from the failure base 302. The evaluation submodule 510 may then compare the normal failure rate with the detected failure rate and obtain the difference between them. If the difference is higher than a pre-set threshold, the evaluation submodule 510 may determine that there is a need for scheduling, such as adjusting the prediction frequency (or sampling frequency) and / or updating the adopted model (anomaly prediction model, device failure prediction model).

[0082] For example, if the normal failure rate is 4% and the detected failure rate is 9%, the difference is 5%, which is higher than the pre-set threshold (e.g., 3%). Therefore, the need for scheduling may be determined by increasing the frequency of predictions, for example, from once a day to twice a day. The evaluation submodule 510 may also determine that the adopted model should be updated.

[0083] In some embodiments, the policy determination submodule 520 may select a scheduling policy from scheduling policies stored in the knowledge base 301 based on scheduling needs. For example, a scheduling policy may indicate prediction timing (sampling timing), prediction frequency (sampling frequency), anomaly prediction model / device failure prediction model to be employed, and / or update timing for each model.

[0084] Furthermore, the scheduler submodule 530 can schedule prediction processing based on a determined policy, such as adjusting the prediction frequency (or sampling frequency). In addition, the scheduler submodule 530 can schedule updates for each model, for example, by triggering the model generation / update module 240 to update each model based on detected device failures stored in the knowledge base 301, the history monitoring database 502, and the external benchmark database 501.

[0085] Therefore, each model can be updated / retrained using new training data (e.g., detected device failures, new inputs from historical monitoring databases and external benchmark databases). Consequently, prediction / detection results can be dynamically adjusted. Detection efficiency can be improved, thereby facilitating cost reduction and avoiding wasted resources.

[0086] Figure 6 shows an exemplary flowchart of a method 600 for device failure detection according to an embodiment of the present disclosure. The process may be implemented by a computing device such as the computer 101 shown in Figure 1.

[0087] In block 610, the computing device may perform phase-based predictions on multiple storage devices to determine multiple sampling ranges and corresponding sampling ratios. Each sampling range may include at least one of the multiple storage devices.

[0088] Therefore, an efficient balance of accuracy, performance, and cost can be provided for fault detection in storage devices.

[0089] In some embodiments, the step of performing phase-based predictions for multiple storage devices may include: performing phase-based predictions based on real-time monitoring data associated with multiple storage devices and multiple models. Each model is trained on benchmark data and historical monitoring data associated with multiple storage devices. Therefore, historical monitoring data and benchmark data, including open data about specific manufacturers, models, and batches, may be used to facilitate detection via supervised or unsupervised algorithms.

[0090] In some embodiments, the benchmark model may include at least two of the following models: an environmental anomaly prediction model, an operational anomaly prediction model, and a SMART anomaly prediction model.

[0091] In some embodiments, phase-based forecasting includes at least two forecasting phases. Later forecasting phases are performed based on the results of earlier forecasting phases. Therefore, the forecasting scope and cost can be reduced by using appropriate features.

[0092] In some embodiments, the step of performing phase-based prediction for multiple storage devices further includes the following phases: In a first phase, environmental anomalies on multiple storage devices may be predicted in order to determine a first sampling range having a first sampling ratio. The first sampling range includes storage devices in a normal environment. In a second phase, operational anomalies on storage devices in an abnormal environment may be predicted in order to determine a second sampling range having a second sampling ratio. The second sampling range includes storage devices that are operating normally in an abnormal environment. In a third phase, device monitoring data anomalies on storage devices that are operating abnormally in an abnormal environment may be predicted in order to determine a third sampling range having a third sampling ratio and a fourth sampling range having a fourth sampling ratio. The third range includes storage devices that are operating abnormally in an abnormal environment and have normal device monitoring data. The fourth range includes storage devices that are operating abnormally in an abnormal environment and have abnormal device monitoring data. Furthermore, the first sampling ratio is lower than the second sampling ratio, which is lower than the third sampling ratio, and which is lower than the fourth sampling ratio. Therefore, different sampling ranges can be assigned to different sampling ratios in order to filter out high-risk devices for predicting device failures, which will be described in more detail later.

[0093] In block 620, a computing device may obtain a sampled dataset by selecting a group of storage devices from each sampling range having a corresponding sampling ratio.

[0094] In block 630, the computing device may detect device failures for a group of storage devices based on a sampled dataset.

[0095] In some embodiments, the execution, acquisition, and detection stages may be implemented multiple times, and the execution stage is scheduled based on a scheduling policy. Thus, predictions may be performed on demand based on a dynamic scheduling policy and recent detection results.

[0096] In some embodiments, a fault base can be generated based on detected device failures. Therefore, for active sampling, the fault base can be built and maintained to store detected anomalies.

[0097] In some embodiments, scheduling needs can be assessed based on failure-based, benchmark data, and historical monitoring data. A scheduling policy can be selected based on these scheduling needs. Therefore, forecasts can be dynamically adjusted based on actual needs.

[0098] In some embodiments, each model is scheduled to be updated based on scheduling needs. Therefore, each model can be dynamically updated based on actual needs.

[0099] In some embodiments, the step of detecting device failures for a group of storage devices based on a sampled dataset may include the step of detecting device failures based on real-time monitoring data associated with the group of storage devices and multiple device failure prediction models. Each device failure prediction model is trained on benchmark data and historical monitoring data associated with multiple storage devices. Therefore, to facilitate detection, historical monitoring data and benchmark data, including open data for specific manufacturers, models, and batches, may be used with supervised or unsupervised algorithms.

[0100] It should be noted that the sequence of blocks described in the above embodiments is for illustrative purposes only. Any other suitable sequence (including the addition, deletion, and / or modification of at least one block) may be implemented to determine the corresponding embodiment.

[0101] In addition, in some embodiments of this disclosure, a system for detecting device failures may be provided. The system may comprise one or more processors, memory linked to at least one of the one or more processors, and a set of computer program instructions stored in the memory. The set of computer program instructions may be executed by at least one of the one or more processors to perform the method of the present invention.

[0102] In some other embodiments of this disclosure, a computer program product for device failure detection may be provided. The computer program product may comprise a computer-readable storage medium having program instructions embodied therein. Program instructions executable by one or more processors may cause one or more processors to execute the method of the present invention.

[0103] The present invention may be a system, method, and / or computer program product in an integration of any possible level of technical detail. The computer program product may include a computer-readable storage medium (or multiple mediums) having computer-readable program instructions that cause a processor to execute aspects of the present invention.

[0104] The descriptions of various embodiments of the present invention are presented for illustrative purposes only and are not intended to be exhaustive or limit to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the described embodiments. The terminology used herein has been selected to best describe the principles of the embodiments, their practical applications, or the technical improvements to the art found in the market, or to enable other persons skilled in the art to understand the embodiments disclosed herein.

Claims

1. A step of performing phase-based predictions on multiple storage devices by one or more processors to determine multiple sampling ranges and corresponding sampling ratios, wherein each sampling range has at least one storage device among the multiple storage devices; A step of acquiring a sampling dataset by one or more processors by selecting a group of storage devices from each of the sampling ranges having the corresponding sampling ratios; and A step in which one or more processors detect device failures for the group of storage devices based on the sampling dataset. Computer implementation methods, including those mentioned above.

2. The step of performing the phase-based prediction for the aforementioned multiple storage devices is: A step in which one or more processors perform the phase-based prediction based on real-time monitoring data associated with the plurality of storage devices and the plurality of models, where each of the plurality of models is trained with benchmark data and historical monitoring data associated with the plurality of storage devices. The computer implementation method according to claim 1, further comprising:

3. The phase-based prediction has at least two prediction phases; The subsequent prediction phase is performed based on the results of the previous prediction phase. A computer implementation method according to any of the above claims.

4. The step of performing the phase-based prediction for the aforementioned multiple storage devices is: In the first phase, the one or more processors predict environmental anomalies on the plurality of storage devices in order to determine a first sampling range having a first sampling ratio, wherein the first sampling range includes the storage devices in a normal environment; In the second phase, the one or more processors predict operational abnormalities on the storage device in an abnormal environment in order to determine a second sampling range having a second sampling ratio, wherein the second sampling range includes the storage device operating normally in the abnormal environment; and In the third phase, in order to determine a third sampling range having a third sampling ratio and a fourth sampling range having a fourth sampling ratio, the one or more processors predict abnormal device monitoring data on the storage device that is in an abnormal environment and operating abnormally, wherein the third range includes storage devices that are in an abnormal environment, operating abnormally, and having normal device monitoring data, and the fourth range includes storage devices that are in an abnormal environment, operating abnormally, and having abnormal device monitoring data. Including; The first sampling ratio is lower than the second sampling ratio, which is lower than the third sampling ratio, which is lower than the fourth sampling ratio. A computer implementation method according to any of the above claims.

5. A step in which the one or more processors implement the steps of executing, acquiring, and detecting multiple times, wherein the steps to be executed are scheduled based on a scheduling policy. A computer implementation method according to any of the preceding claims, further comprising:

6. The step of generating a fault base based on the detected device failure by the one or more processors. A computer implementation method according to any of the preceding claims, further comprising:

7. The process further includes the step of evaluating the need for scheduling based on the failure base, the benchmark data, and the historical monitoring data using one or more of the aforementioned processors. The scheduling policy is selected based on the scheduling needs. The computer implementation method according to claim 6.

8. The computer implementation method according to claim 7, wherein each of the plurality of models is scheduled to be updated based on the scheduling needs.

9. The step of detecting device failures for the group of storage devices based on the sampling dataset is: The step of detecting a device failure by the one or more processors based on real-time monitoring data associated with the group of storage devices and a plurality of device failure prediction models, where each of the plurality of device failure prediction models is trained with benchmark data and historical monitoring data associated with the plurality of storage devices. A computer implementation method according to any of the preceding claims, including the method described above.

10. One or more computer processors; Memory connected to at least one of the aforementioned processors; and A set of computer program instructions stored in the memory, wherein the set of computer program instructions is configured on at least one of the one or more computer processors. To determine multiple sampling ranges and corresponding sampling ratios, phase-based predictions are performed for multiple storage devices, where each sampling range includes at least one of the multiple storage devices; A sampling dataset is obtained by selecting a group of storage devices from each of the sampling ranges having the corresponding sampling ratios; and Based on the sampling dataset, device failures are detected for the group of storage devices. To execute an action; A computer system, including a computer system.

11. Performing the phase-based prediction for the aforementioned multiple storage devices: This includes performing the phase-based prediction based on real-time monitoring data associated with the plurality of storage devices and the plurality of models, Here, each of the aforementioned models is trained with benchmark data and historical monitoring data associated with the aforementioned storage devices. The computer system according to claim 10.

12. The system according to any one of claims 10 to 11, wherein the phase-based prediction has at least two prediction phases, and the later prediction phase is performed based on the results of the earlier prediction phase.

13. Performing the phase-based prediction for the aforementioned multiple storage devices: In the first phase, in order to determine a first sampling range having a first sampling ratio, environmental anomalies on the plurality of storage devices are predicted, wherein the first sampling range includes the storage devices in a normal environment; In the second phase, in order to determine a second sampling range having a second sampling ratio, predict operational abnormalities on the storage device in an abnormal environment, wherein the second sampling range includes the storage device operating normally in the abnormal environment; and In the third phase, in order to determine a third sampling range having a third sampling ratio and a fourth sampling range having a fourth sampling ratio, anomalies in device monitoring data on the storage device that is in an abnormal environment and operating abnormally are predicted, where the third range includes storage devices that are in an abnormal environment, operating abnormally, and having normal device monitoring data, and the fourth range includes storage devices that are in an abnormal environment, operating abnormally, and having abnormal device monitoring data. Including; The first sampling ratio is lower than the second sampling ratio, which is lower than the third sampling ratio, which is lower than the fourth sampling ratio. The computer system according to any one of claims 10 to 12.

14. The aforementioned action is: The execution step, the acquisition step, and the detection step are implemented multiple times, wherein the execution step is scheduled based on a scheduling policy. A computer system according to any one of claims 10 to 13, further comprising:

15. The aforementioned action is: To generate a fault base based on the detected device failure. A computer system according to any one of claims 10 to 14, further comprising:

16. The aforementioned action is: Further including evaluating the need for scheduling based on the failure base, the benchmark data, and the historical monitoring data; The scheduling policy is selected based on the scheduling needs; Each of the aforementioned models is scheduled to be updated based on the scheduling needs. The computer system according to claim 15.

17. Based on the aforementioned sampling dataset, device failures can be detected for the group of storage devices: The device failure is detected based on real-time monitoring data associated with the group of storage devices and multiple device failure prediction models, where each of the multiple device failure prediction models is trained with benchmark data and historical monitoring data associated with the multiple storage devices. A computer system according to any one of claims 10 to 16, including the computer system according to any one of claims 10 to 16.

18. A computer program product comprising a computer-readable storage medium having program instructions thus realized, wherein the program instructions are transmitted to one or more processors: Phase-based prediction is performed on multiple storage devices to determine multiple sampling ranges and corresponding sampling ratios, where each sampling range includes at least one of the multiple storage devices; A sampling dataset is obtained by selecting a group of storage devices from each of the sampling ranges having the corresponding sampling ratios; and Based on the sampling dataset, device failures are detected for the group of storage devices. A computer program product that is executable by one or more processors to perform an action.

19. Performing the phase-based prediction for the aforementioned multiple storage devices: The system includes performing phase-based predictions based on real-time monitoring data associated with the multiple storage devices and multiple models. Each of the aforementioned models is trained with benchmark data and historical monitoring data associated with the aforementioned storage devices. The computer program product according to claim 18.

20. The phase-based prediction has at least two prediction phases; The subsequent prediction phase is performed based on the results of the previous prediction phase. A computer program product according to any one of claims 18 to 19.

Images