Fault detection method and device, distributed storage cluster and storage medium

Abstract

The invention discloses a fault detection method and device, a distributed storage cluster and a storage medium, and the method comprises the steps: under the condition that a physical disk I / O failure alarm reported by a target server node is detected, collecting data such as physical disk alarms, physical disk states and performances, cluster node states and the like of a plurality of physical disks, the physical disk alarm data at least comprises a physical disk I / O failure alarm; determining the number of failed physical disks in the plurality of physical disks according to the physical disk alarm data; under the condition that the number of the faulty physical disks is equal to one, determining the faulty physical disks as target faulty disks; polling the disk state of the target fault disk to obtain a disk state polling result of the target fault disk; and determining the fault type of the target fault disk according to the physical disk alarm data and the disk state polling result. According to the method, the accuracy of disk fault detection in the distributed storage cluster can be improved.

Description

Technical Field

[0001] This application relates to the field of distributed storage technology, and in particular to a fault detection method, apparatus, distributed storage cluster and storage medium. Background Technology

[0002] With the widespread application of distributed storage technology, the reliability and intelligent operation and maintenance level of distributed storage clusters have become key factors in ensuring the continuity of data services. In large-scale distributed storage clusters, physical disks are among the most vulnerable components, and their condition directly affects the stability of the entire system.

[0003] Currently, distributed storage clusters typically employ redundancy mechanisms such as multiple replicas or erasure coding to ensure data reliability and service continuity. When an I / O (Input / Output) anomaly occurs on the physical disk of a server node in the cluster, fault identification generally relies on alarm information reported by the server node, triggering disk isolation and data reconstruction processes. During this process, the system's response to physical disk failures is mostly based on static rule judgments. For example, upon detecting a physical disk I / O failure alarm reported by a server node, it directly assumes that the physical disk has suffered a permanent hardware failure and immediately initiates isolation and reconstruction operations. However, in practical applications, physical disks may experience brief I / O delays or temporary communication interruptions. If such temporary network failures are misjudged as permanent hardware failures, it will lead to unnecessary disk isolation and data reconstruction, not only increasing system load but also potentially affecting the stability and availability of storage services.

[0004] Therefore, accurately determining the nature of the fault after detecting a physical disk I / O failure alarm reported by a server node, and avoiding over-response caused by misjudgment, has become an important issue in improving the intelligent operation and maintenance level of distributed storage clusters. Summary of the Invention

[0005] The main objective of this application is to provide a fault detection method, apparatus, distributed storage cluster, and storage medium, aiming to improve the accuracy of disk fault detection in distributed storage clusters.

[0006] To achieve the above objectives, this application proposes a fault detection method applied to a distributed storage cluster, wherein the distributed storage cluster includes a target server node, and the target server node includes multiple physical disks. The fault detection method includes: When a physical disk I / O failure alarm is detected reported by the target server node, physical disk alarm data of the multiple physical disks is collected, wherein the physical disk alarm data includes at least the physical disk I / O failure alarm; Based on the physical disk alarm data, determine the number of faulty physical disks among the plurality of physical disks; If the number of faulty physical disks is equal to one, the faulty physical disk is identified as the target faulty disk. The disk status of the target faulty disk is polled to obtain the disk status polling result of the target faulty disk; Based on the physical disk alarm data and the disk status polling results, the fault type of the target faulty disk is determined.

[0007] Furthermore, to achieve the above objectives, this application also proposes a fault detection device applied to a distributed storage cluster, the distributed storage cluster including a target server node, the target server node including multiple physical disks, and the fault detection device comprising: The data acquisition module is used to acquire physical disk alarm data of the multiple physical disks when a physical disk I / O failure alarm is detected reported by the target server node, wherein the physical disk alarm data includes at least the physical disk I / O failure alarm. The quantity determination module is used to determine the number of faulty physical disks among the plurality of physical disks based on the physical disk alarm data; The target determination module is used to determine the faulty physical disk as the target faulty disk when the number of faulty physical disks is equal to one. The status polling module is used to poll the disk status of the target faulty disk and obtain the disk status polling result of the target faulty disk. The fault analysis module is used to determine the fault type of the target faulty disk based on the physical disk alarm data and the disk status polling results of the target faulty disk.

[0008] In addition, to achieve the above objectives, this application also proposes a distributed storage cluster, which includes a target server node, the target server node includes multiple physical disks, and the distributed storage cluster uses the fault detection method described above for fault detection.

[0009] In addition, to achieve the above objectives, this application also proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the fault detection method described above.

[0010] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the fault detection method described above.

[0011] This application provides a fault detection method, apparatus, distributed storage cluster, and storage medium, relating to the field of distributed storage technology. The fault detection method is applied to a distributed storage cluster, which includes a target server node and multiple physical disks. The fault detection method includes: upon detecting a physical disk I / O failure alarm reported by the target server node, collecting physical disk alarm data from the multiple physical disks, wherein the physical disk alarm data includes at least the physical disk I / O failure alarm; determining the number of faulty physical disks among the multiple physical disks based on the physical disk alarm data; if the number of faulty physical disks is equal to one, identifying the faulty physical disk as a target faulty disk; polling the disk status of the target faulty disk to obtain a disk status polling result; and determining the fault type of the target faulty disk based on the physical disk alarm data and the disk status polling result.

[0012] This application embodiment effectively solves the problem of misjudgment of disk failure caused by relying solely on static alarm rules in related technologies by introducing a joint analysis mechanism of disk status polling and multi-dimensional alarm data, and significantly improves the accuracy of judgment and the level of intelligent operation and maintenance of distributed storage systems when facing I / O anomalies. Specifically, when the cluster detects a physical disk I / O failure alarm reported by the target server node, it does not immediately perform high-cost operations such as disk isolation or data reconstruction. Instead, it first collects alarm data from all physical disks on the target server node. Based on this alarm data, it accurately identifies the number of abnormal faulty physical disks currently existing on the target server node. If it is confirmed that there is only one faulty physical disk, it is identified as the target faulty disk, thereby eliminating the complex interference caused by multiple disk concurrency anomalies and focusing on the typical single disk failure scenario. Subsequently, by continuously polling the target faulty disk, it obtains real-time disk status information such as its health status, running status, and response latency, forming a disk status polling result. Finally, by combining the original physical disk alarm data and the real-time disk status information obtained from the polling, a comprehensive judgment is made to accurately distinguish between different types of failures, such as temporary network failures and permanent hardware failures.

[0013] This application avoids unnecessary disk isolation and data reconstruction operations caused by misjudging brief I / O jitter as hardware failure, reduces cluster resource consumption, minimizes the impact on normal business, improves the stability and availability of storage services, and enhances the cluster's ability to identify real hardware failures, achieving more accurate and intelligent fault response, and providing strong support for the efficient and reliable operation and maintenance of large-scale distributed storage clusters. Attached Figure Description

[0014] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0015] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a flowchart illustrating the first embodiment of the fault detection method of this application; Figure 2 This is a flowchart illustrating the second embodiment of the fault detection method of this application; Figure 3 This is a schematic diagram of a fault detection scenario in a specific embodiment of this application; Figure 4 This is a schematic diagram of the data acquisition framework in a specific embodiment of this application; Figure 5 This is a schematic diagram of the module structure of the fault detection device in the embodiments of this application.

[0017] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0018] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0019] Currently, distributed storage clusters typically employ redundancy mechanisms such as multiple replicas or erasure coding to ensure data reliability and service continuity. When a physical disk on a server node in the cluster experiences an I / O anomaly, the system generally relies on alarm information reported by the server node for fault identification and triggers disk isolation and data reconstruction processes. During this process, the system's response to physical disk failures is mostly based on static rule judgments. For example, upon detecting a physical disk I / O failure alarm reported by a server node, it directly assumes that the physical disk has suffered a permanent hardware failure and immediately initiates isolation and reconstruction operations. However, in practical applications, physical disks may experience brief I / O delays or temporary communication interruptions. If such temporary network failures are misjudged as permanent hardware failures, it will lead to unnecessary disk isolation and data reconstruction, increasing system load and potentially affecting the stability and availability of storage services.

[0020] Therefore, accurately determining the nature of the fault after detecting a physical disk I / O failure alarm reported by a server node, and avoiding over-response caused by misjudgment, has become an important issue in improving the intelligent operation and maintenance level of distributed storage clusters.

[0021] In contrast, the solution of this application embodiment is a fault detection method applied to a distributed storage cluster. The distributed storage cluster includes a target server node, and the target server node includes multiple physical disks. The fault detection method includes: when a physical disk I / O failure alarm is detected reported by the target server node, collecting physical disk alarm data of the multiple physical disks, wherein the physical disk alarm data includes at least the physical disk I / O failure alarm; determining the number of faulty physical disks among the multiple physical disks based on the physical disk alarm data; if the number of faulty physical disks is equal to one, identifying the faulty physical disk as a target faulty disk; polling the disk status of the target faulty disk to obtain a disk status polling result for the target faulty disk; and determining the fault type of the target faulty disk based on the physical disk alarm data and the disk status polling result.

[0022] This application embodiment effectively solves the problem of misjudgment of disk failure caused by relying solely on static alarm rules in related technologies by introducing a joint analysis mechanism of disk status polling and multi-dimensional alarm data, and significantly improves the accuracy of judgment and the level of intelligent operation and maintenance of distributed storage systems when facing I / O anomalies. Specifically, when the cluster detects a physical disk I / O failure alarm reported by the target server node, it does not immediately perform high-cost operations such as disk isolation or data reconstruction. Instead, it first collects alarm data from all physical disks on the target server node. Based on this alarm data, it accurately identifies the number of abnormal faulty physical disks currently existing on the target server node. If it is confirmed that there is only one faulty physical disk, it is identified as the target faulty disk, thereby eliminating the complex interference caused by multiple disk concurrency anomalies and focusing on the typical single disk failure scenario. Subsequently, by continuously polling the target faulty disk, it obtains real-time disk status information such as its health status, running status, and response latency, forming a disk status polling result. Finally, by combining the original physical disk alarm data and the real-time disk status information obtained from the polling, a comprehensive judgment is made to accurately distinguish between different types of failures, such as temporary network failures and permanent hardware failures.

[0023] This application avoids unnecessary disk isolation and data reconstruction operations caused by misjudging brief I / O jitter as hardware failure, reduces cluster resource consumption, minimizes the impact on normal business, improves the stability and availability of storage services, and enhances the cluster's ability to identify real hardware failures, achieving more accurate and intelligent fault response, and providing strong support for the efficient and reliable operation and maintenance of large-scale distributed storage clusters.

[0024] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0025] This application proposes a fault detection method according to a first embodiment.

[0026] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the fault detection method of this application.

[0027] In this embodiment, the fault detection method is applied to a distributed storage cluster, which includes a target server node, and the target server node includes multiple physical disks. The fault detection method may include steps S100~S500: Step S100: When a physical disk I / O failure alarm is detected reported by the target server node, physical disk alarm data of multiple physical disks is collected, wherein the physical disk alarm data includes at least physical disk I / O failure alarms. Those skilled in the art will understand that a distributed storage cluster refers to a system architecture consisting of multiple server nodes interconnected via a network, used for unified management and provision of data storage services. Through data redundancy mechanisms such as multiple replicas or erasure coding, it achieves high availability, high reliability, and load balancing of data. A server node refers to an independent physical or virtual computing unit within the distributed storage cluster, typically a physical server with its own CPU (Central Processing Unit), memory, operating system, and storage devices. As the basic unit for data storage and service, the server node is responsible for executing I / O requests, participating in data replication, reporting operational status, and triggering alarm mechanisms when anomalies occur.

[0028] Preferably, in this embodiment, the distributed storage cluster is an S2D (Storage Space Direct) distributed storage cluster.

[0029] It should be noted that, in this embodiment, the target server node specifically refers to a server node in the distributed storage cluster that is currently in normal service and has not been isolated. This target server node has not yet been marked as a faulty server node or entered maintenance mode; it is still receiving and processing I / O requests normally and is an active component of the cluster's data plane. This limitation excludes server nodes that are isolated, offline, or under maintenance in the distributed storage cluster, ensuring that the fault detection method provided in this embodiment only responds to "active anomalies" that are still affecting cluster stability. This avoids duplicate processing of known faulty server nodes, improving diagnostic efficiency and resource utilization.

[0030] It should also be noted that, in this embodiment, physical disks specifically refer to actual hardware storage devices that exist in the target server node, are currently in normal service, and are not isolated, including HDDs (Hard Disk Drives) and SSDs (Solid State Drives). Physical disks are one of the most vulnerable components in a distributed storage system, and their status directly affects data accessibility and system stability. Each physical disk has unique identification information (such as device path, serial number, globally unique name, etc., also known as disk identifier) ​​and supports standard command interfaces (such as Minicomputer System Interface, Serial Advanced Technology Accessories, Non-volatile Memory Fast Access, etc.) for status query and management.

[0031] It should be noted that, in this embodiment, a physical disk I / O failure alarm refers to an event notification generated by the operating system or disk controller and reported to the cluster management layer when a physical disk in the target server node fails to perform I / O operations such as read, write, or synchronization due to reasons such as underlying hardware errors, communication interruptions, or command timeouts. This physical disk I / O failure alarm typically includes fields such as a timestamp, disk identifier, and error code.

[0032] For example, in practical applications, such as the Windows operating system, the physical disk I / O failure alarm can be a 203 alarm reported by the storage space driver in the Windows event log, that is, in the operation log with the log path Microsoft-Windows-StorageSpaces-Driver / Operational, the status code indicates a 203 event indicating that the physical disk I / O operation failed.

[0033] In this embodiment, physical disk alarm data refers to the collection of various alarms reported by the target server node for all physical disks within a preset time window (e.g., the last 5 minutes). This includes not only physical disk I / O failure alarms reported by the target server node this time, but also other related alarms such as physical disk disconnection alarms and data buffer error alarms. Specifically, a physical disk disconnection alarm refers to an alarm indicating that the physical disk has lost communication with the server node; in the Windows operating system, this corresponds to alarm 15 in the system log. A data buffer error alarm refers to an alarm indicating that the disk controller or driver has encountered a cached data verification failure or a DMA (Direct Memory Access) transfer error during I / O processing; in the Windows operating system, this corresponds to alarm 153 in the system log.

[0034] This embodiment, upon detecting a physical disk I / O failure alarm reported by the target server node, does not immediately isolate or rebuild the physical disk indicated by the alarm. Instead, it collects all physical disk alarm data from all physical disks on the target server node, obtaining alarm logs for all active physical disks over a recent period. This provides more comprehensive fault context information than a single alarm, laying a data foundation for accurately defining the scope and nature of the fault. This design breaks through the coarse-grained processing mode of "responding to alarms as they appear" in traditional solutions, reflecting an evolution from "local event response" to "global state awareness."

[0035] Step S200: Based on the physical disk alarm data, determine the number of faulty physical disks among the multiple physical disks; Those skilled in the art will know that, in practical applications, physical disk-related alarms are generated with a unique disk identifier field to indicate the specific physical disk associated with the alarm.

[0036] In this embodiment, a faulty physical disk refers to a physical disk that is explicitly pointed to by at least one alarm in the physical disk alarm data.

[0037] In this embodiment, by parsing the physical disk alarm data collected in step S100 and extracting the disk identification information carried in each alarm record, all faulty physical disks currently existing in the target server node can be identified, thereby counting the number of faulty physical disks.

[0038] Step S300: If the number of faulty physical disks is equal to one, the faulty physical disk is identified as the target faulty disk. It should be noted that, in this embodiment, the target faulty disk refers to a physical disk that is further filtered from the faulty physical disks and determined by the system to be truly faulty.

[0039] In this embodiment, when the number of detected faulty physical disks is equal to one, it is determined that the current situation is a single disk failure scenario, and the root cause of the target server node reporting a physical disk I / O failure alarm is the single faulty physical disk.

[0040] Step S400: Poll the disk status of the target faulty disk to obtain the disk status polling result of the target faulty disk; In this embodiment, the disk status of the target faulty disk can be queried by periodically sending commands. The disk status may include, but is not limited to, running status, health status, and performance data (which can be obtained by querying the physical disk's performance counter).

[0041] For example, the running status and health status of the target failed disk can be obtained periodically through the Get-PhysicalDisk command in PowerShell, thereby achieving the polling of the disk status of the target failed disk.

[0042] In this embodiment, the disk status polling result refers to a set of time-series status data obtained by continuously performing disk status query operations on the target faulty disk within a preset polling period (such as continuously for 10 seconds, performing disk status query operations once per second), or a comprehensive status judgment result generated after analyzing and summarizing the set of time-series status data.

[0043] Specifically, this set of time-series state data may include: Operational Status Polling Results: Record the changes in the OperationalStatus of the target faulty disk during the polling period, such as whether it recovers from "Lost Communication" (indicating that the operation status is lost) to "OK" (indicating that the operation status is normal), or remains lost; Health status polling results: Record the changes in the HealthStatus of the target faulty disk during the polling period, such as whether it recovers from "Unhealthy" (indicating an unhealthy health status) to "Healthy" (indicating a healthy health status), or remains unhealthy throughout; Polling results of other auxiliary status indicators: such as changes in SMART (Self-Monitoring Analysis and Reporting Technology) related attributes, including temperature, firmware version, number of reallocated sectors, and number of pending errors.

[0044] The overall state determination results can be divided into two categories: The first poll result indicates that the target faulty disk's operating status remains "Lost Communication" throughout the entire polling period, or the target faulty disk's health status remains "Unhealthy," indicating that the target faulty disk has experienced a hardware failure and requires disk isolation or even data reconstruction. The second round of queries indicates that during the query period, the target faulty disk's operating status has recovered from "disconnected" to "normal," or the target faulty disk's health status has recovered from "unhealthy" to "healthy." This indicates that the target faulty disk has experienced a network failure and there is no need to isolate it. Instead, the operations and maintenance personnel should be notified to perform a manual check to prevent the network failure from recurring.

[0045] For example, in one feasible implementation, step S400 described above may include steps S410 to S430: Step S410: Poll the operating status of the target faulty disk to obtain the polling result of the operating status of the target faulty disk; Step S420: Poll the health status of the target faulty disk to obtain the health status polling result of the target faulty disk; Step S430: Generate disk status polling results for the target faulty disk based on the polling results of the running status and health status of the target faulty disk.

[0046] Furthermore, in a feasible implementation, step S430 above may include steps S431 to S432: Step S431: If the polling result of the target faulty disk's running status is the first running status polling result, or the polling result of the target faulty disk's health status is the first health status polling result, then determine that the polling result of the target faulty disk's disk status is the first polling result. Step S432: If the polling result of the target faulty disk's running status is the second running status polling result, or the polling result of the target faulty disk's health status is the second health status polling result, determine that the polling result of the target faulty disk's disk status is the second polling result. Among them, the first operating status polling result indicates that the operating status is always out of contact during the polling period; the first health status polling result indicates that the health status is always unhealthy during the polling period; the second operating status polling result indicates that the operating status changes from out of contact to normal during the polling period; and the second health status polling result indicates that the health status changes from unhealthy to healthy during the polling period. This embodiment introduces a periodic polling mechanism, achieving a technological leap from "static alarm response" to "dynamic behavior observation," significantly improving the intelligence level of fault detection. Traditional methods often immediately determine a physical disk I / O failure alarm as a permanent fault and initiate isolation and reconstruction processes. This can easily misjudge temporary communication interruptions caused by transient link jitter, driver anomalies, or power fluctuations as hardware damage, leading to unnecessary data migration and resource overhead. In contrast, this method continuously queries the status of the target faulty disk (e.g., once per second for 10 seconds) to obtain time-series changes in its Operational Status, Health Status, and other SMART attributes. This accurately captures whether the disk possesses self-recovery capabilities, effectively distinguishing between temporary network faults and permanent hardware faults, significantly reducing the false positive rate.

[0047] This disk status polling mechanism targeting faulty disks not only provides crucial information for refined fault classification but also enables the system to possess a self-healing awareness capability of "observation-judgment-decision": for disks with recoverable status, the system can postpone isolation and only enhance monitoring to avoid high-overhead data reconstruction operations; for disks that are continuously disconnected or whose health status is deteriorating, timely isolation and reconstruction are performed to ensure data reliability. Therefore, this embodiment improves the accuracy of fault identification while optimizing the rationality of operation and maintenance strategies, reducing unnecessary network bandwidth and computing resource consumption, and comprehensively enhancing the stability, availability, and operational efficiency of the distributed storage cluster.

[0048] Step S500: Determine the fault type of the target faulty disk based on the physical disk alarm data and disk status polling results.

[0049] It should be noted that in this embodiment, the fault types of the target faulty disk are mainly divided into two categories—network faults and hardware faults. Network faults typically manifest as temporary connectivity problems, such as network jitter or momentary switch disconnections, and are characterized by their ability to recover on their own after a certain period. Hardware faults involve problems with the internal components of the disk, such as media damage or aging electronic components, and usually require hardware replacement to resolve.

[0050] For example, in one feasible implementation, step S500 described above may include steps S510 to S520: Step S510: If the error code of the physical disk I / O failure alarm of the target faulty disk is the first error code, the physical disk alarm data includes the physical disk disconnection alarm of the target faulty disk, the physical disk alarm data includes the data buffer error alarm of the target faulty disk, or the disk status polling result is the first polling result, the fault type of the target faulty disk is determined to be a hardware fault. Step S520: If the error code of the physical disk I / O failure alarm of the target faulty disk is the second error code, the physical disk alarm data of the target faulty disk does not include physical disk disconnection alarm and data buffer error alarm, and the disk status polling result is the second polling result, then the fault type of the target faulty disk is determined to be a network fault. The first error code indicates a hardware error, and the second error code indicates a connection error.

[0051] It should be noted that, in this embodiment, a hardware error refers to permanent or irreversible physical damage or functional failure of internal components of the physical disk (such as platters, heads, controller chip, and cache module), resulting in the inability to read or write data normally. A connection error refers to a situation where the physical disk itself is functioning normally, but due to a momentary interruption, signal interference, or protocol abnormality in the external connection link (such as SAS / SATA cables, backplane, driver, or network switch), the server node is temporarily unable to access the disk.

[0052] The first error code refers to a physical disk I / O failure alarm that explicitly indicates the cause of the I / O failure is an unrecoverable defect in the physical disk (i.e., a hardware error). The second error code refers to a physical disk I / O failure alarm that explicitly indicates the cause of the I / O failure is not due to damage to the storage medium itself, but rather due to an interruption in the transmission path or a temporary connection problem (i.e., a connection error).

[0053] This implementation method achieves accurate identification and intelligent differentiation of physical disk fault types by constructing a fault classification logic based on multi-dimensional criteria fusion. Traditional fault handling mechanisms often rely on single alarm events for rapid response, making it difficult to distinguish the essential differences between temporary link jitter and permanent hardware damage, easily leading to false isolation or missed responses. This implementation method innovatively combines error code semantic analysis, alarm type combination judgment, and dynamic status polling results to form a decision model with context-aware capabilities. When the target faulty disk exhibits a first error code, a disconnection alarm, a buffer error alarm, or presents a first polling result of "always disconnected / unhealthy," the system determines it as a hardware fault, immediately isolates the disk, and decides whether to rebuild the data as needed. Conversely, if only a second error code (connection error) appears, without serious hardware alarms, and the disk status is recoverable (second polling result), it is determined as a network fault, and an observation and monitoring strategy is adopted, or maintenance personnel are notified for manual inspection. This judgment mechanism based on multi-source evidence collaborative verification not only significantly improves the accuracy and scientific nature of fault diagnosis, but also endows the system with the ability of "delayed decision-making and intelligent self-healing". While ensuring data reliability, it effectively avoids excessive operation and maintenance caused by instantaneous anomalies, thereby achieving a better balance between stability, resource efficiency and automated operation and maintenance level. This is the core technological breakthrough of this application in realizing intelligent fault detection.

[0054] By combining physical disk alarm data with disk status polling results, this embodiment can achieve fine-grained classification of fault types. The advantage of this embodiment is that it does not rely solely on the initially received alarm signal, but comprehensively considers the subsequent behavior patterns of the disk, making the final fault determination more scientific and reasonable. This facilitates more targeted repair measures, reduces unnecessary maintenance costs, and improves the overall stability and availability of the system.

[0055] This application embodiment effectively solves the problem of misjudgment of disk failure caused by relying solely on static alarm rules in related technologies by introducing a joint analysis mechanism of disk status polling and multi-dimensional alarm data, and significantly improves the accuracy of judgment and the level of intelligent operation and maintenance of distributed storage systems when facing I / O anomalies. Specifically, when the cluster detects a physical disk I / O failure alarm reported by the target server node, it does not immediately perform high-cost operations such as disk isolation or data reconstruction. Instead, it first collects alarm data from all physical disks on the target server node. Based on this alarm data, it accurately identifies the number of abnormal faulty physical disks currently existing on the target server node. If it is confirmed that there is only one faulty physical disk, it is identified as the target faulty disk, thereby eliminating the complex interference caused by multiple disk concurrency anomalies and focusing on the typical single disk failure scenario. Subsequently, by continuously polling the target faulty disk, it obtains real-time disk status information such as its health status, running status, and response latency, forming a disk status polling result. Finally, by combining the original physical disk alarm data and the real-time disk status information obtained from the polling, a comprehensive judgment is made to accurately distinguish between different types of failures, such as temporary network failures and permanent hardware failures.

[0056] This application avoids unnecessary disk isolation and data reconstruction operations caused by misjudging brief I / O jitter as hardware failure, reduces cluster resource consumption, minimizes the impact on normal business, improves the stability and availability of storage services, and enhances the cluster's ability to identify real hardware failures, achieving more accurate and intelligent fault response, and providing strong support for the efficient and reliable operation and maintenance of large-scale distributed storage clusters.

[0057] Based on the first embodiment described above, this application proposes a fault detection method according to a second embodiment.

[0058] In the second embodiment of this application, the same or similar content as in the above embodiments can be referred to the above description, and will not be repeated hereafter.

[0059] Compared with the first embodiment described above, this embodiment introduces a disk group partitioning mechanism and a cache disk priority diagnosis strategy in multi-disk concurrent failure scenarios to cope with more complex system-level failure modes, and further improve the accuracy of fault root cause location and the level of intelligent operation and maintenance response.

[0060] In this embodiment, the multiple physical disks include at least one cache disk and at least one data disk, and there is a one-to-N cache binding relationship between the cache disk and the data disk, where N is a positive integer; In this embodiment, the cache binding relationship means that one cache disk serves N data disks to accelerate read and write operations on these data disks. This is common in distributed storage systems that adopt a tiered storage architecture, such as S2D-based read-write hybrid storage pools or hyperconverged infrastructure systems that use dedicated SSDs as the caching layer. The cache disk caches data read from and written to the data disks simultaneously, and returns a success message when the data read / write operation on the cache disk is completed.

[0061] As those skilled in the art will recognize, a cache disk refers to a high-speed physical disk (typically an SSD, such as an NVMe SSD) in a distributed storage architecture specifically designed to improve I / O performance. Its main functions are to cache frequently accessed data, accelerate read operations, or temporarily store write logs to reduce access latency to the data disk. While the cache disk does not directly store a complete copy of user data, its operational status directly impacts the I / O performance and stability of the data disk it serves. The data disk, on the other hand, refers to the actual storage medium used for persistently storing user data. It can be an HDD or SSD and is responsible for carrying data copies or erasure coding fragments, serving as the primary carrier of data persistence.

[0062] In this embodiment, the one-to-N cache binding relationship between the cache disk and the data disks refers to the logical binding relationship between one cache disk and N data disks in the target server node, forming an independent I / O processing unit. For example, in S2D, one SSD cache disk can be configured to serve multiple HDD data disks on the same server node, forming a storage unit group (i.e., a disk group) of "1 cache disk + N HDDs". While this structure improves performance, it also introduces a new fault propagation path—a cache disk failure may cause I / O latency or failure in multiple data disks it maps to.

[0063] like Figure 2 As shown, Figure 2 This is a flowchart illustrating the second embodiment of the fault detection method of this application.

[0064] Following step S200 above, the fault detection method may further include steps S600 to S800: Step S600: If the number of faulty physical disks is greater than one, determine whether the faulty physical disks belong to the same disk group. The disk group is obtained by grouping multiple physical disks based on cache binding relationship. Physical disks in different disk groups do not have cache binding relationship, and the number of cache disks in all disk groups is one. In this embodiment, a disk group refers to a functional unit that combines physical disks with logical dependencies based on a one-to-N cache binding relationship between cache disks and data disks. Each disk group contains one and only one cache disk and N data disks bound to it. Different disk groups are logically independent of each other and have no shared components or mapping dependencies.

[0065] For example, in a target server node, if there are two SSD cache disks (SSD1 and SSD2), each serving three HDD data disks (HDD1~HDD3 are cached by SSD1, and HDD4~HDD6 are cached by SSD2), then it can be divided into two disk groups: Disk group 1: {SSD1, HDD1, HDD2, HDD3}; Disk group 2: {SSD2, HDD4, HDD5, HDD6}.

[0066] In this embodiment, when the number of faulty physical disks is determined to be greater than one, the system enters a multi-disk fault analysis path unique to this embodiment. At this point, the system first parses the affiliation relationships of each faulty physical disk to determine whether they all belong to the same disk group. This determination process can be achieved by querying the configuration metadata of the distributed storage system, such as reading the "virtual disk" or "storage tier" binding relationships in S2D, or by matching through the cache mapping table maintained by the driver layer.

[0067] By introducing the concept of disk groups, this embodiment can shift from the perspective of "individual physical disks" to that of "disk groups," identifying whether there is an inherent relationship between multiple failed physical disks. If multiple failed physical disks belong to the same disk group, it indicates that the problem may originate from the cache disks within that group, rather than the data disks themselves; conversely, if they are distributed across groups, it is more likely to involve systemic failures at the server node level (such as power supply, backplane, or controller). This mechanism effectively improves the ability to locate the root cause of failures in multi-failure scenarios.

[0068] Step S700: If the faulty physical disk belongs to the same disk group, determine the cache disk in the disk group to which the faulty physical disk belongs as the target faulty disk, and perform the step of polling the disk status of the target faulty disk. In this embodiment, when multiple failed physical disks belong to the same disk group, the system infers that the cache disk of that disk group may be the common root cause of multi-disk I / O anomalies. This is because the cache disk, as a critical hub in the I / O path, will cause delays, timeouts, or failures when performing read and write operations on all data disks it serves once its performance deteriorates or communication is interrupted. This will manifest as "multiple data disks reporting I / O failures simultaneously" in the alarm data.

[0069] Furthermore, when multiple disks fail and are concentrated in the same disk group, based on the actual statistically obtained average failure probabilities of the data disk and cache disk, it can be seen that the probability of a single cache disk failing within the disk group is much greater than the probability of multiple disks failing within the disk group. Therefore, in this case, it can be basically confirmed that a single cache disk within the disk group has failed.

[0070] Therefore, the system identifies the cache disk in the disk group as the "target fault disk" and proceeds to step S400 in the first embodiment to perform a disk status polling operation on it to obtain real-time disk status information such as its running status and health status. By combining the physical disk fault data and the disk status polling results, the fault type of the target fault disk is determined, providing a basis for subsequent fault handling of the distributed cluster. This determines whether to isolate the target fault disk or even rebuild the data, or to observe and wait or even notify the operation and maintenance personnel to conduct manual inspection.

[0071] For example, if HDD1, HDD2, and HDD3 all report I / O failure alarms, and all three belong to disk group 1 (cached by SSD1), the system will prioritize diagnosing the status of SSD1 rather than immediately performing isolation operations on the three HDDs.

[0072] This embodiment's cache disk priority diagnostic strategy embodies a diagnostic approach "from phenomenon to root cause." Traditional methods, when faced with multiple disk alarms, often isolate all faulty disks one by one, or directly determine that the server node is faulty and isolate it, which can easily lead to misjudgments and wasted resources. This embodiment, however, by identifying the functional dependencies within the disk group, focuses the diagnostic attention on potential "common cause components"—the cache disk—significantly improving troubleshooting efficiency. If the disk status polling results show that the cache disk is in a "disconnected" or "unhealthy" state, or if the physical disk alarm data contains a physical disk disconnection alarm or data buffer error alarm for the cache disk, then it can be identified as the root cause of the fault, and timely replacement or isolation can be implemented. If the cache disk is in a normal state, and the physical disk alarm data does not contain a physical disk disconnection alarm or data buffer error alarm for the cache disk, then further investigation can be conducted to determine whether the root cause is an individual data disk problem or a target server node problem, avoiding blind operations.

[0073] Step S800: If the faulty physical disk does not belong to the same disk group, determine that the target server node is faulty.

[0074] In this embodiment, when multiple faulty physical disks are distributed across different disk groups (i.e., not belonging to the same disk group), it indicates that the fault has crossed independent I / O processing units and is no longer limited to a specific cache disk-data disk mapping unit. This situation typically means the problem lies in a common component or infrastructure layer of the server node, such as: a power module malfunction causing unstable power supply to some disks, a motherboard or backplane failure affecting communication between multiple disks, a RAID / HBA controller failure, an operating system or driver malfunction, or a network switching module failure.

[0075] This embodiment classifies such cross-group concurrent failures as target server node failures, rather than single disk or local component problems. This judgment mechanism achieves a leap from "disk-level diagnosis" to "node-level diagnosis." By analyzing the spatial distribution characteristics of the failed disks (whether they are in the same group), the system can identify potential systemic risks. Once a node failure is confirmed, a higher-priority operational response (such as isolating the server node) can be triggered, avoiding the isolation of individual disks when the entire node is abnormal, and preventing accidental operations from exacerbating system instability.

[0076] The second embodiment of this application, based on the first embodiment "refined diagnosis of single disk faults", further constructs an intelligent diagnostic framework for multi-disk concurrent fault scenarios. Its core contribution is: the introduction of a disk group logical model and a cache disk priority diagnosis mechanism, realizing the technical evolution from "individual diagnosis" to "system correlation analysis".

[0077] Specifically, traditional distributed storage systems, when faced with multiple disk I / O failures, typically employ parallel processing or isolation strategies, making it difficult to identify the inherent connections between failures. This can easily lead to misjudging a chain reaction triggered by a cache disk anomaly as simultaneous failure of multiple data disks, resulting in unnecessary data reconstruction and resource consumption. This embodiment innovatively utilizes a "one-to-N" cache binding relationship between cache disks and data disks to construct logical units for disk groups. It then performs root cause inference based on the intra-group / extra-group distribution characteristics of the failed disks—if multiple failed disks belong to the same group, the shared cache disk is diagnosed first; if they are distributed across groups, the diagnosis escalates to node-level fault determination.

[0078] This mechanism not only significantly improves diagnostic accuracy in multi-disk failure scenarios but also effectively reduces the risk of misoperation and operational complexity. More importantly, it reveals the inherent laws of "functional dependency" and "fault propagation" in storage systems, enabling fault detection to shift from passive response to proactive reasoning, and providing a key technical path for building intelligent operation and maintenance systems with causal reasoning capabilities.

[0079] In summary, this embodiment improves the full-scenario coverage capability of the fault detection method by introducing three layers of logic: disk group partitioning, cache disk priority diagnosis, and node-level fault determination. This enables the method to not only accurately identify single-disk hardware / network faults but also effectively cope with complex system-level anomalies, thus comprehensively improving the reliability, stability, and intelligent operation and maintenance level of distributed storage clusters in high-concurrency, large-scale deployment environments.

[0080] Based on the first embodiment described above, this application proposes a fault detection method according to a third embodiment.

[0081] In the third embodiment of this application, the same or similar content as the above embodiments can be referred to the above description, and will not be repeated hereafter.

[0082] In this embodiment, the multiple physical disks include at least two cache disks. After step S200, the fault detection method may further include steps S910 to S920: Step S910: If the number of faulty physical disks is greater than one, determine the number of faulty cache disks among the faulty physical disks. It should be noted that, in this embodiment, the fault cache disk refers to the cache disk that is explicitly pointed to by at least one alarm in the physical disk alarm data.

[0083] Step S920: If the number of faulty cache disks is greater than one, determine that the target server node is faulty.

[0084] In this embodiment, when the number of faulty physical disks is determined to be greater than one, the system enters a multi-disk fault analysis path unique to this embodiment. At this point, the system first parses the disk type of each faulty physical disk and identifies the number of faulty cache disks. When the number of faulty cache disks is detected to be greater than one, the system determines that the target server node is faulty. This is because in distributed storage systems, cache disks, as key components for improving I / O performance, typically have high reliability designs. While individual cache disk failures may occasionally occur, the probability of multiple cache disks failing simultaneously is extremely low. In contrast, the probability of a server node itself failing is much higher. Therefore, when the number of faulty physical disks is detected to be greater than one, and it is further determined that the number of faulty cache disks is greater than one, it can be reasonably inferred that this phenomenon is not caused by individual disk aging or random damage, but is more likely due to a common fault source at the server node level.

[0085] Based on this statistical pattern and causal logic of failures, this embodiment innovatively uses "multiple cache disk failures" as the key criterion for determining the failure of target server nodes, thus breaking through the limitations of traditional methods that rely solely on the individual disk status or the number of alarms for isolated judgments. This failure judgment mechanism not only significantly improves the ability to perceive hidden system-level failures and avoids fragmented processing caused by misjudging node-level problems as independent failures of multiple disks, but also provides the cluster management layer with a higher-level health assessment dimension, supporting the early isolation of potentially unstable server nodes, preventing the spread of failures and performance degradation, and effectively ensuring the continuity of data services and the overall operational stability of the cluster.

[0086] In one feasible implementation, the multiple physical disks further include at least two data disks, and there is a one-to-N cache binding relationship between the cache disk and the data disks, where N is a positive integer. After step S910 above, the fault detection method may further include steps S930 to S950: Step S930: When the number of fault cache disks is equal to one, determine whether all faulty data disks in the faulty physical disk have a cache binding relationship with the faulty cache disk. Step S940: If all the faulty data disks have a cache binding relationship with the faulty cache disks, the faulty data disks are identified as the target faulty disks, and the disk status of the target faulty disks is polled. Step S950: If there is a cache binding relationship between the faulty data disk and the faulty cache disk, determine that the target server node is faulty.

[0087] This implementation determines whether each faulty data disk in the faulty physical disk has a cache binding relationship with the faulty cache disk. If such a relationship exists, it indicates that the I / O anomalies of these faulty data disks are likely caused by a chain reaction triggered by a failure of their shared faulty cache disk, rather than a failure of the data disks themselves. Therefore, the system identifies the faulty cache disk as the target faulty disk and proceeds to step S400 in the first embodiment. If no such relationship exists, it indicates that the fault has crossed independent I / O processing units and cannot be explained by a single cache disk failure. It is more likely that the target server node itself has failed. Therefore, the system determines it as a "target server node failure" and triggers a higher-level operation and maintenance response.

[0088] In scenarios with multiple disks concurrently and only one cache disk alarming, this implementation method achieves accurate tracing of the fault propagation path through cache binding relationship consistency analysis. This avoids misjudging the chain of I / O failures caused by cache disk anomalies as simultaneous damage to multiple data disks, thereby preventing unnecessary disk isolation and data reconstruction operations, significantly reducing resource consumption and business impact, and improving the scientific nature of fault diagnosis and operational efficiency.

[0089] In a second feasible implementation, the multiple physical disks further include at least two data disks, and there is a one-to-N cache binding relationship between the cache disk and the data disks, where N is a positive integer. After step S910 above, the fault detection method may further include steps S960 to S980: Step S960: When the number of faulty cache disks is zero, determine the number of target cache disks that have a cache binding relationship with the faulty data disk in the faulty physical disk; Step S970: If the number of target cache disks is equal to one, the target cache disk is identified as the target faulty disk, and the disk status of the target faulty disk is polled. Step S980: If the number of target cache disks is greater than one, determine that the target server node is faulty.

[0090] This implementation determines the number of target cache disks associated with all failed data disks, assuming the number of failed cache disks is zero. When the number of target cache disks is one, it infers that the concurrent failures of these failed data disks may originate from a hidden anomaly in their shared cache disk that did not report an alarm. In this case, although the cache disk did not trigger an explicit alarm, its abnormal behavior has caused read / write timeouts or failures in the data disks it serves. Therefore, the system identifies this target cache disk as the target failed disk and executes the status polling operation in step S400 to obtain its operating status, health status, and performance indicators. This allows the system to determine whether a failure truly exists and, if a failure does exist, to lock down the failure type as either a hardware failure or a network failure, in order to determine whether subsequent fault handling requires isolating the cache disk.

[0091] When the number of target cache disks is greater than one, it indicates that the fault has spread across multiple independent cache disk-data disk mapping units. Since the probability of multiple cache disks experiencing hidden faults simultaneously is extremely low, the system reasonably infers that the problem lies in the common infrastructure of the target server node, and therefore directly determines it as a target server node fault.

[0092] This implementation breaks through the traditional reliance on "explicit alarms," ​​enabling the identification of latent faults that, while not triggering cache disk alarms, actually affect data disk I / O. By tracing the cache dependencies of the faulty data disk in reverse, the system proactively detects potential common-cause components, enhancing its ability to perceive complex hardware and software anomalies. This not only improves the comprehensiveness of fault diagnosis but also provides key technical support for building intelligent storage systems with predictive maintenance capabilities.

[0093] To facilitate understanding of the fault detection method in the above embodiments of this application, a specific embodiment is provided: like Figure 3 As shown, this specific embodiment provides a fault detection and self-healing method based on a Microsoft S2D distributed storage cluster, aiming to solve problems such as insufficient scenario coverage, stringent triggering conditions, difficulty in identifying root causes of faults, and lack of risk control in existing self-healing mechanisms. The specific technical solution of this embodiment is as follows: (1) Monitoring and Alarms: Alarms serve as the trigger condition for self-healing. Based on data analysis, the 203 alarm in the Microsoft-Windows-StorageSpaces-Driver / Operational section of the Windows log was selected as the trigger condition, and a certain anti-repetition time was set to prevent a single fault from running repeatedly. The alarms are associated with fault detection scripts through the self-healing configuration platform. Once a relevant alarm occurs, the inspection script will be automatically sent to detect and locate the fault.

[0094] (2) Fault Detection: Fault detection is the core module of the entire fault self-healing solution. It is responsible for collecting data, analyzing faults, and summarizing conclusions for the S2D distributed storage cluster, ultimately obtaining an accurate analysis result. The fault detection module is also divided into different sub-modules, such as the single-disk fault detection sub-module and the NVMe+HDD binding fault detection sub-module, according to the fault type.

[0095] (3) Risk control: Risk control is responsible for assessing whether the isolation scheme in fault detection is safe. For example, there is a limit to the number of disks that can be isolated in a cluster, and no more than one physical node can be isolated in a cluster.

[0096] (4) Fault Handling: After the risk control monitoring process has passed, the handling measures are carried out. Commonly used handling measures generally include: isolating physical disks, isolating physical machines, disabling network cards, manual intervention, etc.

[0097] (5) Results display: Clearly notify users of fault detection results, risk results, and fault isolation results, supporting methods such as email and instant messaging tools.

[0098] The technical solution of this specific embodiment forms a complete distributed storage fault detection and self-healing system through the collaborative work of the above five core modules.

[0099] like Figure 4 As shown in the figure, this specific embodiment, through operational practice, determines that data collection includes the following aspects: Physical disks: Physical disk offline alarms (15 alarms in the system log), physical disk unresponsive alarms (48 alarms in the system log), physical I / O failure alarms (203 alarms in the storage space driver log), physical disk performance data (based on performance counters), physical disk real-time status (Get-PhysicalDisk command), physical disk error count (ClusterLog), and physical disk Smart data.

[0100] System disk: Physical disk reset alarm (550 alarm in storage port driver log), physical disk IO time exceeds a certain threshold (557 alarm in storage port driver log), physical disk performance data (based on performance counter Counter).

[0101] Network interface card (NIC): NIC disconnection alarm (alarm 13 in the system log), NIC status (Get-NetAdapter), and network Ping test results.

[0102] RAID card: physical disk data, RAID card hardware status, node storage view (i.e., the physical disks seen on each server node).

[0103] Physical machine: memory, CPU, node status (Get-ClusterNode command).

[0104] In this specific embodiment, data collection covers 10+ types of logs and performance data from physical disks, system disks, network cards, and RAID cards, ensuring comprehensive fault identification.

[0105] It should be noted that the above embodiments are only used to assist in understanding this application and do not constitute a limitation on the fault detection method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0106] In addition, please refer to Figure 5 , Figure 5 This is a schematic diagram of the module structure of the fault detection device in the embodiments of this application.

[0107] This application also provides a fault detection device applied to a distributed storage cluster, the distributed storage cluster including a target server node, the target server node including multiple physical disks, the fault detection device comprising: The data acquisition module 10 is used to acquire physical disk alarm data of multiple physical disks when a physical disk I / O failure alarm is detected reported by the target server node. The physical disk alarm data includes at least physical disk I / O failure alarms. The quantity determination module 20 is used to determine the number of faulty physical disks among multiple physical disks based on physical disk alarm data. The target determination module 30 is used to determine the faulty physical disk as the target faulty disk when the number of faulty physical disks is equal to one. The status polling module 40 is used to poll the disk status of the target faulty disk and obtain the disk status polling result of the target faulty disk. The fault analysis module 50 is used to determine the fault type of the target fault disk based on the physical disk alarm data and the disk status polling results of the target fault disk.

[0108] In one embodiment, the status polling module 40 is further configured to: The operating status of the target faulty disk is polled to obtain the polling results of the operating status of the target faulty disk; The health status of the target faulty disk is polled to obtain the health status polling results of the target faulty disk. Based on the polling results of the target failed disk's running status and health status, generate the disk status polling results for the target failed disk.

[0109] In one embodiment, the status polling module 40 is further configured to: If the polling result of the target faulty disk's running status is the first running status polling result, or the polling result of the target faulty disk's health status is the first health status polling result, then the polling result of the target faulty disk's disk status is determined to be the first polling result. If the polling result of the target faulty disk's running status is the second running status polling result, or the polling result of the target faulty disk's health status is the second health status polling result, then the polling result of the target faulty disk's disk status is determined to be the second polling result. Among them, the first operating status polling result indicates that the operating status is always out of contact during the polling period; the first health status polling result indicates that the health status is always unhealthy during the polling period; the second operating status polling result indicates that the operating status changes from out of contact to normal during the polling period; and the second health status polling result indicates that the health status changes from unhealthy to healthy during the polling period. Fault analysis module 50 is also used for: If the error code of the physical disk I / O failure alarm of the target faulty disk is the first error code, the physical disk alarm data includes the physical disk disconnection alarm of the target faulty disk, the physical disk alarm data includes the data buffer error alarm of the target faulty disk, or the disk status polling result is the first polling result, the fault type of the target faulty disk is determined to be a hardware fault. If the error code of the physical disk I / O failure alarm of the target faulty disk is the second error code, the physical disk alarm data of the target faulty disk does not include physical disk disconnection alarms and data buffer error alarms, and the disk status polling result is the second polling result, the fault type of the target faulty disk is determined to be a network fault. The first error code indicates a hardware error, and the second error code indicates a connection error.

[0110] In one embodiment, the plurality of physical disks include at least one cache disk and at least one data disk, and there is a one-to-N cache binding relationship between the cache disk and the data disk, where N is a positive integer. The fault detection device is further configured to: If the number of faulty physical disks is greater than one, determine whether the faulty physical disks belong to the same disk group. The disk group is obtained by grouping multiple physical disks based on cache binding relationship. Physical disks in different disk groups do not have cache binding relationship, and the number of cache disks in all disk groups is one. If the failed physical disk belongs to the same disk group, the cache disk in the disk group to which the failed physical disk belongs is identified as the target failed disk, and the disk status of the target failed disk is polled. If the faulty physical disk does not belong to the same disk group, the target server node is identified as faulty.

[0111] In one embodiment, the plurality of physical disks includes at least two cache disks, and the fault detection device is further configured to: If the number of failed physical disks is greater than one, determine the number of failed cache disks among the failed physical disks; If the number of faulty cache disks is greater than one, the target server node is determined to be faulty.

[0112] In one embodiment, the plurality of physical disks further includes at least two data disks, and there is a one-to-N cache binding relationship between the cache disk and the data disks, where N is a positive integer. The fault detection device is further used for: When the number of fault cache disks is equal to one, determine whether all faulty data disks in the faulty physical disk have a cache binding relationship with the fault cache disk; If all faulty data disks have a cache binding relationship with faulty cache disks, the faulty data disk is identified as the target faulty disk, and the disk status of the target faulty disk is polled. When there is a cache binding relationship between the faulty data disk and the faulty cache disk, the target server node is identified as faulty.

[0113] In one embodiment, the plurality of physical disks further includes at least two data disks, and there is a one-to-N cache binding relationship between the cache disk and the data disks, where N is a positive integer. The fault detection device is further used for: When the number of faulty cache disks is zero, determine the number of target cache disks that have cache binding relationships with the faulty data disks in the faulty physical disks; If the number of target cache disks is equal to one, the target cache disk is identified as the target failed disk, and the disk status of the target failed disk is polled. If the number of target cache disks is greater than one, the target server node is determined to be faulty.

[0114] The fault detection device provided in this application, employing the fault detection method in the above embodiments, can improve the accuracy of disk fault detection in distributed storage clusters. Compared with the prior art, the beneficial effects of the fault detection device provided in this application are the same as those of the fault detection method provided in the above embodiments, and other technical features in this fault detection device are the same as those disclosed in the fault detection method in the above embodiments, and will not be repeated here.

[0115] In addition, this application also provides a distributed storage cluster, which includes a target server node, which includes multiple physical disks, and the distributed storage cluster uses the fault detection method in the above embodiments for fault detection.

[0116] The distributed storage cluster provided in this application, employing the fault detection method described in the above embodiments, can improve the accuracy of disk fault detection in the distributed storage cluster. Compared with the prior art, the beneficial effects of the distributed storage cluster provided in this application are the same as those of the fault detection method provided in the above embodiments, and other technical features in this distributed storage cluster are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0117] In addition, this application also provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to perform the steps of the fault detection method in the above embodiments.

[0118] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory, optical fibers, portable compact disk read-only memory, optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (Radio Frequency), etc., or any suitable combination thereof.

[0119] The aforementioned computer-readable storage medium may be included in a distributed storage cluster; or it may exist independently without being assembled into a distributed storage cluster.

[0120] The aforementioned computer-readable storage medium carries one or more programs. When these programs are executed by the distributed storage cluster, the distributed storage cluster: upon detecting a physical disk I / O failure alarm reported by the target server node, collects physical disk alarm data from multiple physical disks, wherein the physical disk alarm data includes at least physical disk I / O failure alarms; determines the number of faulty physical disks among the multiple physical disks based on the physical disk alarm data; if the number of faulty physical disks is equal to one, identifies the faulty physical disk as the target faulty disk; polls the disk status of the target faulty disk to obtain the disk status polling result of the target faulty disk; and determines the fault type of the target faulty disk based on the physical disk alarm data and the disk status polling result.

[0121] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0122] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0123] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0124] The computer-readable storage medium provided in this application stores computer-readable program instructions (i.e., a computer program) for performing the steps of the above-described fault detection method, which can improve the accuracy of disk fault detection in a distributed storage cluster. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the fault detection method provided in the above embodiments, and will not be repeated here.

[0125] Furthermore, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the fault detection method as described in the above embodiments.

[0126] The computer program product provided in this application can improve the accuracy of disk fault detection in distributed storage clusters. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as the beneficial effects of the fault detection method provided in the above embodiments, and will not be repeated here.

[0127] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A fault detection method characterized by, The fault detection method is applied to a distributed storage cluster, the distributed storage cluster comprises a target server node, the target server node comprises a plurality of physical disks, and the fault detection method comprises the following steps: In the case that the physical disk I / O failure alarm reported by the target server node is detected, physical disk alarm data of the plurality of physical disks is collected, wherein the physical disk alarm data at least comprises the physical disk I / O failure alarm; According to the physical disk alarm data, the number of fault physical disks in the plurality of physical disks is determined; In the case that the number of fault physical disks is equal to one, the fault physical disk is determined as a target fault disk; The disk state of the target fault disk is polled to obtain a disk state polling result of the target fault disk; According to the physical disk alarm data and the disk state polling result, the fault type of the target fault disk is determined.

2. The fault detection method of claim 1, wherein, The step of polling the disk state of the target fault disk to obtain the disk state polling result of the target fault disk comprises the following steps: The running state of the target fault disk is polled to obtain a running state polling result of the target fault disk; The health state of the target fault disk is polled to obtain a health state polling result of the target fault disk; According to the running state polling result and the health state polling result of the target fault disk, the disk state polling result of the target fault disk is generated.

3. The fault detection method of claim 2, wherein, The step of generating the disk state polling result of the target fault disk according to the running state polling result and the health state polling result of the target fault disk comprises the following steps: In the case that the running state polling result of the target fault disk is a first running state polling result, or the health state polling result of the target fault disk is a first health state polling result, it is determined that the disk state polling result of the target fault disk is a first polling result; In the case that the running state polling result of the target fault disk is a second running state polling result, or the health state polling result of the target fault disk is a second health state polling result, it is determined that the disk state polling result of the target fault disk is a second polling result; Wherein, the first running state polling result indicates that the running state is always lost during polling, the first health state polling result indicates that the health state is always unhealthy during polling, the second running state polling result indicates that the running state is switched from lost to normal during polling, and the second health state polling result indicates that the health state is switched from unhealthy to healthy during polling; The step of determining the fault type of the target fault disk according to the physical disk alarm data and the disk state polling result comprises the following steps: determining that the target failed disk is of a hardware failure type in a case where an error code of a physical disk I / O failure alarm of the target failed disk is a first error code, the physical disk alarm data of the target failed disk comprises a physical disk offline alarm, the physical disk alarm data of the target failed disk comprises a data buffer error alarm, or the disk state polling result is a first polling result; determining that the target failed disk is of a network failure type in a case where an error code of a physical disk I / O failure alarm of the target failed disk is a second error code, the physical disk alarm data of the target failed disk does not comprise a physical disk offline alarm and a data buffer error alarm, and the disk state polling result is a second polling result; wherein the first error code indicates a hardware error, and the second error code indicates a connection error.

4. The failure detection method according to any one of claims 1 to 3, characterized by, The plurality of physical disks comprises at least one cache disk and at least one data disk, and a cache binding relationship of one-to-N exists between the cache disk and the data disk, where N is a positive integer, and after the step of determining the failed physical disk in the plurality of physical disks according to the physical disk alarm data, the method further comprises: in a case where the number of the failed physical disks is greater than one, determining whether the failed physical disks belong to a same disk group, wherein the disk group is obtained by grouping the plurality of physical disks based on the cache binding relationship, the physical disks in different disk groups do not have the cache binding relationship, and the number of cache disks in all disk groups is one; in a case where the failed physical disks belong to the same disk group, determining the cache disk in the disk group to which the failed physical disks belong as a target failed disk, and performing the step of polling the disk state of the target failed disk; in a case where the failed physical disks do not belong to the same disk group, determining that the target server node is failed.

5. The fault detection method according to any one of claims 1 to 3, wherein The plurality of physical disks comprises at least two cache disks, and after the step of determining the failed physical disk in the plurality of physical disks according to the physical disk alarm data, the method further comprises: in a case where the number of the failed physical disks is greater than one, determining the number of failed cache disks in the failed physical disks; in a case where the number of the failed cache disks is greater than one, determining that the target server node is failed.

6. The fault detection method of claim 5, wherein, The plurality of physical disks further comprises at least two data disks, and a cache binding relationship of one-to-N exists between the cache disk and the data disk, where N is a positive integer, and after the step of determining the number of failed cache disks in the failed physical disks, the method further comprises: in a case where the number of the failed cache disks is equal to one, determining whether the failed data disks in the failed physical disks all have the cache binding relationship with the failed cache disk; in a case where the failed data disks all have the cache binding relationship with the failed cache disk, determining the failed data disks as target failed disks, and performing the step of polling the disk state of the target failed disk; In a case where the failed data disk does not have the cache binding relationship with the failed cache disk, it is determined that the target server node is faulty.

7. The fault detection method of claim 5, wherein, The plurality of physical disks further include at least two data disks, and the cache disk and the data disk have a cache binding relationship in one-to-N, where N is a positive integer. After the step of determining the number of failed cache disks in the failed physical disks, the method further includes: In a case where the number of failed cache disks is equal to zero, it is determined that the number of target cache disks having the cache binding relationship with the failed data disk in the failed physical disks is equal to one; In a case where the number of target cache disks is equal to one, the target cache disk is determined as a target failed disk, and the step of polling the disk state of the target failed disk is performed; In a case where the number of target cache disks is greater than one, it is determined that the target server node is faulty.

8. A fault detection apparatus characterized by comprising: The fault detection device is applied to a distributed storage cluster, the distributed storage cluster includes a target server node, the target server node includes a plurality of physical disks, and the fault detection device includes: A data collection module is configured to collect physical disk alarm data of the plurality of physical disks in a case where a physical disk I / O failure alarm reported by the target server node is detected, where the physical disk alarm data at least includes the physical disk I / O failure alarm; A number determination module is configured to determine a number of failed physical disks in the plurality of physical disks according to the physical disk alarm data; A target determination module is configured to determine the failed physical disk as a target failed disk in a case where the number of failed physical disks is equal to one; A state polling module is configured to poll a disk state of the target failed disk to obtain a disk state polling result of the target failed disk; A fault analysis module is configured to determine a fault type of the target failed disk according to the physical disk alarm data and the disk state polling result of the target failed disk.

9. A distributed storage cluster, characterized in that, The distributed storage cluster includes a target server node, the target server node includes a plurality of physical disks, and the distributed storage cluster adopts the fault detection method of any one of claims 1 to 8 for fault detection.

10. A storage medium, characterized by The storage medium is a computer readable storage medium, and the computer readable storage medium stores a computer program. When the computer program is executed by a processor, the steps of the fault detection method of any one of claims 1 to 8 are implemented.

Images