Distributed storage system fault detection method, device and equipment

Abstract

Embodiments of the present application provide a distributed storage system fault detection method, device and equipment. The method is applied to any server node in a distributed storage system, the distributed storage system comprising two machine frames, and each machine frame is provided with two server nodes. The method comprises: detecting the communication state of the remaining three server nodes except itself through a data network, an on-board direct connection network and an Ethernet direct connection network; determining the fault type of the distributed storage system according to the communication state; when the fault type is a whole machine frame fault, scaling down the distributed coordination service cluster deployed on each server node from a four-node configuration to a two-node configuration comprising two server nodes in the current machine frame; re-selecting a master node for the distributed coordination service cluster according to the two server nodes in the current machine frame, and restoring the distributed coordination service. The method provided by the present application effectively solves the service unavailability problem caused by the whole machine frame fault.

Description

Technical Field

[0001] This application relates to the field of distributed storage system technology, and in particular to a method, apparatus and device for fault detection in a distributed storage system. Background Technology

[0002] With the expansion of storage system scale and the diversification of server hardware form factors, the 2U2N architecture (i.e., integrating two physically independent server nodes in a chassis with a height of 2 rack units) has been widely used in data centers.

[0003] In a 2U2N-based dual-chassis four-node storage system, each chassis contains two server nodes, and nodes within the same chassis typically share a power supply module. When a chassis experiences a power failure, both server nodes within that chassis will simultaneously fail. This causes the distributed coordination service (ZooKeeper, ZK) cluster deployed on it to fail to complete the master node election because the number of surviving nodes does not meet the majority arbitration condition, resulting in service interruption. Therefore, a highly available fault-healing mechanism that can accurately identify chassis failures and automatically restore service is urgently needed. Summary of the Invention

[0004] This application provides a method, apparatus, and device for fault detection in a distributed storage system. These methods utilize three communication channels—data network, onboard direct connection network, and Ethernet direct connection network—to collaboratively detect the connectivity status of each server node and accurately identify overall chassis faults. After confirming that both nodes in the same chassis are communicating normally and possess complete ZK service files, the ZK cluster is safely scaled down from four nodes to a two-node configuration within the same chassis, and a new master node is elected to restore service, effectively resolving service unavailability issues caused by overall chassis failures.

[0005] In a first aspect, embodiments of this application provide a fault detection method for a distributed storage system, applicable to any server node in the distributed storage system. The distributed storage system includes two chassis, each chassis containing two server nodes. The method includes:

[0006] The communication status of the other three server nodes (excluding itself) is detected through the data network, onboard direct connection network, and Ethernet direct connection network.

[0007] Determine the fault type of the distributed storage system based on the communication status;

[0008] When the fault type is a whole chassis fault, and the current server node communicates normally with another server node in its chassis, and the service integrity files of the distributed coordination service exist on both the current server node and the other server node in its chassis, the distributed coordination service cluster deployed on each server node will be scaled down from a four-node configuration to a two-node configuration containing two server nodes in the current chassis. The service integrity files include the configuration file, transaction log file and snapshot file of the distributed coordination service.

[0009] Based on the two server nodes currently in the chassis, a new master node is elected for the distributed coordination service cluster, and the distributed coordination service is restored.

[0010] By collaboratively detecting node connectivity through three heterogeneous communication channels—data network, onboard direct connection network, and Ethernet direct connection network—it can effectively distinguish between similar but fundamentally different fault scenarios such as chassis power failure and cross-chassis network partitioning, significantly improving the accuracy of fault diagnosis. Scaling down is only performed when a chassis failure is confirmed, and communication between the two nodes within the same chassis is normal and service files are complete. This avoids the formation of independent ZooKeeper clusters for each chassis during network partitioning, thus eliminating metadata conflicts and data inconsistencies caused by multiple masters.

[0011] In one possible implementation, determining the fault type of the distributed storage system based on the communication status includes:

[0012] Based on the communication status obtained through the data network, onboard direct connection network and Ethernet direct connection network, with the current server node as the source node, recursive route probing is performed on the other three server nodes other than itself to obtain the set of reachable paths from the source node to each target node. The target node is any node in the distributed storage system other than the source node.

[0013] When the set of reachable paths contains only one other server node within the same chassis, and two server nodes in another chassis cannot be reached through any relay path, the fault type is determined to be a chassis-wide fault.

[0014] When the set of reachable paths includes some or all of the server nodes of another chassis, the fault type is determined to be cross-chassis network partition.

[0015] Through the aforementioned recursive route detection mechanism, the true reachability of the network topology is analyzed in depth, going beyond surface-level communication interruptions. This allows for a precise distinction between two easily confused fault scenarios: overall physical chassis failure and logical network isolation. This provides the foundation for the subsequent secure execution of ZooKeeper cluster scaling-down operations, effectively preventing misjudgments of a chassis failure in network partitioning scenarios that could lead to two chassis forming independent coordinated service clusters (i.e., dual-master). Ultimately, this fundamentally ensures data consistency and service reliability in the distributed storage system.

[0016] In one possible implementation, performing recursive route probing includes:

[0017] Set a hop count threshold and a single path detection timeout for each probe;

[0018] If no response is received from the target node or relay node within the timeout period, the path is marked as unreachable.

[0019] If the recursion depth reaches the hop count threshold and reachability is still not determined, the probe is terminated and the fault type is determined based on the existing path set.

[0020] By introducing a dual constraint mechanism of timeout control and hop count limit, the three major problems that traditional recursive probing may face in a distributed storage environment—excessive probing time, excessive resource consumption, and logical deadlock—can be effectively solved. This can avoid delays in fault recovery due to waiting for unresponsive nodes, prevent excessive processor resources or memory consumption from explosive recursive expansion, and avoid getting stuck in infinite probing in network loops or misconfiguration scenarios.

[0021] In one possible implementation, after the distributed coordination service is restored, the method further includes:

[0022] Check whether the two server nodes in the other chassis meet the expansion conditions. The expansion conditions are that the two server nodes in the current chassis can establish a connection with the two server nodes in the other chassis through the data network, and the two server nodes in the other chassis have the service integrity file of the distributed coordination service.

[0023] When the expansion conditions are met, the distributed coordination service cluster will be restored from a two-node configuration to a four-node configuration, and the complete cluster state data will be reloaded to synchronize the transaction log.

[0024] This automatic scaling mechanism allows for seamless rebuilding of a four-node highly available ZooKeeper cluster without manual intervention after a complete chassis failure. It not only restores the original fault tolerance (tolerating single-node failures) but also ensures metadata consistency and service continuity, achieving a closed-loop self-healing process from security degradation to full recovery. This effectively solves the problems of high latency and error susceptibility caused by manual configuration recovery in traditional operations and maintenance, significantly improving the level of automated operations and maintenance and business resilience of distributed storage systems in complex failure scenarios.

[0025] In one possible implementation, when determining whether a service integrity file exists on the current server node and another server node within its chassis, the method further includes:

[0026] Retrieve different types of content from the service integrity file;

[0027] Based on a preset hash algorithm, the corresponding hash value is calculated for each type of service integrity file.

[0028] The calculated hash value is compared item by item with the baseline hash value pre-stored in the local or cluster metadata;

[0029] If the hash values ​​of all service integrity files are consistent with the corresponding baseline hash values, then the service integrity files are determined to be complete and have not been tampered with, and the corresponding server nodes are allowed to participate in scaling down or scaling up operations.

[0030] If the hash value of any service integrity file is inconsistent with the baseline hash value, the corresponding server node is marked as disabled and prohibited from joining the distributed coordination service cluster.

[0031] The above-described integrity verification process based on hash verification can effectively prevent cluster startup failures or data inconsistencies caused by file corruption, configuration drift, or security attacks, significantly improving the reliability and security of scaling down / expanding operations, and providing a solid data foundation for the high availability and self-healing capabilities of distributed storage systems.

[0032] In one possible implementation, before scaling down the distributed coordination service cluster deployed on each server node from a four-node configuration to a two-node configuration containing two server nodes within the current chassis, the method further includes:

[0033] Obtain the system load information of the two server nodes in the current chassis. The system load information includes CPU utilization, memory usage, and input / output throughput.

[0034] If the system load information of at least one server node does not exceed the preset threshold, a scaling-down operation is performed. The scaling-down operation is to reduce the distributed coordination service cluster deployed on each server node from a four-node configuration to a two-node configuration that includes two server nodes in the current chassis.

[0035] If the system load information of both server nodes exceeds the preset threshold, the scaling down operation will be paused and a system alarm message will be generated.

[0036] By introducing this load-aware mechanism, the problem of creating new bottlenecks during service recovery can be effectively avoided, achieving coordinated optimization of fault self-healing and resource assurance. Even if the preconditions such as overall system failure, normal communication, and file integrity are met, resource availability verification is still required before scaling down can be performed, thus ensuring both high availability and service quality stability in complex environments.

[0037] Secondly, embodiments of this application provide a fault detection device for a distributed storage system, applied to any server node in a distributed storage system. The distributed storage system includes two chassis, each chassis housing two server nodes. The device includes:

[0038] The detection module is used to detect the communication status of the other three server nodes besides itself via the data network, onboard direct connection network and Ethernet direct connection network.

[0039] The fault determination module is used to determine the fault type of the distributed storage system based on the communication status.

[0040] The scaling-down module is used to scale down the distributed coordination service cluster deployed on each server node from a four-node configuration to a two-node configuration containing two server nodes in the current chassis when the fault type is a whole chassis fault, and the current server node and another server node in its chassis have service integrity files for the distributed coordination service.

[0041] The service recovery module is used to re-elect a master node for the distributed coordination service cluster based on the two server nodes in the current chassis, and to restore the distributed coordination service.

[0042] In one possible implementation, the fault determination module is specifically used for:

[0043] Based on the communication status obtained through the data network, onboard direct connection network and Ethernet direct connection network, with the current server node as the source node, recursive route probing is performed on the other three server nodes other than itself to obtain the set of reachable paths from the source node to each target node. The target node is any node in the distributed storage system other than the source node.

[0044] When the set of reachable paths contains only one other server node within the same chassis, and two server nodes in another chassis cannot be reached through any relay path, the fault type is determined to be a chassis-wide fault.

[0045] When the set of reachable paths includes some or all of the server nodes of another chassis, the fault type is determined to be cross-chassis network partition.

[0046] In one possible implementation, the fault determination module is further configured to:

[0047] Set a hop count threshold and a single path detection timeout for each probe;

[0048] If no response is received from the target node or relay node within the timeout period, the path is marked as unreachable.

[0049] If the recursion depth reaches the hop count threshold and reachability is still not determined, the probe is terminated and the fault type is determined based on the existing path set.

[0050] In one possible implementation, the service recovery module is also used for:

[0051] Check whether the two server nodes in the other chassis meet the expansion conditions. The expansion conditions are that the two server nodes in the current chassis can establish a connection with the two server nodes in the other chassis through the data network, and the two server nodes in the other chassis have the service integrity file of the distributed coordination service.

[0052] When the expansion conditions are met, the distributed coordination service cluster will be restored from a two-node configuration to a four-node configuration, and the complete cluster state data will be reloaded to synchronize the transaction log.

[0053] In one possible implementation, the shrinkage module is specifically used for:

[0054] Retrieve different types of content from the service integrity file;

[0055] Based on a preset hash algorithm, the corresponding hash value is calculated for each type of service integrity file.

[0056] The calculated hash value is compared item by item with the baseline hash value pre-stored in the local or cluster metadata;

[0057] If the hash values ​​of all service integrity files are consistent with the corresponding baseline hash values, then the service integrity files are determined to be complete and have not been tampered with, and the corresponding server nodes are allowed to participate in scaling down or scaling up operations.

[0058] If the hash value of any service integrity file is inconsistent with the baseline hash value, the corresponding server node is marked as disabled and prohibited from joining the distributed coordination service cluster.

[0059] In one possible implementation, the shrinkage module is also used for:

[0060] Obtain the system load information of the two server nodes in the current chassis. The system load information includes CPU utilization, memory usage, and input / output throughput.

[0061] If the system load information of at least one server node does not exceed the preset threshold, a scaling-down operation is performed. The scaling-down operation is to reduce the distributed coordination service cluster deployed on each server node from a four-node configuration to a two-node configuration that includes two server nodes in the current chassis.

[0062] If the system load information of both server nodes exceeds the preset threshold, the scaling down operation will be paused and a system alarm message will be generated.

[0063] Thirdly, embodiments of this application provide a distributed storage system fault detection device, comprising:

[0064] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method shown in any of the first aspects.

[0065] Fourthly, embodiments of this application provide a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to perform the method shown in any of the first aspects.

[0066] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the method shown in any of the first aspects.

[0067] The distributed storage system fault detection method, apparatus, and device provided in this application utilize three heterogeneous communication channels—data network, onboard direct connection network, and Ethernet direct connection network—to collaboratively detect node connectivity. This effectively distinguishes between similar but fundamentally different fault scenarios, such as whole-chassis power failure and cross-chassis network partitioning, significantly improving the accuracy of fault diagnosis. Scaling down is only performed when a whole-chassis failure is confirmed, and communication between the two nodes within the same chassis is normal and service files are complete. This avoids the formation of independent ZooKeeper clusters in each chassis during network partitioning, thus eliminating metadata conflicts and data inconsistencies caused by multiple masters. Under strict preconditions, the four-node ZooKeeper cluster is securely scaled down to a dual-node configuration within the same chassis, and a new master node is elected based on this dual-node configuration. This ensures that the distributed coordination service can quickly recover after a whole-chassis failure, guaranteeing continuous availability of upper-layer storage services. This enables distributed storage systems that would otherwise be unavailable due to whole-chassis power failure to have rapid self-recovery capabilities, solving the ZooKeeper service interruption problem caused by whole-chassis failure in a dual-chassis four-node architecture, and enhancing the reliability of the distributed storage system. Attached Figure Description

[0068] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0069] Figure 1 A schematic diagram illustrating an application scenario provided in an embodiment of this application;

[0070] Figure 2 A flowchart illustrating a fault detection method for a distributed storage system provided in an embodiment of this application;

[0071] Figure 3 A schematic diagram of a two-chassis, four-node network architecture provided in this application embodiment;

[0072] Figure 4 A schematic diagram of a frame failure provided in an embodiment of this application;

[0073] Figure 5 This application provides a flowchart illustrating a method for determining the fault type of a distributed storage system.

[0074] Figure 6 A schematic diagram of a fault scenario for cross-frame network partitioning provided in an embodiment of this application;

[0075] Figure 7 A flowchart illustrating another method for fault detection in a distributed storage system provided in this application embodiment;

[0076] Figure 8This is a schematic diagram of the structure of a distributed storage system fault detection device provided in an embodiment of this application;

[0077] Figure 9 This is a schematic diagram of the structure of a fault detection device for a distributed storage system provided in an embodiment of this application.

[0078] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0079] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0080] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0081] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with relevant laws, regulations and standards, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0082] To facilitate understanding, the following will be combined with... Figure 1 The application scenarios applicable to the embodiments of this application will be described.

[0083] Figure 1 This is a schematic diagram illustrating an application scenario provided by an embodiment of this application. Please refer to [link / reference]. Figure 1This application scenario includes client 102 and server cluster 101. Server cluster 101 consists of two physical chassis: chassis 10 and chassis 20. Each chassis deploys two server nodes: chassis 10 contains server node 1 and server node 2, and chassis 20 contains server node 3 and server node 4, forming a four-node distributed storage architecture.

[0084] Client 102 can initiate data storage requests to server cluster 101 via the data network. The data storage requests can be forwarded to the ZK coordination service for metadata management and consistency control via the storage service component. The ZK cluster is initially deployed on four server nodes, using a four-node configuration to ensure high availability.

[0085] When a chassis failure occurs (e.g., chassis 20 experiences a power outage causing server nodes 3 and 4 to lose power simultaneously), server node 1 or server node 2 will automatically trigger the fault detection method of this embodiment. The communication status of the remaining three server nodes is detected through the data network, onboard direct connection network (for communication between nodes within the same chassis), and Ethernet direct connection network (HA network, used for cross-chassis detection). Based on the multi-channel detection results, if the current fault type is determined to be a chassis failure, and it is confirmed that communication between server node 1 and server node 2 is normal, and both have complete ZK service integrity files (including configuration files, transaction logs, and snapshot files), the original four-node ZK cluster is safely scaled down to a two-node configuration containing only server node 1 and server node 2, and a new master node is elected to restore ZK service.

[0086] After scaling down, data storage requests from client 102 can still be processed normally through server node 1 or server node 2, ensuring business continuity. After standby frame 20 is powered back on and completes node self-test, the scaling up process can be further executed, re-adding server node 3 and server node 4 to the ZK cluster to restore the original four-node high-redundancy architecture.

[0087] This embodiment achieves automatic recovery of ZK services without relying on external intervention through multi-network collaborative detection and intelligent scaling-down mechanism, effectively solving the service interruption problem caused by whole-frame failure in 2U2N dual-machine framework, and significantly improving the high availability capability of distributed storage system.

[0088] With the continuous expansion of storage system scale and the constant evolution of server hardware form factors, high-density, high-energy-efficiency server architectures are becoming increasingly prevalent in modern data centers. Among them, the 2U2N model, as a typical example, is widely used in distributed storage, cloud computing, and big data scenarios due to its ability to integrate two physically independent computing nodes within a limited space. In a typical dual-chassis, four-node storage deployment architecture, the distributed storage system comprises two physical chassis, each deploying two 2U2N server nodes, for a total of four server nodes forming the storage cluster. It is worth noting that the two server nodes within the same chassis typically share a power module and some chassis infrastructure. While this design improves space utilization and power supply efficiency, it also introduces a single point of failure. If the entire chassis loses power or experiences a hardware-level failure, both server nodes within that chassis will simultaneously fail.

[0089] In this architecture, the distributed coordination service, as a core component ensuring cluster metadata consistency, service discovery, and master-slave election, is crucial for high availability. Related technologies typically deploy ZooKeeper (ZK) service instances on four server nodes, forming a four-node ZK cluster, and rely on a quorum-based consensus mechanism to achieve master node election and transaction log synchronization. For a four-node cluster, according to this quorum-based consensus mechanism, the majority threshold is 3, meaning at least three server nodes must be online to complete master node election and provide services.

[0090] However, when a complete chassis failure causes two ZK nodes to go offline simultaneously, the number of remaining surviving nodes is only 2, which is insufficient to meet the majority requirement. The ZK cluster will then become unavailable, which will prevent the upper-layer storage services from performing metadata operations, volume management, or client access, resulting in a global business interruption.

[0091] This application proposes a fault detection method for a distributed storage system. This method can be executed by any server node, utilizing three heterogeneous communication channels: a data network for regular service communication, an onboard direct connection network for heartbeat and status synchronization between nodes within the same chassis, and an Ethernet direct connection network (HA network) dedicated to cross-chassis high availability detection. These channels collaboratively detect the real-time communication status of the other three server nodes (excluding the node itself). Based on the multi-channel detection results, different fault scenarios, such as chassis-wide failure and cross-chassis network partitioning, are accurately distinguished. If the fault type is confirmed to be chassis-wide failure, and communication between the two server nodes within the current chassis is normal, and both nodes contain service integrity files (including configuration files, transaction logs, and snapshot files), a secure scaling-down process is triggered. The original four-node ZK cluster is dynamically adjusted to a two-node configuration containing only the two server nodes within the current chassis, and a new master node is elected based on this two-node configuration to quickly restore distributed coordination services. This ensures the integrity of transaction data and the continuity of services, effectively solving the ZK cluster unavailability problem caused by chassis-wide failure, and significantly improving the high availability and self-healing level of the 2U2N dual-chassis storage system.

[0092] The method described in this application will now be illustrated through specific embodiments. It should be noted that the following embodiments may exist independently or in combination with each other; identical or similar content will not be repeated in different embodiments.

[0093] Figure 2 This is a flowchart illustrating a fault detection method for a distributed storage system provided in an embodiment of this application. Please refer to... Figure 2 The method may include:

[0094] S201. Detect the communication status of the other three server nodes besides itself through the data network, onboard direct connection network, and Ethernet direct connection network.

[0095] The execution entity in this application embodiment can be a fault detection device or a fault detection apparatus installed within a fault detection device. The fault detection apparatus can be implemented through software or a combination of software and hardware. The fault detection device can be a server.

[0096] A fault detection method for a distributed storage system can be applied to any server node in the distributed storage system, which includes two chassis, each containing two server nodes. Specifically, the distributed storage system adopts a dual-chassis, four-node architecture, comprising two physical chassis (e.g., chassis 10 and chassis 20), with two independent server nodes deployed in each chassis, forming a total of four server nodes.

[0097] The current server node can detect the communication status of the other three server nodes besides itself through three types of heterogeneous communication networks: data network, onboard direct connection network, and Ethernet direct connection network.

[0098] Among them, the data network refers to the backbone network (usually gigabit or 10-gigabit Ethernet) used for routine business communication, carrying client requests, storing data transmission, and message interaction between distributed coordination services.

[0099] Onboard direct-connect network refers to a dedicated network in which two server nodes within the same chassis are directly interconnected through the motherboard's integrated network card. It features low latency and high reliability and is mainly used for heartbeat detection, local status synchronization, and auxiliary leader election between nodes in the same chassis.

[0100] Ethernet Direct Connect Network (also known as HA Network) refers to a highly available dedicated network that connects server nodes across chassis point-to-point via independent physical network cables. It is specifically designed to assist in judging cross-chassis connectivity when data network is abnormal, so as to avoid misjudgment due to network partitioning.

[0101] Figure 3 This is a schematic diagram of a two-chassis, four-node network architecture provided in an embodiment of this application. Figure 3 As shown, the distributed storage system involved in this embodiment includes two physical chassis, chassis 10 and chassis 20. Two server nodes are deployed in each chassis, forming a four-node architecture. Server node 1 and server node 2 are located in chassis 10, and server node 3 and server node 4 are located in chassis 20. This distributed storage system can achieve highly reliable interconnection between nodes through three independent communication networks: a data network, an onboard direct-connect network, and an Ethernet direct-connect network, to support fault detection, state synchronization, and service self-healing.

[0102] The data network consists of router 50 connecting the network interface cards (NICs) of each server node, such as NIC 110 of server node 1, NIC 210 of server node 2, NIC 310 of server node 3, and NIC 410 of server node 4, forming the backbone communication channel. This network can be used to carry client requests, store data transmission, and facilitate routine message exchange between distributed coordination services. It provides primary communication capabilities for the ZK cluster during normal operation of the distributed coordination service and serves as a basic probing path in failure scenarios.

[0103] An onboard direct-connect network refers to a dedicated, low-latency, highly reliable point-to-point link between two server nodes within each chassis, connected directly via integrated network interface cards (NICs) on the motherboard. For example, server node 1 and server node 2 are directly connected via NICs 110 and 210, forming an onboard direct-connect network. This network bypasses external switching equipment, possesses high stability, and is primarily used for heartbeat detection between nodes within the same chassis, local state synchronization, and assisting in leader election. It is a key indicator for determining whether the entire chassis has failed.

[0104] A Direct Ethernet Attached (HA) network refers to a dedicated, high-availability network where server nodes across different chassis are directly connected point-to-point via independent Ethernet cables. For example, server node 1's port 11 is directly connected to server node 3's port 31, server node 1's port 12 is directly connected to server node 4's port 41, server node 2's port 21 is directly connected to server node 3's port 32, and server node 2's port 22 is directly connected to server node 4's port 42, thus forming a Direct Ethernet Attached network. This network is independent of the data network and is dedicated to cross-chassis communication status detection. Even when the data network is interrupted, it can still determine whether a server node in another chassis is reachable, effectively distinguishing between chassis-wide failures and cross-chassis network partitions.

[0105] Figure 4 This is a schematic diagram of a frame failure provided in an embodiment of this application. Figure 4 As shown, when the chassis 20 loses power and the entire chassis fails, server node 1 or server node 2 in the chassis 10 can detect through the data network that server node 3 and server node 4 are unreachable. It can confirm that communication with the nodes in the same chassis is normal through the onboard direct connection network, and can verify through the Ethernet direct connection network that server node 3 and server node 4 cannot be reached through the relay path.

[0106] By combining the detection results from three types of networks, it can be accurately determined that chassis 20 is a chassis-wide fault, thereby triggering a safe scaling-down process and avoiding false alarms that could lead to dual-master issues. Thus, through a multi-channel redundant communication design, accurate identification of complex network fault scenarios is achieved, providing a solid technical foundation for automatic fault recovery in distributed coordination services.

[0107] S202. Determine the fault type of the distributed storage system based on the communication status.

[0108] The current server can determine the fault type of the distributed storage system based on the communication status obtained from the above multi-channel detection. Typical fault types include whole chassis failure (i.e., a chassis fails completely, and both nodes within it are unreachable) and cross-chassis network partition (i.e., communication within two chassis is normal, but the data network between the chassis is partially or completely interrupted).

[0109] S203. When the fault type is a whole chassis fault, and the current server node communicates normally with another server node in its chassis, and the service integrity file of the distributed coordination service exists on both the current server node and another server node in its chassis, the distributed coordination service cluster deployed on each server node will be scaled down from a four-node configuration to a two-node configuration including two server nodes in the current chassis.

[0110] In this embodiment of the application, the service integrity file includes the configuration file, transaction log file, and snapshot file of the distributed coordination service.

[0111] When the fault type is determined to be a chassis failure, and the following three conditions are met simultaneously, the safe scaling-down process is triggered. The three conditions are: First, the current server node and another server node in its chassis can communicate normally via at least the onboard direct connection network or the data network; Second, the service integrity files of the distributed coordination service exist on both the current server node and its co-chair nodes; Third, the distributed coordination service cluster is currently unavailable (e.g., unable to complete master node election).

[0112] Among them, the distributed coordination service is an open-source coordination framework used to maintain configuration information, naming services, distributed synchronization and group services, and plays a core role in ensuring metadata consistency in distributed storage systems.

[0113] Service integrity files include three key types of files: configuration files, transaction log files, and snapshot files.

[0114] The configuration file defines the list of Zookeeper cluster members, ports, and runtime parameters. The transaction log file records all committed transaction operations, ensuring data persistence. Snapshot files are periodically generated memory images of the data tree, used to accelerate the recovery process.

[0115] Only when all three types of files exist and are complete are the corresponding server nodes considered to have the ability to independently restore the ZK service.

[0116] After meeting the above conditions, the distributed coordination service cluster deployed on each server node can be dynamically scaled down from the original four-node configuration to a two-node configuration containing only the two server nodes within the current chassis. This scaling-down operation can be achieved by updating the local ZK configuration file and restarting the service, ensuring that the distributed coordination service cluster contains only two live and trusted nodes.

[0117] S204. Based on the two server nodes in the current chassis, re-elect the master node for the distributed coordination service cluster and restore the distributed coordination service.

[0118] Based on the scaled-down two-node cluster, the master node re-election process can be executed, and a new master node (Leader) can be successfully elected, restoring the full distributed coordination service function.

[0119] Through the above mechanism, even in severe failure scenarios such as power failure of the entire frame, the distributed storage system can still automatically and safely maintain the availability of ZK services, avoid storage service interruption, and significantly improve the high availability and self-healing capability of the 2U2N dual-machine frame architecture.

[0120] The distributed storage system fault detection method provided in this application utilizes three heterogeneous communication channels—data network, onboard direct connection network, and Ethernet direct connection network—to collaboratively detect node connectivity. This effectively distinguishes between similar but fundamentally different fault scenarios, such as whole-chassis power failure and cross-chassis network partitioning, significantly improving the accuracy of fault diagnosis. Scaling down is only performed when a whole-chassis failure is confirmed, and communication between the two nodes within the same chassis is normal and service files are complete. This avoids the formation of independent ZK clusters (i.e., "dual-master") in two chassis during network partitioning, thus eliminating metadata conflicts and data inconsistencies caused by multiple masters. Under strict preconditions, a four-node ZooKeeper cluster is securely scaled down to a dual-node configuration within the same chassis, and a new master node is elected based on this dual-node configuration. This ensures that the distributed coordination service can quickly recover after a whole-chassis failure, guaranteeing continuous availability of upper-layer storage services. Both nodes involved in the scaling down are required to have complete service integrity files (including configuration files, transaction logs, and snapshots), ensuring that the recovered ZK cluster has the latest and consistent transaction state, preventing service startup failures or data loss due to missing or truncated logs. To address the single point of failure issue caused by the hardware characteristics of 2U2N models sharing a power supply within the same chassis, an automated, closed-loop self-healing mechanism is provided. This enables the distributed storage system, which would otherwise be unusable due to a power outage of the entire chassis, to have rapid self-recovery capabilities. It solves the problem of ZooKeeper service interruption caused by a chassis failure in the 2U2N dual-chassis four-node architecture, thereby enhancing the reliability of the distributed storage system.

[0121] Based on the above embodiments, the following, in conjunction with Figure 5 This application provides a detailed explanation of the process for determining the fault type of a distributed storage system.

[0122] Figure 5 This is a flowchart illustrating a method for determining the fault type of a distributed storage system, as provided in an embodiment of this application. Please refer to [link / reference]. Figure 5 The method includes:

[0123] S501. Based on the communication status obtained through the data network, onboard direct connection network and Ethernet direct connection network, take the current server node as the source node and perform recursive route probing on the other three server nodes excluding itself to obtain the set of reachable paths from the source node to each target node.

[0124] In this embodiment of the application, the target node is any node in the distributed storage system other than the source node.

[0125] Specifically, using the current server node as the source node, and based on the underlying connectivity information obtained through three communication channels—data network, onboard direct connection network, and Ethernet direct connection network—recursive route probing can be performed on the other three server nodes (i.e., target nodes) besides itself. This probing process not only checks whether the source node can communicate directly with the target node, but also further attempts to initiate indirect probing to other unreachable nodes through confirmed reachable intermediate nodes as relays, thereby constructing a complete set of reachable paths.

[0126] Specifically, recursive route probing can be performed as follows: The source node first sends probe requests to three target nodes. For target nodes that cannot respond directly, the source node can traverse its currently known reachable neighbor nodes (such as nodes within the same frame or directly connected nodes across frames), forward the probe requests through these neighbor nodes, and recursively execute the same logic.

[0127] During the detection process, each successfully established communication path is recorded, ultimately forming a set of reachable paths containing all reachable target nodes and their corresponding paths. Each path consists of a series of consecutive node hops, reflecting the actual communication link from the source node to the target node.

[0128] S502. When the reachable path set contains only one other server node within the same chassis, and it is impossible to reach two server nodes in another chassis through any relay path, the fault type is determined to be a chassis-wide fault.

[0129] The fault type can be determined based on the contents of the reachable path set. If the reachable path set only contains another server node within the same chassis (e.g., the source node is located in chassis 10 and can only reach another server node in chassis 10), and there is no path that can reach the two server nodes in another chassis via relay (i.e., both server nodes in chassis 20 are unreachable, whether directly or indirectly), then the current fault type is determined to be a chassis-wide fault. Figure 4 As shown. This type of chassis failure is usually caused by a power outage, hardware failure, or power interruption in another chassis, and is considered a single point of large-scale failure.

[0130] S503. When the set of reachable paths contains some or all of the server nodes of another chassis, determine the fault type as cross-chassis network partition.

[0131] If the reachable path set includes some or all of the server nodes of another chassis (for example, although the source node cannot reach chassis 20 through the data network, it can reach at least one of the server nodes directly or via a relay through the Ethernet direct connection network), then the current fault type is determined to be a cross-chassis network partition. This cross-chassis network partition is usually caused by a failure of the data network switching equipment, a loose network cable, or a configuration error. Communication within the two chassis is normal, but the backbone network between the chassis is interrupted.

[0132] Figure 6 This is a schematic diagram illustrating a fault scenario for cross-frame network partitioning provided in an embodiment of this application. Figure 6 As shown, both the data network and the Ethernet direct connection network of server node 4 are abnormal, which is manifested as the interruption of the data network connection and the Ethernet direct connection network connection between server node 1 and server node 4 (marked as ×), and the interruption of the data network connection and the Ethernet direct connection network connection between server node 2 and server node 4 (marked as ×).

[0133] However, the onboard direct network of server node 4 continues to operate normally.

[0134] At this point, if server node 1 performs the fault detection process as the source node, its detection results are as follows: server node 4 cannot be reached through the data network. However, it can communicate directly with server node 3 through the Ethernet direct connection network or the data network, and then server node 3 can indirectly detect server node 4 through the onboard direct connection network. Simultaneously, server node 1 can communicate with server node 2 through the onboard direct connection network.

[0135] Therefore, starting from server node 1, the set of reachable paths includes the first path and the second path. The first path is from server node 1 to server node 3 and then to server node 4, and the second path is from server node 1 to server node 2.

[0136] Since the reachable path set includes server nodes 3 and 4 of another chassis (chassis 20), cross-chassis communication paths still exist despite the data network interruption. Therefore, the current fault type can be determined to be cross-chassis network partitioning rather than chassis-wide fault.

[0137] In this scenario, it's unnecessary to trigger a ZooKeeper cluster downsizing operation, thus avoiding a dual-master issue caused by misjudgment leading to each chassis electing its own master node. Simultaneously, the Ethernet direct connection network can be used to maintain connectivity for cross-chassis coordination services, automatically resuming normal operation once the data network is restored.

[0138] Through the aforementioned recursive route detection mechanism, the true reachability of the network topology is analyzed in depth, going beyond surface-level communication interruptions. This allows for a precise distinction between two easily confused fault scenarios: overall physical chassis failure and logical network isolation. This provides the foundation for the subsequent secure execution of ZooKeeper cluster scaling-down operations, effectively preventing misjudgments of a chassis failure in network partitioning scenarios that could lead to two chassis forming independent coordinated service clusters (i.e., dual-master). Ultimately, this fundamentally ensures data consistency and service reliability in the distributed storage system.

[0139] In one possible implementation, performing recursive route probing includes: setting a hop count threshold and a single path probing timeout for each probe; if no response is received from the target node or relay node within the timeout period, the path is marked as unreachable; if the recursion depth reaches the hop count threshold and reachability is still not determined, the probe is terminated and a fault type determination is made based on the existing path set.

[0140] Specifically, to ensure that the fault detection process is efficient and reliable and does not get stuck in an infinite loop or be blocked for a long time due to abnormal network conditions, two key parameters can be preset for each recursive route probing task: a hop count threshold and a single path probing timeout.

[0141] The hop count threshold limits the maximum number of relay hops in a probe path, for example, set to 2 or 3 hops. The single path probe timeout limits the maximum time from sending a probe request to waiting for a response, for example, 500 milliseconds or 1 second.

[0142] Subsequently, during the recursive probing process, the following judgment logic can be executed for each communication path to be verified: if the source node neither receives a direct response from the target node nor obtains indirect reachability feedback through any relay node within the timeout period of a single path probing, the path is determined to be unreachable and can be excluded from the candidate path set. Simultaneously, the depth of the current recursive call (i.e., the number of relay hops used) can be monitored in real time. If, during the probing of a specific path, the recursion depth reaches a preset hop count threshold (e.g., reachability has not been confirmed after attempting forwarding through two intermediate nodes), further expansion of that path is immediately terminated to avoid the probing process getting out of control due to network loops, unresponsive nodes, or excessive topology complexity.

[0143] Once all possible probe paths have been completed (or terminated early due to timeout / hop count limitations), all successfully established communication links can be aggregated to form a final set of reachable paths. Even if this set of reachable paths does not cover all target nodes, fault type can still be determined based on the existing known reachable path information. For example, if only nodes within the same frame can be reached, it is determined to be a frame-wide fault; if nodes across frames can be reached via direct Ethernet connection or relays, it is determined to be a cross-frame network partition.

[0144] By introducing a dual constraint mechanism of timeout control and hop count limit, this embodiment can effectively solve the three major problems that traditional recursive probing may face in a distributed storage environment: excessive probing time, excessive resource consumption, and logical deadlock. This can avoid delaying fault recovery due to waiting for unresponsive nodes, prevent excessive processor resources or memory consumption due to explosive recursive expansion, and avoid getting stuck in infinite probing in network loop or misconfiguration scenarios.

[0145] In one possible implementation, after restoring the distributed coordination service, the method further includes: detecting whether the two server nodes of the other chassis meet the expansion conditions, wherein the expansion conditions are that the two server nodes in the current chassis can establish a connection with the two server nodes of the other chassis through the data network, and the two server nodes of the other chassis both have the service integrity file of the distributed coordination service; when the expansion conditions are met, the distributed coordination service cluster is restored from a two-node configuration to a four-node configuration, and the complete cluster state data is reloaded to synchronize the transaction log.

[0146] Specifically, after successfully restoring the distributed coordination service, automatic scaling and recovery steps can be further executed to rebuild the original redundant cluster architecture once the failed chassis returns to normal. First, another chassis (i.e., the previously failed chassis) can be continuously or periodically monitored to determine whether the previously failed chassis meets the two necessary scaling conditions: network connectivity and service integrity.

[0147] Network connectivity refers to the ability of two server nodes within the current chassis to establish bidirectional communication connections with two server nodes in another chassis via the data network (i.e., the main business communication network). This means that not only can the local chassis actively access the peer node, but the peer node can also respond to requests, ensuring that subsequent heartbeats, log synchronization, and transaction broadcasting among cluster members can proceed normally.

[0148] Service integrity refers to the existence of complete service integrity files for the distributed coordination service on both server nodes in another chassis. Service integrity files include three key components: configuration files, transaction log files, and snapshot files.

[0149] Service integrity files are the foundation for server nodes to independently participate in service coordination. The absence of any file may lead to service startup failure or inconsistent status.

[0150] When both network connectivity and service integrity requirements for scaling are met, it can be determined that the other chassis has fully recovered and is capable of rejoining the cluster. At this point, any server node in the current chassis (usually the master node) can trigger the scaling process. This involves dynamically updating the distributed coordination service configuration files of the local and peer nodes, restoring the original two-node cluster configuration to a four-node configuration, and reintegrating the two server nodes from the other chassis; restarting or hot-loading the distributed coordination service process, enabling all four nodes to re-establish sessions and participate in cluster consensus.

[0151] After the new cluster is formed, the current server node can coordinate the cluster state recovery operation, collect the transaction identifiers (ZooKeeper TransactionID, zxid) at the end of the local transaction logs reported by each node in the distributed coordination service cluster, and identify the node with the highest zxid as the synchronization benchmark. Here, zxid is a unique, incrementing identifier used in the ZooKeeper system for globally sorting transactions; a larger value indicates a newer transaction.

[0152] Subsequently, using the server node corresponding to the highest zxid as the data source, other server nodes are driven to pull the missing transaction log entries from it and replay the missing transactions locally until the data state of all server nodes is consistent with the data source, thereby ensuring that all replicas of the entire distributed coordination service cluster eventually reach a consistent state.

[0153] This automatic scaling mechanism allows for seamless rebuilding of a four-node highly available ZooKeeper cluster without manual intervention after a complete chassis failure. It not only restores the original fault tolerance (tolerating single-node failures) but also ensures metadata consistency and service continuity, achieving a closed-loop self-healing process from security degradation to full recovery. This effectively solves the problems of high latency and error susceptibility caused by manual configuration recovery in traditional operations and maintenance, significantly improving the level of automated operations and maintenance and business resilience of distributed storage systems in complex failure scenarios.

[0154] In some embodiments, before restoring the distributed coordination service cluster from a two-node configuration to a four-node configuration, the process may further include: after detecting that two server nodes in another chassis meet the expansion conditions, sending a pre-join request to the two server nodes in the other chassis; receiving the transaction identifier (zxid) at the end of the local transaction log and the service start timestamp returned by each server node in the other chassis; if the zxid of any server node is higher than the highest zxid in the current cluster, or its service start timestamp is later than the time of the most recent master node election in the current distributed coordination service cluster, then delaying the joining operation of that server node and triggering that server node to perform local data rollback to a state consistent with the current distributed coordination service cluster; after confirming that both server nodes in the other chassis are in a state compatible with the current cluster, formally incorporating them into the cluster configuration to complete the four-node recovery.

[0155] By introducing a pre-joining state negotiation mechanism during the scaling-up of the distributed coordination service cluster from two nodes back to four nodes, server nodes with inconsistent states can be prevented from joining the distributed coordination service cluster.

[0156] Before formally incorporating a server node from another chassis into the current cluster, the current cluster master node first sends a pre-join request to that server node and obtains its transaction identifier (zxid) at the end of its local transaction log and its service startup timestamp. By comparing the zxid with the highest zxid in the current cluster and the startup timestamp with the time of the most recent master node election in the current cluster, it determines whether the node is in a compatible state with the current cluster. If a server node from another chassis is found to have an abnormally high zxid (possibly carrying unsynchronized phantom transactions) or a startup time earlier than the current cluster's master election completion time (possibly with residual expired state), its incorporation is delayed and a local data rollback is triggered to force it to align with the current cluster state before it is allowed to formally participate in the cluster. This avoids master election conflicts or data forks caused by fault-recovered nodes carrying invalid high zxids or expired data. By effectively identifying and handling server nodes that were briefly elected before a power outage but failed to complete consensus, the potential threat to the current cluster's authority state is eliminated. By adding a runtime status verification step to the automatic recovery process, the expansion conditions include not only network connectivity and service integrity files, but also logical state compatibility, thereby improving the reliability and data security of the distributed storage system under complex failure scenarios.

[0157] In one possible implementation, when determining whether a service integrity file exists on the current server node and another server node within its chassis, the method further includes: acquiring multiple types of content from the service integrity file; calculating the corresponding hash value for each type of service integrity file based on a preset hash algorithm; comparing the calculated hash value with a baseline hash value pre-stored locally or in the cluster metadata; if the hash values ​​of all service integrity files match the corresponding baseline hash value, the service integrity file is determined to be complete and unaltered, allowing the corresponding server node to participate in scaling down or scaling up operations; if the hash value of any service integrity file does not match the baseline hash value, the corresponding server node is marked as disabled and prohibited from joining the distributed coordination service cluster.

[0158] Specifically, when determining whether a service integrity file exists on the current server node and another server node within its chassis, a security verification process based on cryptographic verification is further included.

[0159] First, the various types of content in the service integrity files can be obtained separately. Service integrity files include at least three key components: configuration files, transaction log files, and snapshot files.

[0160] Subsequently, a hash value can be calculated for each type of service integrity file based on a preset hash algorithm (such as a collision-resistant one-way hash function like a secure hash algorithm or a message digest algorithm). This hash value serves as a unique digital fingerprint of the file content, sensitively reflecting any byte-level modification or corruption.

[0161] Next, each calculated hash value can be compared item by item with a baseline hash value pre-stored in the local secure zone or cluster metadata storage. The baseline hash value is an authoritative reference value that is generated and permanently stored when the service is running normally and the file status is trustworthy.

[0162] If the hash values ​​calculated for all categories of service integrity files are completely consistent with the corresponding baseline hash values, then the service integrity files on the server node are determined to be complete, tamper-proof, and in a trusted state, allowing the server node to participate in subsequent scaling-down or scaling-up operations.

[0163] If the hash value of any type of service integrity file is inconsistent with the baseline hash value (including cases of missing files, truncated content, accidental modification or tampering, etc.), the server node can be immediately marked as disabled and excluded from the member management module of the distributed coordination service cluster, prohibiting it from joining the cluster or participating in master node election.

[0164] The above-described integrity verification process based on hash verification can effectively prevent cluster startup failures or data inconsistencies caused by file corruption, configuration drift, or security attacks, significantly improving the reliability and security of scaling down / expanding operations, and providing a solid data foundation for the high availability and self-healing capabilities of distributed storage systems.

[0165] In one possible implementation, before scaling down the distributed coordination service cluster deployed on each server node from a four-node configuration to a two-node configuration including two server nodes in the current chassis, the method further includes: obtaining system load information of the two server nodes in the current chassis, including CPU utilization, memory usage, and input / output throughput; if the system load information of at least one server node does not exceed a preset threshold, then a scaling-down operation is performed, which involves scaling down the distributed coordination service cluster deployed on each server node from a four-node configuration to a two-node configuration including two server nodes in the current chassis; if the system load information of both server nodes exceeds the preset threshold, then the scaling-down operation is paused and a system alarm message is generated.

[0166] In this embodiment of the application, before scaling down the distributed coordination service cluster deployed on each server node from a four-node configuration to a two-node configuration containing two server nodes in the current chassis, the method further includes a system resource load analysis mechanism to ensure that the scaling down operation will not cause service performance degradation or secondary failures due to overload of target node resources.

[0167] Specifically, the current server node can first obtain real-time system load information of the two server nodes within the current chassis through system monitoring interfaces (such as process information file systems, performance counters, or agent acquisition modules). System load information includes at least three key indicators: Central Processing Unit (CPU) utilization, memory usage, and input / output throughput.

[0168] CPU utilization reflects the current computing resource usage of a server node. Memory utilization represents the proportion of used physical memory to total available memory and is used to analyze caching and transaction processing capabilities. Input / output throughput measures the data read / write rate of disks or storage devices and directly affects transaction log writing and snapshot loading performance.

[0169] Subsequently, each of the above load metrics can be compared with a preset threshold. Specifically, the preset thresholds can be dynamically configured based on hardware specifications, Service Level Agreements (SLAs), or historical load baselines. For example, the CPU utilization threshold can be set to 80%, the memory utilization threshold to 85%, and the input / output throughput threshold to 70% of the device's maximum bandwidth.

[0170] If the system load information of at least one server node does not exceed the corresponding preset threshold (i.e., the server node has sufficient remaining resources), it is determined that the current chassis has the capacity to support a two-node ZooKeeper cluster, and a scaling-down operation is allowed, that is, the distributed coordination service cluster is safely switched from the original four-node configuration to a two-node configuration containing only two server nodes in the current chassis.

[0171] If the system load information of both server nodes exceeds their respective preset thresholds (i.e., both nodes in the same frame are under high load), the scaling down operation will be paused to avoid forced merging of services under resource constraints, which could lead to increased response latency, transaction backlog, or even service crashes. System alarm information (such as log records or operation and maintenance platform notifications) will be generated simultaneously to prompt the administrator to intervene and investigate or wait for the load to drop.

[0172] By introducing this load-aware mechanism, the problem of creating new bottlenecks during service recovery can be effectively avoided, achieving coordinated optimization of fault self-healing and resource assurance. Even if the preconditions such as overall system failure, normal communication, and file integrity are met, resource availability verification is still required before scaling down can be performed, thus ensuring both high availability and service quality stability in complex environments.

[0173] Figure 7 This is a flowchart illustrating another method for fault detection in a distributed storage system provided in an embodiment of this application. Figure 7 As shown, the current server node can detect the communication status of the other three server nodes (excluding itself) via the data network, onboard direct connection network, and Ethernet direct connection network. Based on the communication status, it determines whether there is a chassis-wide failure. If the failure type is determined to be a cross-chassis network partition, the distributed coordination service cluster is not scaled down. Instead, the connectivity of the cross-chassis coordination service is maintained via the Ethernet direct connection network, and normal operation is automatically restored after the data network recovers. If the failure type is determined to be a chassis-wide failure, and the current server node communicates normally with another server node in its chassis, and the service integrity files for the distributed coordination service exist on both the current server node and the other server node in its chassis, the distributed coordination service cluster deployed on each server node is scaled down from a four-node configuration to a two-node configuration including the two server nodes in the current chassis. Then, based on the two server nodes in the current chassis, a new master node is elected for the distributed coordination service cluster, and the distributed coordination service is restored. Then, check whether the two server nodes of the other chassis meet the expansion conditions. If the expansion conditions are met, restore the distributed coordination service cluster from a two-node configuration to a four-node configuration and reload the complete cluster status data to synchronize the transaction log.

[0174] Figure 8 This is a schematic diagram of a fault detection device for a distributed storage system provided in an embodiment of this application. Please refer to... Figure 8 The distributed storage system fault detection device may include:

[0175] The detection module 81 is used to detect the communication status of the other three server nodes besides itself via the data network, the onboard direct connection network, and the Ethernet direct connection network.

[0176] The fault determination module 82 is used to determine the fault type of the distributed storage system based on the communication status.

[0177] The scaling-down module 83 is used to scale down the distributed coordination service cluster deployed on each server node from a four-node configuration to a two-node configuration containing two server nodes in the current chassis when the fault type is a whole chassis fault, and the current server node communicates normally with another server node in its chassis, and the service integrity files of the distributed coordination service exist on both the current server node and the other server node in its chassis.

[0178] Service recovery module 84 is used to re-elect the master node for the distributed coordination service cluster based on the two server nodes in the current chassis, and to restore the distributed coordination service.

[0179] In one possible implementation, the fault determination module 82 is specifically used for:

[0180] Based on the communication status obtained through the data network, onboard direct connection network, and Ethernet direct connection network, with the current server node as the source node, recursive route probing is performed on the other three server nodes excluding itself to obtain the set of reachable paths from the source node to each target node. The target node is any other node in the distributed storage system other than the source node.

[0181] When the set of reachable paths contains only one server node within the same chassis, and two server nodes in another chassis cannot be reached through any relay path, the fault type is determined to be a chassis-wide fault.

[0182] When the set of reachable paths includes some or all of the server nodes of another chassis, the fault type is determined to be cross-chassis network partition.

[0183] In one possible implementation, the fault determination module 82 is further configured to:

[0184] Set a hop count threshold and a single path detection timeout for each probe.

[0185] If no response is received from the target node or relay node within the timeout period, the path is marked as unreachable.

[0186] If the recursion depth reaches the hop count threshold and reachability is still not determined, the probe is terminated and the fault type is determined based on the existing path set.

[0187] In one possible implementation, the service recovery module 84 is further configured to:

[0188] Check whether the two server nodes in the other chassis meet the expansion conditions. The expansion conditions are that the two server nodes in the current chassis can establish a connection with the two server nodes in the other chassis through the data network, and the two server nodes in the other chassis have the service integrity file of the distributed coordination service.

[0189] When the expansion conditions are met, the distributed coordination service cluster will be restored from a two-node configuration to a four-node configuration, and the complete cluster state data will be reloaded to synchronize the transaction log.

[0190] In one possible implementation, the shrinkage module 83 is specifically used for:

[0191] Retrieve the various types of content from the service integrity file.

[0192] Based on a preset hash algorithm, the corresponding hash value is calculated for each type of service integrity file.

[0193] The calculated hash value is compared item by item with the baseline hash value pre-stored in the local or cluster metadata.

[0194] If the hash values ​​of all service integrity files are consistent with the corresponding baseline hash values, then the service integrity files are determined to be complete and have not been tampered with, and the corresponding server nodes are allowed to participate in scaling down or scaling up operations.

[0195] If the hash value of any service integrity file is inconsistent with the baseline hash value, the corresponding server node is marked as disabled and prohibited from joining the distributed coordination service cluster.

[0196] In one possible implementation, the shrinkage module 83 is further configured to:

[0197] Obtain the system load information of the two server nodes in the current chassis. The system load information includes CPU utilization, memory usage, and input / output throughput.

[0198] If the system load information of at least one server node does not exceed the preset threshold, a scaling-down operation is performed. The scaling-down operation involves reducing the distributed coordination service cluster deployed on each server node from a four-node configuration to a two-node configuration containing two server nodes within the current chassis.

[0199] If the system load information of both server nodes exceeds the preset threshold, the scaling down operation will be paused and a system alarm message will be generated.

[0200] The distributed storage system fault detection device provided in this application embodiment can execute the technical solution shown in the above method embodiment. Its implementation principle and beneficial effects are similar, and will not be described again here.

[0201] Figure 9 This is a schematic diagram of the structure of a fault detection device for a distributed storage system provided in an embodiment of this application. Please refer to [link / reference]. Figure 9 The fault detection device for the distributed storage system may include a memory 91 and a processor 92. Exemplarily, the memory 91 and the processor 92 are interconnected via a bus 93.

[0202] The memory 91 is used to store program instructions; the processor 92 is used to execute the program instructions stored in the memory, so that the data processing device performs the method shown in the above method embodiment.

[0203] The distributed storage system fault detection device provided in this application embodiment can execute the technical solution shown in the above method embodiment. Its implementation principle and beneficial effects are similar, and will not be described again here.

[0204] This application provides a computer-readable storage medium storing computer-executable instructions, which are used to implement the above-described method when executed by a processor.

[0205] This application embodiment may also provide a computer program product, including a computer program that, when executed by a processor, can implement the above-described method.

[0206] All or part of the steps in the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a readable memory. When the program is executed, it performs the steps of the above-described method embodiments; and the aforementioned memory (storage medium) includes: read-only memory (ROM), random access memory (RAM), flash memory, hard disk, solid-state drive, magnetic tape, floppy disk, optical disc, and any combination thereof.

[0207] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processing unit of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0208] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0209] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0210] Obviously, those skilled in the art can make various modifications and variations to the embodiments of this application without departing from the spirit and scope of this application. Therefore, if these modifications and variations to the embodiments of this application fall within the scope of the claims of this application and their equivalents, this application also intends to include these modifications and variations.

[0211] In this application, the term "comprising" and its variations can refer to non-limiting inclusion; the term "or" and its variations can refer to "and / or". The terms "first", "second", etc., in this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. In this application, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

Claims

1. A fault detection method for a distributed storage system, characterized in that, The method is applied to any server node in a distributed storage system, wherein the distributed storage system includes two chassis, each chassis containing two server nodes, and the method includes: The communication status of the other three server nodes (excluding itself) is detected through the data network, onboard direct connection network, and Ethernet direct connection network. Based on the communication status, determine the fault type of the distributed storage system; When the fault type is a whole-frame fault, and the current server node communicates normally with another server node in its own frame, and the service integrity files of the distributed coordination service exist on both the current server node and the other server node in its own frame, the distributed coordination service cluster deployed on each server node will be scaled down from a four-node configuration to a two-node configuration containing two server nodes in the current frame. The service integrity files include the configuration file, transaction log file, and snapshot file of the distributed coordination service. Based on the two server nodes in the current chassis, a new master node is elected for the distributed coordination service cluster, and the distributed coordination service is restored.

2. The method according to claim 1, characterized in that, Determining the fault type of the distributed storage system based on the communication status includes: Based on the communication status obtained through the data network, onboard direct connection network and Ethernet direct connection network, taking the current server node as the source node, recursive route probing is performed on the other three server nodes excluding itself to obtain a set of reachable paths from the source node to each target node, where the target node is any other node in the distributed storage system excluding the source node. When the set of reachable paths contains only one other server node within the same chassis, and two server nodes in another chassis cannot be reached through any relay path, the fault type is determined to be a chassis-wide fault. When the set of reachable paths includes some or all of the server nodes of another chassis, the fault type is determined to be a cross-chassis network partition.

3. The method according to claim 2, characterized in that, The recursive route probing includes: Set a hop count threshold and a single path detection timeout for each probe; If no response is received from the target node or relay node within the timeout period, the path is marked as unreachable. If the recursion depth reaches the hop count threshold and reachability is still not determined, the probe is terminated and the fault type is determined based on the existing path set.

4. The method according to claim 1, characterized in that, After restoring the distributed coordination service, the method further includes: The system checks whether the two server nodes in the other chassis meet the expansion conditions. The expansion conditions are that the two server nodes in the current chassis can establish a connection with the two server nodes in the other chassis through the data network, and the service integrity file of the distributed coordination service exists on the two server nodes in the other chassis. When the expansion conditions are met, the distributed coordination service cluster is restored from a two-node configuration to a four-node configuration, and the complete cluster status data is reloaded to synchronize the transaction log.

5. The method according to any one of claims 1 to 4, characterized in that, When determining whether the service integrity file exists on the current server node and another server node within its chassis, the method further includes: Obtain the various types of content from the service integrity file respectively; Based on a preset hash algorithm, the corresponding hash value is calculated for each type of service integrity file. The calculated hash value is compared item by item with the baseline hash value pre-stored in the local or cluster metadata; If the hash value of all service integrity files is consistent with the corresponding baseline hash value, then the service integrity file is determined to be complete and has not been tampered with, and the corresponding server node is allowed to participate in the scaling down or scaling up operation. If the hash value of any of the service integrity files is inconsistent with the baseline hash value, the corresponding server node is marked as disabled and prohibited from joining the distributed coordination service cluster.

6. The method according to claim 1, characterized in that, Before scaling down the distributed coordination service cluster deployed on each server node from a four-node configuration to a two-node configuration including two server nodes within the current chassis, the method further includes: Obtain the system load information of the two server nodes in the current chassis. The system load information includes CPU utilization, memory usage, and input / output throughput. If the system load information of at least one server node does not exceed a preset threshold, a scaling-down operation is performed. The scaling-down operation is to reduce the distributed coordination service cluster deployed on each server node from a four-node configuration to a two-node configuration containing two server nodes in the current chassis. If the system load information of both server nodes exceeds the preset threshold, the scaling down operation is paused and a system alarm message is generated.

7. A fault detection device for a distributed storage system, characterized in that, An apparatus applicable to any server node in a distributed storage system, wherein the distributed storage system comprises two chassis, each chassis housing two server nodes, the apparatus comprising: The detection module is used to detect the communication status of the other three server nodes besides itself via the data network, onboard direct connection network and Ethernet direct connection network. The fault determination module is used to determine the fault type of the distributed storage system based on the communication status. The scaling-down module is used to scale down the distributed coordination service cluster deployed on each server node from a four-node configuration to a two-node configuration containing two server nodes in the current frame when the fault type is a whole-frame fault, and the current server node communicates normally with another server node in its frame, and the service integrity files of the distributed coordination service exist on both the current server node and the other server node in its frame. The service recovery module is used to re-elect a master node for the distributed coordination service cluster based on the two server nodes in the current chassis, and to restore the distributed coordination service.

8. A fault detection device for a distributed storage system, characterized in that, include: At least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, in, The computer instructions are used to cause the computer to perform the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 6.

Images