Storage dual-active fault processing method and device based on distributed cooperative detection and storage medium

Abstract

The application discloses a storage dual-active fault processing method and device based on distributed cooperative detection and a storage medium, and relates to the technical field of storage management. The method comprises the following steps: defining nodes as data storage nodes in a site A and a site B; constructing a fault detection cluster based on a SWIM protocol according to the nodes in the site A and the site B; constructing an arbitration node independent of the fault detection cluster; when the fault detection cluster determines that a site-level fault occurs, sending a site-level fault detection result to the arbitration node; and according to the site fault detection result, the arbitration node performs fault arbitration based on a preset decision rule and generates a fault recovery scheme. The application can avoid single-point fault risks, realize accurate positioning of faults, and automatically select an optimal site to continue service when a site-level fault occurs, thereby improving the quality of fault recovery and business continuity.

Description

Technical Field

[0001] This application relates to the field of storage management technology, specifically to a storage dual-active fault handling method, device, and storage medium based on distributed collaborative detection. Background Technology

[0002] Existing active-active storage architectures typically consist of storage arrays located in two data centers (such as site A and site B), using synchronous mirroring technology to achieve dual data writes, thereby providing a single, highly available storage volume. The fault detection technology in existing active-active storage architectures relies on a centralized heartbeat detection mechanism; and the fault decision-making method depends on preset static decision rules.

[0003] However, the centralized heartbeat detection mechanism and static decision rules in the existing active-active storage architecture have the following drawbacks: they cannot quickly and accurately identify the type of failure, are prone to triggering single points of failure, leading to unnecessary service switching or even business interruption, and the failure decision mechanism is rigid and cannot make the optimal failure recovery choice. Summary of the Invention

[0004] In view of the deficiencies in the existing technology, the technical problem to be solved by this application is: how to achieve fast and accurate fault detection and intelligent fault decision-making in a dual-active storage architecture.

[0005] To achieve the above objectives, in a first aspect, embodiments of this application provide a storage dual-active fault handling method based on distributed collaborative detection, the steps of which include: Define the nodes as data storage nodes in site A and site B; Based on the SWIM protocol, a fault detection cluster is constructed according to the nodes of site A and site B; and an arbitration node independent of the fault detection cluster is constructed. When the fault detection cluster determines that a site-level fault has occurred, the site-level fault detection result is sent to the arbitration node; Based on the site fault detection results, the arbitration node performs fault arbitration based on the preset winning rules and generates a fault recovery plan.

[0006] In conjunction with the first aspect, in one implementation, the process by which the fault detection cluster determines that a site-level fault has occurred includes: S1. Node X sends a dping probe message to node Y to determine whether node Y is reachable. If it is, it means that both node X and node Y are working normally, and the fault detection ends; otherwise, proceed to S2. S2. Determine if node Y and node X are at the same station. If yes, proceed to S3; otherwise, proceed to S4. S3. Node X sends an iping request to the remaining nodes in the same site. After receiving the iping request, the remaining nodes send a dping probe message to node Y. When at least two remaining nodes determine that node Y is unreachable, node Y is marked as faulty. S4. Node X sends a dping probe message to the remaining nodes in the same site as node Y. When all remaining nodes are unreachable, the site where node Y is located is marked as having a suspected site-level fault. S5. Node X sends dping probe messages to the remaining nodes in the same site to determine whether there are at least two remaining nodes that can reach it. If so, it is determined that there is a site-level fault in the site where node Y is located. At this time, node X sends the site-level fault detection result to the arbitration node; otherwise, node X is marked as suspected faulty.

[0007] In conjunction with the first aspect, in one implementation, the process of determining whether node Y is reachable includes: After node X sends a dping probe message to node Y, it determines whether node X receives a dping response from node Y within a preset timeout period. If it does, node Y is reachable; otherwise, node Y is unreachable.

[0008] In conjunction with the first aspect, in one implementation, after marking node X as potentially faulty in step S5, node fault diagnosis is performed. This node fault diagnosis process includes: Node Z at the same site as node X sends an iping request to the remaining nodes in the fault detection cluster. After receiving the iping request, the remaining nodes send dping probe messages to node X. When all remaining nodes determine that node X is unreachable, node X is determined to be a faulty node.

[0009] In conjunction with the first aspect, in one implementation, when node X sends the site-level fault detection result to the arbitration node, node X sends an iping request to the remaining nodes at the same site. After receiving the iping request, each of the remaining nodes repeats operations S1-S5 in parallel. When any node determines that there is a site-level fault at the site where node Y is located, that node sends the site-level fault detection result to the arbitration node.

[0010] In conjunction with the first aspect, in one implementation, before the arbitration node performs fault arbitration based on the site fault detection result, the arbitration node determines whether it has received the same site-level fault result from at least two nodes at the same site within a preset time. If so, the arbitration node performs fault arbitration based on the site fault detection result; otherwise, the arbitration node does not perform fault arbitration.

[0011] In conjunction with the first aspect, in one implementation, the process of the arbitration node performing fault arbitration based on preset winning rules and generating a fault recovery plan according to the site fault detection results includes: The arbitration node sends health check requests to the nodes in site A and site B respectively. Based on the feedback results of the health check requests, it determines the number of normally functioning nodes in site A and site B. Then, it determines the winning site by combining the preset winning rules and uses the winning site to provide read and write services.

[0012] In conjunction with the first aspect, in one implementation, the preset winning rules include: When the number of normally functioning nodes in both site A and site B is not zero, a split-brain fault occurs. In this case, it is determined whether the number of normally functioning nodes in site A and site B are equal. If they are equal, the winning site is determined according to the preset priority rules; otherwise, the site with more normally functioning nodes is selected as the winning site. If either the number of normally functioning nodes in site A or the number of normally functioning nodes in site B is 0, the site with a non-zero number is selected as the winning site. When the number of normally functioning nodes at site A and site B are both 0, it indicates that both sites are experiencing a failure.

[0013] Secondly, embodiments of this application provide a storage dual-active fault handling device based on distributed collaborative detection. The storage dual-active fault handling device based on distributed collaborative detection includes a processor, a memory, and a storage dual-active fault handling program based on distributed collaborative detection stored in the memory and executable by the processor. When the storage dual-active fault handling program based on distributed collaborative detection is executed by the processor, it implements the method provided in the first aspect.

[0014] Thirdly, embodiments of this application provide a computer-readable storage medium storing a storage dual-active fault handling program based on distributed collaborative detection, wherein when the storage dual-active fault handling program based on distributed collaborative detection is executed, it implements the method provided in the first aspect.

[0015] Compared with the prior art, the advantages of this application are: By introducing the distributed SWIM protocol and building a fault detection cluster by combining the data storage nodes of two sites, the risk of single point of failure can be avoided, and the fault can be accurately located. Furthermore, by using an arbitration node independent of the fault detection cluster to perform layered analysis and arbitration of the fault, and by making comprehensive decisions based on information from multiple parties and preset winning rules, the optimal site can be automatically selected to continue service, thereby improving the quality of fault recovery and business continuity.

[0016] By conducting node communication self-checks and majority voting acceptance before site-level fault reporting, a dual false alarm filtering mechanism is established, avoiding unnecessary service switching or even business interruption, thereby improving the operational stability of the active-active storage architecture. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the process by which the fault detection cluster in the application determines the occurrence of a site-level fault. Figure 2 This is a schematic diagram of the hardware structure of the storage dual-active fault handling device based on distributed collaborative detection involved in the embodiments of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content and operations / steps, nor does it necessarily have to be performed in the order described. For example, some operations / steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.

[0021] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0022] In a first aspect, embodiments of this application provide a storage dual-active fault handling method based on distributed collaborative detection, the method comprising the following steps: Define the nodes as data storage nodes in site A and site B; Based on the SWIM protocol, a fault detection cluster is constructed according to the nodes of site A and site B; and an arbitration node independent of the fault detection cluster is constructed. When the fault detection cluster determines that a site-level fault has occurred, the site-level fault detection result is sent to the arbitration node; Based on the site fault detection results, the arbitration node performs fault arbitration based on the preset winning rules and generates a fault recovery plan.

[0023] By introducing the distributed SWIM protocol and building a fault detection cluster by combining the data storage nodes of the two sites, the iping and dping mechanisms of the SWIM protocol can avoid the risk of single point of failure and achieve accurate fault location. Furthermore, by using an arbitration node independent of the fault detection cluster to perform layered analysis and arbitration of faults, and by making comprehensive decisions based on information from multiple parties and preset winning rules, the optimal site can be automatically selected to continue service, thereby improving the quality of fault recovery and business continuity.

[0024] In one embodiment, when constructing the fault detection cluster, the nodes in the fault detection cluster are labeled as site A and site B.

[0025] Based on this, refer to Figure 1 , Figure 1 This is a flowchart illustrating the process by which the fault detection cluster in the application determines if a site-level fault has occurred. For example... Figure 1 As shown, the process by which the fault detection cluster determines a site-level fault includes: S1. Node X periodically (e.g., once every 1 second) sends dping probe messages to node Y to determine whether node Y is reachable. If it is, it means that both node X and node Y are working normally, and the fault detection ends; otherwise, proceed to S2. S2. Determine if node Y and node X are at the same station (based on the station markers of node Y and node X). If yes, proceed to S3; otherwise, proceed to S4. S3. Node X sends an iping request to the remaining nodes in the same site. After receiving the iping request, the remaining nodes send a dping probe message to node Y. When at least two remaining nodes determine that node Y is unreachable, node Y is marked as faulty. S4. Node X sends a dping probe message to the remaining nodes in the same site as node Y. When all remaining nodes are unreachable, the site where node Y is located is marked as having a suspected site-level fault. S5. Node X sends dping probe messages to the remaining nodes in the same site to determine whether there are at least two remaining nodes that can reach it. If so, it is determined that there is a site-level fault in the site where node Y is located. At this time, node X sends the site-level fault detection result to the arbitration node; otherwise, node X is marked as suspected faulty.

[0026] Thus, the iping and dping mechanisms are used to locate node faults and report site-level faults, providing a basis for hierarchical fault decision-making. Furthermore, the dping communication self-check of node X in S5 achieves the first layer of filtering for false fault reports.

[0027] The process for determining whether node Y is reachable includes: After node X sends a dping probe message to node Y, it determines whether node X receives a dping response from node Y within a preset timeout period. If it does, node Y is reachable; otherwise, node Y is unreachable.

[0028] Similarly, the method for determining whether at least two remaining nodes are reachable in step S5 is the same as the process for determining whether node Y is reachable described above.

[0029] Furthermore, while node X sends the site-level fault detection result to the arbitration node, node X sends an iping request to the remaining nodes at the same site. After receiving the iping request, each of the remaining nodes repeats the above S1-S5 operations in parallel. When any node determines that there is a site-level fault at the site where node Y is located, that node sends the site-level fault detection result to the arbitration node.

[0030] By reporting site-level faults individually from multiple nodes, we can provide multi-party information for the comprehensive decision-making of the arbitration node, and provide a data foundation for building a second layer of filtering for false fault reports.

[0031] In one embodiment, after marking node X as potentially faulty in S5 above, node fault diagnosis is performed. The node fault diagnosis process includes: Node Z, located at the same site as node X, sends an iping request to the remaining nodes in the fault detection cluster. Upon receiving the iping request, the remaining nodes send dping probe messages to node X. If all remaining nodes determine that node X is unreachable (referring to the process described above for determining the reachability of node Y), then node X is identified as a faulty node. At this point, the anti-entropy mechanism based on the SWIM protocol propagates within the fault detection cluster, reaching a consensus, and the remaining nodes at the same site as node X take over node X's storage services.

[0032] In this way, the fault detection cluster can directly locate node-level faults, effectively avoiding the risk of single-point failures.

[0033] In one embodiment, before the arbitration node performs fault arbitration based on the site fault detection result, the arbitration node determines whether it has received the same site-level fault result from at least two other nodes at the same site (from other nodes at the site where the fault reporting node is located) within a preset time (e.g., 60s). If so, the arbitration node performs fault arbitration based on the site fault detection result; otherwise, the arbitration node does not perform fault arbitration.

[0034] This majority voting method serves as a second layer of filtering for false alarms, preventing unnecessary service switching or even business interruptions.

[0035] Based on this, according to the site fault detection results, the arbitration node performs fault arbitration based on preset winning rules, and the process of generating a fault recovery plan includes: The arbitration node sends health check requests to the nodes in site A and site B respectively. Based on the feedback results of the health check requests, it determines the number of normally functioning nodes in site A and site B. Then, it determines the winning site by combining the preset winning rules and uses the winning site to provide read and write services.

[0036] The aforementioned pre-defined winning rules include: When the number of normally functioning nodes in site A and site B are both not zero, it indicates a split-brain (i.e., network partitioning between sites) failure. In this case, it is determined whether the number of normally functioning nodes in site A and site B are equal. If they are, the winning site is determined according to the preset priority rules; otherwise, the site with more normally functioning nodes is selected as the winning site. In this case, the non-winning sites can still provide data services in read-only mode. If either the number of normally functioning nodes in site A or the number of normally functioning nodes in site B is 0, the site with a non-zero number is selected as the winning site. When the number of normally functioning nodes in site A and site B are both 0, it indicates that both sites have failed, and there is no winning site.

[0037] In this embodiment, the arbitration node is independent of the fault detection cluster and is not affected by the nodes in the fault detection cluster. It performs lightweight arbitration of faults only based on the site-level fault reporting results and preset winning rules. It can automatically select the optimal site to continue service based on the preset winning rules, thereby improving the quality of fault recovery and business continuity.

[0038] The following describes the fault judgment process for site A and site B using four typical fault scenarios. Site A includes nodes 1, 2, and 3; site B includes nodes 4, 5, and 6. After site A and site B communicate with each other via the SWIM protocol, an arbitration node is established to communicate with both site A and site B.

[0039] Scenario 1: In a scenario where site A experiences a complete failure (such as a power outage), site A crashes due to a mains power interruption, with nodes 1, 2, and 3 momentarily offline. The fault diagnosis process for site A in this case includes: S1, Node 4 sends a dping probe message to Node 1, but Node 4 does not receive a dping response from Node 1 within the predetermined timeout period; S2 and Node 4 send dping probe messages to Node 2 and Node 3 in sequence. Node 4 does not receive dping responses from Node 2 and Node 3 within the predetermined timeout period, and Node 4 determines that there is a suspected site-level fault in site A. S3 and Node 4 send dping probe messages to Node 5 and Node 6. After confirming that Node 5 and Node 6 are reachable, Node 4 determines that there is a site-level fault in Site A and reports it to the arbitration node. S4, Node 5, and Node 6 repeat the S1-S3 process (e.g., in S2, Node 5 sends dping probe messages to Node 2 and Node 3 in turn; in S3, Node 5 sends dping probe messages to Node 4 and Node 6 to confirm that Node 4 and Node 6 are reachable). After independently determining that there is a site-level fault in Site A, Node 5 and Node 6 independently report the site-level fault event of Site A to the arbitration node. S5. Within 60 seconds, the arbitration node receives the site-level fault detection results reported by nodes 4, 5, and 6, confirming the credibility of the event; then it sends health check requests to nodes 1-6, and the feedback results are: all sites A are unresponsive, and all sites B are normal. S6. The arbitration node, based on the preset winning rules (see the preset winning rules above), determines that site B wins and issues a "site A confirms the fault, site B takes over the service" instruction to site B. At this time, site B independently undertakes the read and write services.

[0040] Scenario 2: In a split-brain failure scenario between site A and site B, the backbone network connecting site A and site B is interrupted, causing a network link between site A and site B to be broken. However, the internal networks of site A and site B themselves, as well as their connections with the arbitration node, remain normal. The process for determining if a split-brain failure has occurred includes: S1. Node 1 fails to send a dping probe message to Node 4. Node 1 then sends dping probe messages to Node 5 and Node 6, which also fail. Node 1 determines that there is a suspected site-level fault at site B. In parallel, node 4 failed to send a dping probe message to node 1. Node 4 then sent dping probe messages to nodes 2 and 3, which also failed. Node 4 determined that there was a suspected site-level fault at site A. S2. Node 1 sends dping probe messages to Node 2 and Node 3. After confirming that Node 2 and Node 3 are reachable, Node 1 determines that there is a site-level fault at site B and reports it to the arbitration node. In parallel, node 4 sends dping probe messages to nodes 5 and 6. After confirming that nodes 5 and 6 are reachable, node 4 determines that there is a site-level fault in site A and reports it to the arbitration node. S3, Node 2, and Node 3 repeat the S1-S2 process (e.g., in S1, Node 2 sends dping probe messages to Node 4, Node 5, and Node 6 in sequence, and the sending fails; in S2, Node 2 sends dping probe messages to Node 1 and Node 3 to confirm that Node 1 and Node 3 are reachable). After independently determining that there is a site-level fault in Site B, Node 2 and Node 3 independently report the site-level fault event of Site B to the arbitration node. In parallel, nodes 5 and 6 repeat the S1-S2 process. After independently determining that there is a site-level fault in site A, nodes 5 and 6 independently report the site-level fault event of site A to the arbitration node. S4. Within 60 seconds, the arbitration node receives the site-level fault detection results reported by nodes 1-6, and then sends a health check request to all nodes of site A and site B. At this time, the feedback result is: site A and site B are all normal. S5. The arbitration node, based on preset priority rules (e.g., site A has a higher priority than site B), determines that site A wins and issues a "Win, continue service" instruction to site A, and a "Loss, stop service" instruction to site B. At this time, site A undertakes read and write services, while site B stops read and write services or enters read-only mode.

[0041] In scenario 3, where Node 1 experiences network isolation, Node 1 suffers a complete loss of connection with all other nodes (including Nodes 2-6 and the arbitrator node) due to a local switch failure. However, Node 1's operating system and storage service processes may still be running. The process for determining if Node 1 is experiencing network isolation includes: S1. Node 1 fails to send a dping probe message to Node 4. Node 1 then sends dping probe messages to Node 5 and Node 6, which also fail. Node 1 determines that there is a suspected site-level fault at site B.

[0042] S2. Node 1 sends dping probe messages to Node 2 and Node 3. If the sending fails, it is determined that Node 1 is suspected of being faulty. At this time, it is a node-level fault and will not be reported to the arbitration node. S3. Node 2 fails to send a dping probe message to Node 1. Node 2 randomly selects two nodes, such as Node 3 and Node 4, and initiates an iping request to Node 3 and Node 4 (requesting to probe Node 1). After receiving the iping request, Node 3 and Node 4 send a dping probe message to Node 1 but fail and send the failure result back to Node 2. Node 4 sends a dping probe message to Node 1 but fails and sends the failure result back to Node 2. S5. After receiving the failure result feedback, node 2 determines that node 1 is a faulty node.

[0043] In scenario four, where node 1 experiences a hardware failure, node 1 completely shuts down due to a hardware failure (such as a motherboard or power supply failure). However, the network connection of its rack remains normal, and the physical links between node 1 and nodes 2 and 3 within site A, as well as nodes 4, 5, and 6 within site B, are intact. Node 1 is unresponsive. The process for determining if node 1 has experienced a hardware failure includes: S1, Node 2 sends a dping probe message to Node 1; at the same time, Node 4 sends a dping probe message to Node 1. S2, Node 2, and Node 4 did not receive a dping response from Node 1 within the scheduled timeout period; S3 and Node 2 randomly select two nodes, such as Node 3 and Node 4. Node 2 sends an iping request to Node 3 and Node 4 (requesting to probe Node 1).

[0044] After receiving the iping request, Node 3 and Node 4 send a dping probe message to Node 1, but Node 3 fails and sends the failure result back to Node 2. Similarly, Node 4 sends a dping probe message to Node 1, but Node 4 fails and sends the failure result back to Node 2. S5. After receiving the failure feedback, Node 2 determines that Node 1 is the faulty node. At this time, Node 2 and Node 3 take over the storage service of Node 1.

[0045] Secondly, embodiments of this application provide a storage dual-active fault handling device based on distributed collaborative detection. The storage dual-active fault handling device based on distributed collaborative detection can be a personal computer (PC), laptop computer, server, or other device with data processing capabilities.

[0046] Reference Figure 2 , Figure 2This is a schematic diagram of the hardware structure of a storage dual-active fault handling device based on distributed collaborative detection, as described in an embodiment of this application. In this embodiment, the storage dual-active fault handling device based on distributed collaborative detection may include a processor, a memory, a communication interface, and a communication bus.

[0047] The communication bus can be of any type and is used to interconnect the processor, memory, and communication interface.

[0048] The communication interface includes input / output (I / O) interfaces, physical interfaces, and logical interfaces used for interconnecting devices within the storage dual-active fault handling device based on distributed collaborative detection, as well as interfaces used for interconnecting the storage dual-active fault handling device with other devices (such as other computing devices or user equipment). Physical interfaces can be Ethernet interfaces, fiber optic interfaces, ATM interfaces, etc.; user equipment can be displays, keyboards, etc.

[0049] Memory can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), flash memory, optical storage, hard disk, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), etc.

[0050] The processor can be a general-purpose processor, which can call a storage dual-active fault handling program based on distributed cooperative detection stored in memory and execute the storage dual-active fault handling method based on distributed cooperative detection provided in the embodiments of this application. For example, the general-purpose processor can be a central processing unit (CPU). The method executed when the storage dual-active fault handling program based on distributed cooperative detection is called can be referred to in the various embodiments of the storage dual-active fault handling method based on distributed cooperative detection of this application, and will not be repeated here.

[0051] Those skilled in the art will understand that Figure 2 The hardware structure shown does not constitute a limitation of this application and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0052] Thirdly, embodiments of this application also provide a computer-readable storage medium.

[0053] The computer-readable storage medium of this application stores a storage dual-active fault handling program based on distributed cooperative detection, wherein when the storage dual-active fault handling program based on distributed cooperative detection is executed by a processor, it implements the steps of the storage dual-active fault handling method based on distributed cooperative detection as described above.

[0054] The method implemented when the storage dual-active fault handling program based on distributed collaborative detection is executed can be referred to in the various embodiments of the storage dual-active fault handling method based on distributed collaborative detection in this application, and will not be repeated here.

[0055] It should be noted that the sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0056] The terms "comprising" and "having," and any variations thereof, in the specification, claims, and accompanying drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus. The terms "first," "second," and "third," etc., are used to distinguish different objects, etc., and do not indicate a sequence, nor do they limit "first," "second," and "third" to different types.

[0057] In the description of the embodiments of this application, terms such as "exemplary," "for example," or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary," "for example," or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary," "for example," or "for instance" is intended to present the relevant concepts in a concrete manner.

[0058] In the description of the embodiments of this application, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. The "and / or" in the text is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of this application, "multiple" means two or more.

[0059] In some processes described in the embodiments of this application, multiple operations or steps are included in a specific order. However, it should be understood that these operations or steps may not be executed in the order they appear in the embodiments of this application, or they may be executed in parallel. The sequence number of the operation is only used to distinguish different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed sequentially or in parallel, and these operations or steps may be combined.

[0060] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device to execute the methods described in the various embodiments of this application.

[0061] The above are merely specific embodiments of this application, but the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application. Therefore, the protection scope of this application should be determined by the scope of the claims.

Claims

1. A storage dual-active fault handling method based on distributed collaborative detection, characterized in that, The steps of this method include: Define the nodes as data storage nodes in site A and site B; Based on the SWIM protocol, a fault detection cluster is constructed according to the nodes of site A and site B; and an arbitration node independent of the fault detection cluster is constructed. When the fault detection cluster determines that a site-level fault has occurred, the site-level fault detection result is sent to the arbitration node; Based on the site fault detection results, the arbitration node performs fault arbitration based on the preset winning rules and generates a fault recovery plan.

2. The storage dual-active fault handling method based on distributed collaborative detection as described in claim 1, characterized in that: The process by which the fault detection cluster determines a site-level fault includes: S1. Node X sends a dping probe message to node Y to determine whether node Y is reachable. If it is, it means that both node X and node Y are working normally, and the fault detection ends; otherwise, proceed to S2. S2. Determine if node Y and node X are at the same station. If yes, proceed to S3; otherwise, proceed to S4. S3. Node X sends an iping request to the remaining nodes in the same site. After receiving the iping request, the remaining nodes send a dping probe message to node Y. When at least two remaining nodes determine that node Y is unreachable, node Y is marked as faulty. S4. Node X sends dping probe messages to the remaining nodes in the same site as node Y. When all remaining nodes are unreachable, the site where node Y is located is marked as having a suspected site-level fault. S5. Node X sends dping probe messages to the remaining nodes in the same site to determine whether there are at least two remaining nodes that can reach it. If so, it is determined that there is a site-level fault in the site where node Y is located. At this time, node X sends the site-level fault detection result to the arbitration node; otherwise, node X is marked as suspected faulty.

3. The storage dual-active fault handling method based on distributed collaborative detection as described in claim 2, characterized in that: The process for determining whether node Y is reachable includes: After node X sends a dping probe message to node Y, it determines whether node X receives a dping response from node Y within a preset timeout period. If it does, node Y is reachable; otherwise, node Y is unreachable.

4. The storage dual-active fault handling method based on distributed collaborative detection as described in claim 2, characterized in that: After marking node X as potentially faulty in step S5, node fault diagnosis is performed. The node fault diagnosis process includes: Node Z at the same site as node X sends an iping request to the remaining nodes in the fault detection cluster. After receiving the iping request, the remaining nodes send dping probe messages to node X. When all remaining nodes determine that node X is unreachable, node X is determined to be a faulty node.

5. The storage dual-active fault handling method based on distributed collaborative detection as described in claim 2, characterized in that: When node X sends the site-level fault detection result to the arbitration node, node X sends an iping request to the remaining nodes at the same site. After receiving the iping request, each of the remaining nodes repeats operations S1-S5 in parallel. When any node determines that there is a site-level fault at the site where node Y is located, that node sends the site-level fault detection result to the arbitration node.

6. The storage dual-active fault handling method based on distributed collaborative detection as described in claim 5, characterized in that: Before the arbitration node performs fault arbitration based on the site fault detection results, the arbitration node determines whether it has received the same site-level fault results from at least two nodes at the same site within a preset time. If so, the arbitration node performs fault arbitration based on the site fault detection results. Otherwise, the arbitration node will not execute fault arbitration.

7. The storage dual-active fault handling method based on distributed collaborative detection as described in claim 6, characterized in that: The process by which the arbitration node performs fault arbitration based on preset winning rules and generates a fault recovery plan according to the site fault detection results includes: The arbitration node sends health check requests to the nodes in site A and site B respectively. Based on the feedback results of the health check requests, it determines the number of normally functioning nodes in site A and site B. Then, it determines the winning site by combining the preset winning rules and uses the winning site to provide read and write services.

8. The storage dual-active fault handling method based on distributed collaborative detection as described in claim 7, characterized in that: The preset winning rules include: When the number of normally functioning nodes in both site A and site B is not zero, a split-brain fault occurs. In this case, it is determined whether the number of normally functioning nodes in site A and site B are equal. If they are equal, the winning site is determined according to the preset priority rules; otherwise, the site with more normally functioning nodes is selected as the winning site. If either the number of normally functioning nodes in site A or the number of normally functioning nodes in site B is 0, the site with a non-zero number is selected as the winning site. When the number of normally functioning nodes at site A and site B are both 0, it indicates that both sites are experiencing a failure.

9. A storage dual-active fault handling device based on distributed collaborative detection, characterized in that, The storage dual-active fault handling device based on distributed collaborative detection includes a processor, a memory, and a storage dual-active fault handling program based on distributed collaborative detection stored in the memory and executable by the processor. When the storage dual-active fault handling program based on distributed collaborative detection is executed by the processor, it implements the steps of the storage dual-active fault handling method based on distributed collaborative detection as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a storage dual-active fault handling program based on distributed collaborative detection, wherein when the storage dual-active fault handling program based on distributed collaborative detection is executed, it implements the steps of the storage dual-active fault handling method based on distributed collaborative detection as described in any one of claims 1 to 8.

Images