A dual-layer arbitration method that effectively solves split-brain problems in two computer rooms

Abstract

The present invention discloses a double-layer arbitration method for effectively solving the brain split of dual computer rooms, comprising the following steps: S1: unloading the arbitration capability from the application layer to the arbitration cluster, registering services and querying the address and service of the master node through the priority access of the arbitration node in the computer room, configuring the cost computer room priority and remote backup access mode for the access of the three arbitration nodes; S2: each computer room contains two arbitration nodes, one arbitration node is a standard Follower node, and the other arbitration node is an Observer node. When the Leader / Follower in the computer room is down and cannot be accessed, priority is switched to accessing the Observer node; S3: after manual confirmation of the fault, the Observer node in the computer room is quickly promoted to the standard Follower role, realizing rapid recovery of the arbitration service and business recovery in the surviving computer room. The present invention improves the high availability of the arbitration capability, reduces the interruption frequency caused by the arbitration cluster failure, simplifies the fault scenario, and solves the problem of business interruption due to extreme faults.

Description

Technical Field

[0001] The present invention relates to the field of cloud computing distributed technology, and in particular to a double-layer arbitration method that effectively solves brain split in two computer rooms. Background Art

[0002] In the era of cloud computing, distributed deployment scenarios have gradually replaced previous centralized deployment scenarios. Through the collaborative work of multiple points, multiple resource pools, and multiple data centers, higher availability and fault escape capabilities are provided. However, the problem that comes with it is the consistency of distributed deployment. The solution to the consistency problem is often achieved through distributed consensus algorithms, such as Paxos and Raft. The core of these consensus algorithms is actually majority election. Election is a common problem in distributed system practice. By breaking the peer relationship between nodes, the selected leader (or master, coordinator) helps to achieve transaction atomicity and improve decision-making efficiency, thereby ensuring consistency. The quorum approach can achieve decision consistency in the case of network fragmentation and select a single leader to carry services in the leader election scenario.

[0003] The principle of majority is intuitive. If the total number of nodes is 2N+1, a resolution is passed if it is approved by at least N+1 nodes. In leader elections, in a fragmented network scenario, only the node with a majority of nodes can elect a leader, which prevents the emergence of multiple leaders. In summary, the core of the majority leader election algorithm is the principle of majority rule. The node with the most votes wins, avoiding the occurrence of multiple master-brain splits. Imagine using an even-numbered cluster. If two nodes each receive half of the votes, which one should be elected as the leader? The answer is that in this case, a leader cannot be elected and a new vote must be held. However, even with a new vote, there is a high probability that two nodes will have the same number of votes. Therefore, the majority leader election algorithm usually uses an odd number of nodes. However, in many real-world scenarios, applications are often deployed in two resource pools, and the two resource pools cannot evenly distribute an odd number of nodes. (If they are forcibly distributed across two resource pools, the entire service will become unavailable if the resource pool containing multiple nodes crashes.) Therefore, an additional third arbitration node is usually required to solve the majority leader election problem. However, the introduction of a third arbitration node greatly increases the complexity of failure scenarios (such as split-brain scenarios), the difficulty of operation and maintenance, and the complexity of recovery. Summary of the Invention

[0004] The purpose of this section is to summarize some aspects of the embodiments of the present invention and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the abstract and title of this application to avoid obscuring the purpose of this section, the abstract and the title of the invention, and such simplifications or omissions should not be used to limit the scope of the present invention.

[0005] In view of the above problems existing in the prior art, the present invention is proposed.

[0006] Therefore, the purpose of the present invention is to provide a two-tier arbitration method that effectively solves the problem of "the introduction of a third arbitration node greatly increases the complexity of failure scenarios (such as split-brain scenarios), the difficulty of operation and maintenance, and the complexity of recovery."

[0007] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0008] A dual-layer arbitration method for effectively resolving split-brain issues in two data centers includes the following steps:

[0009] S1: Offloads arbitration capabilities from the application layer to the arbitration cluster. Prioritizes access to arbitration nodes in the data center, registers services, and queries the address and services of the master node. For access to the three arbitration nodes, prioritizes the data center and configures remote backup access.

[0010] S2: Each data center has two arbitration nodes. One is a standard Follower node, which participates in leader election and configuration synchronization. The other is an Observer node, which performs only configuration synchronization. If the Leader / Follower node in this data center fails and becomes inaccessible, access to the Observer node is prioritized.

[0011] S3: When both arbitration nodes fail, for example, due to an arbitration node failure or a resource pool failure, after manual confirmation of the failure, the Observer node in this data center is quickly promoted to a standard Follower role, enabling rapid recovery of arbitration services and business recovery in the surviving data center.

[0012] As a preferred solution to the dual-layer arbitration method for effectively solving the brain split in two computer rooms described in the present invention, in step S2, when both nodes in the computer room are inaccessible, a heartbeat check of the application master node is started to ensure high availability within the single computer room and reduce problems caused by arbitration node failures.

[0013] As a preferred solution of the dual-layer arbitration method for effectively solving the brain split in two computer rooms described in the present invention, when the master node has no heartbeat, an attempt is made to seize the master through the arbitration node, thereby reducing the time and complexity of application master election.

[0014] As a preferred solution to the dual-layer arbitration method for effectively resolving split-brain issues in two computer rooms described in the present invention, the application nodes in each computer room preferentially access the arbitration node in their own computer room. When the arbitration node in their own computer room is unavailable, the application nodes read information from the shadow arbitration node in their own computer room. When all arbitration nodes in their own computer room are down, the application nodes access the peer computer room and the third arbitration node to obtain configuration information.

[0015] As a preferred solution to the dual-layer arbitration method for effectively solving the brain split in dual computer rooms described in the present invention, if the application node in the computer room is the master node and the arbitration cluster can access it normally, it will work normally and report the heartbeat regularly. If the arbitration access is different, the slave node is accessed from the configuration information obtained last time. If the master node can access it, it will work normally, otherwise it will be prohibited from writing.

[0016] As a preferred solution to the dual-layer arbitration method for effectively solving the brain split problem in dual computer rooms described in the present invention, if the application node in the computer room is a slave node, it will normally monitor the configuration information on the arbitration cluster. If all arbitration nodes have different access, they will try to access the master node from the configuration information obtained last time. If the master node can access it, it is normal. Otherwise, writing is prohibited. If the arbitration node access is normal and the master node registration information has expired, it will register itself as the master node and provide services.

[0017] As a preferred solution to the dual-layer arbitration method for effectively resolving a split-brain problem in two computer rooms described in the present invention, if the network between the two computer rooms is disconnected and both lose connection with the third arbitration node, the shadow arbitration node is quickly promoted to the follower role in the preferred computer room as needed, ensuring the normal operation of the arbitration cluster in the priority computer room and quickly restoring services in the priority computer room.

[0018] As a preferred solution of the dual-layer arbitration method for effectively solving the brain split in two computer rooms described in the present invention, the arbitration cluster is configured with a priority preemptive mode and a fair non-preemptive mode, and the service RT and continuity are guaranteed by supporting two switching back modes.

[0019] As a preferred solution to the dual-layer arbitration method for effectively solving the brain split problem in two computer rooms described in the present invention, three standard arbitration clusters are deployed in a containerized manner and have an 8C32G configuration. Each shadow arbitration node is configured as a 4C32G container, and all configuration data is stored in memory, improving the performance of the entire arbitration cluster.

[0020] As a preferred solution to the dual-computer room split-brain problem described in the present invention, the arbitration cluster prioritizes cluster configuration. Services in both computer rooms access the arbitration cluster through priority configuration, enabling master node preemption or slave node monitoring. The arbitration node cluster ensures information maintenance of the master node, including failover and failback logic according to the pre-set logic. In addition to the arbitration node failure scenario, the dual-computer room-level arbitration solution has been verified in typical computer room-level failure scenarios.

[0021] Beneficial effects of the present invention:

[0022] 1. Improve the high availability of arbitration capabilities and reduce the frequency of interruptions caused by arbitration cluster failures:

[0023] The general arbitration node selection is mostly to deploy a single arbitration node of a physical machine or virtual machine on the arbitration node, or to deploy a master-slave arbitration node (the standby node needs to temporarily pull up the arbitration service manually). The failure of a single-node arbitration node or the master-slave switch will lead to consistency risks (involving the heartbeat detection synchronization requirements of the expansion room). This solution entrusts the consistency of arbitration to the arbitration cluster. First, it reduces the negotiation time and complexity of application consistency. Second, it deploys a shadow arbitration node in each computer room (near real-time synchronization of the arbitration cluster configuration) to provide read-only capabilities. Therefore, even if the arbitration node in the computer room fails, there is no impact on the application level. In addition, when both arbitration nodes in the computer room fail, the application needs to query the heartbeat information from the opposite arbitration node or the service master node, which greatly reduces the complexity and latency of application consistency negotiation.

[0024] 2. Simplified fault scenarios:

[0025] In a dual-data center scenario, the introduction of arbitration nodes complicates the failure scenario and requires handling arbitration node failures. This solution significantly optimizes and improves several extreme scenarios.

[0026] 3. Solve the problem of business interruption caused by extreme failures:

[0027] The two-tiered arbitration solution in this solution can resolve service interruptions in extreme scenarios. In these extreme scenarios, shadow nodes can be upgraded in seconds by using the node role conversion commands or configuration files provided by the arbitration cluster, either through command lines or by refreshing and restarting shadow nodes. This allows for rapid restoration of the arbitration cluster's capabilities within a single data center, ensuring the business preemption logic within that data center and enabling rapid service recovery. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. Those skilled in the art can also derive other drawings based on these drawings without inventive effort. Among them:

[0029] Figure 1 A flowchart of the multi-node arbitration optimization architecture of the present invention;

[0030] Figure 2 A schematic diagram of the configuration of the arbitration node access priority of the present invention;

[0031] Figure 3 Schematic diagram of master selection configuration information within the arbitration node of the present invention;

[0032] Figure 4 This is a schematic diagram of verification of the scenario implementation of the present invention. DETAILED DESCRIPTION

[0033] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0034] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0035] Secondly, the term "one embodiment" or "embodiment" herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in various places throughout this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive of other embodiments.

[0036] Furthermore, the present invention is described in detail with reference to schematic diagrams. For ease of illustration, when describing the embodiments of the present invention, cross-sectional views illustrating device structures may be partially enlarged and not to scale. Furthermore, the schematic diagrams are merely illustrative and should not limit the scope of protection of the present invention. Furthermore, in actual production, the three-dimensional dimensions of length, width, and depth should be included.

[0037] Reference Figure 1-4 The present invention provides a dual-layer arbitration method for effectively solving the brain split problem in two computer rooms, comprising the following steps:

[0038] S1: Offloads arbitration capabilities from the application layer to the arbitration cluster. Prioritizes access to arbitration nodes in the data center, registers services, and queries the address and services of the master node. For access to the three arbitration nodes, prioritizes the data center and configures remote backup access.

[0039] S2: Each data center has two arbitration nodes. One is a standard Follower node, which participates in leader election and configuration synchronization. The other is an Observer node, which performs only configuration synchronization. When the Leader / Follower node in this data center fails and becomes inaccessible, access to the Observer node is prioritized. When both nodes in this data center become inaccessible, a heartbeat check is initiated on the application's master node to ensure high availability within the single data center and mitigate issues caused by arbitration node failures. When the master node loses its heartbeat, an attempt is made to seize the master position through the arbitration node, reducing the time and complexity of application master election.

[0040] S3: When both arbitration nodes fail, for example, due to an arbitration node failure or a resource pool failure, after manual confirmation of the failure, the Observer node in this data center is quickly promoted to a standard Follower role, enabling rapid recovery of arbitration services and business recovery in the surviving data center.

[0041] Application nodes in each data center prioritize access to the arbitration node in their own data center (for both reading and writing; if the node is not the primary node, the node forwards write requests). If the arbitration node in their own data center is unavailable, they can read information from the shadow arbitration node in their own data center. If all arbitration nodes in their own data center are down, they access the peer data center and the third arbitration node to obtain configuration information.

[0042] If the application node in this computer room is the master node and the arbitration cluster can access it normally, it will work normally and report heartbeats regularly (for lease renewal). If the arbitration cluster has different accesses, it will access the slave node from the configuration information obtained last time. If the master node can be accessed, it will work normally, otherwise it will be disabled.

[0043] If the application node in this computer room is a slave node, it will monitor the configuration information on the arbitration cluster as normal (according to access priority). If all arbitration nodes have different access, they will try to access the master node from the configuration information obtained last. If the master node can access it, then normal operation is normal; otherwise, write access is prohibited. If the arbitration node access is normal and the master node registration information has expired, it will register itself as the master node and provide services.

[0044] In extreme scenarios, if the network between the two data centers is disconnected and both lose connection with the third arbitration node, the shadow arbitration node can be quickly promoted to the follower role in the preferred data center as needed, ensuring the normal operation of the arbitration cluster in the priority data center and quickly restoring services in the priority data center.

[0045] Arbitration clusters can be configured in either prioritized preemptive mode or fair non-preemptive mode. In some scenarios, minimizing cross-data center access is crucial. Therefore, when the original master node recovers from a downtime, prioritized preemptive mode is required, forcing the master node back to its original data center. In other scenarios, cross-data center access is not important, and for business continuity, non-preemptive mode is supported. Upon recovery, the original master node can simply join the cluster as a slave node. By supporting both failover methods, service RT and continuity are maximized.

[0046] Three standard arbitration clusters are deployed in containers, with 8C32G configuration. Each shadow arbitration node can be configured as a 4C32G container. All configuration data is stored in memory to improve the performance of the entire arbitration cluster. The arbitration node cluster deployment form is as follows Figure 1 Deployed in the resource pool deployment mode.

[0047] The arbitration cluster prioritizes cluster configuration. Dual-data center services access the arbitration cluster based on priority configuration, enabling master node preemption or slave node monitoring. The arbitration node cluster ensures master node information maintenance, including failover and failback logic.

[0048] In addition to the arbitration node failure scenario, the two-tier arbitration solution was verified in typical data center-level failure scenarios (such as Figure 4 shown):

[0049] Scenarios 1-3, 6-8, 9-11: After a single resource pool, single link, or some dual-link scenarios fail, the overall service is not affected.

[0050] Scenarios 4-5 and 12-14: In isolated resource pools, or when one data center and the third quorum node are both down, a quick cluster operation and maintenance command is required in the remaining data centers to promote the shadow quorum node to a standard quorum node and add it to the quorum. This ensures that the quorum cluster can recover in seconds, and services in the surviving data center can recover in seconds.

[0051] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A dual-layer arbitration method that effectively solves the split-brain problem in two computer rooms, characterized in that: The following steps are involved: S1: Offloads arbitration capabilities from the application layer to the arbitration cluster. Prioritizes access to arbitration nodes in the data center, registers services, and queries the address and services of the master node. For access to the three arbitration nodes, prioritizes the data center and configures remote backup access. S2: Each data center has two arbitration nodes. One is a standard Follower node, which participates in leader election and configuration synchronization. The other is an Observer node, which performs only configuration synchronization. If the Leader / Follower node in this data center fails and becomes inaccessible, access to the Observer node is prioritized. S3: When both arbitration nodes fail, the Observer node in the data center is quickly promoted to a standard Follower role, enabling rapid recovery of arbitration services and business recovery in the surviving data center.

2. A dual-layer arbitration method for effectively resolving a split-brain problem in two computer rooms according to claim 1, characterized in that: In step S2, when both nodes in the computer room are inaccessible, a heartbeat check of the application master node is started to ensure high availability within the single computer room and reduce problems caused by arbitration node failures.

3. A dual-layer arbitration method for effectively resolving a split-brain problem in two computer rooms according to claim 2, characterized in that: When the master node has no heartbeat, the arbitration node attempts to seize the master position, reducing the time and complexity of application master election.

4. A dual-layer arbitration method for effectively resolving a split-brain problem in two computer rooms according to claim 1, characterized in that: The application nodes in each computer room give priority to accessing the arbitration node in the computer room. When the arbitration node in the computer room is unavailable, the application nodes read the information on the shadow arbitration node in the computer room. When all arbitration nodes in the computer room are down, the application nodes access the peer computer room and the third arbitration node to obtain configuration information.

5. A dual-layer arbitration method for effectively resolving split-brain issues in two computer rooms according to claim 1, characterized in that: If the application node in this computer room is the master node and the arbitration cluster can access it normally, it will work normally and report heartbeats regularly. If the arbitration access is different, the slave node will be accessed from the configuration information obtained last time. If the master node can be accessed, it will work normally, otherwise it will be prohibited from writing.

6. A dual-layer arbitration method for effectively resolving a split-brain problem in two computer rooms according to claim 1, characterized in that: If the application node in this computer room is a slave node, it will monitor the configuration information on the arbitration cluster normally. If all arbitration nodes have different access, they will try to access the master node from the configuration information obtained last time. If the master node can access it, it is normal. Otherwise, it is forbidden to write. If the arbitration node access is normal and the master node registration information expires, it will register itself as the master node and provide services.

7. A dual-layer arbitration method for effectively resolving split-brain issues in two computer rooms according to claim 1, characterized in that: If the network between the two computer rooms is disconnected and both lose connection with the third arbitration node, the shadow arbitration node in the preferred computer room can be quickly promoted to the follower role as needed to ensure the normal operation of the arbitration cluster in the priority computer room and quickly restore services in the priority computer room.

8. A dual-layer arbitration method for effectively resolving a split-brain problem in two computer rooms according to claim 1, characterized in that: The arbitration cluster is configured with a priority preemptive mode and a fair non-preemptive mode, and supports two switchback methods to ensure service RT and continuity.

9. A dual-layer arbitration method for effectively resolving a split-brain problem in two computer rooms according to claim 1, characterized in that: The three standard arbitration clusters are deployed in a containerized manner and configured with 8C32G. Each shadow arbitration node is configured as a 4C32G container. All configuration data is stored in memory, improving the performance of the entire arbitration cluster.

10. A dual-layer arbitration method for effectively resolving split-brain issues in two computer rooms according to claim 1, characterized in that: The arbitration cluster completes cluster configuration first. Dual-data center services access the arbitration cluster through priority configuration, enabling master node preemption or slave node monitoring. The arbitration node cluster ensures information maintenance of the master node, including failover and failback logic according to the configured logic. In addition to the arbitration node failure scenario, the two-tier arbitration solution is verified in typical data center-level failure scenarios.

Images