A distributed cluster and state synchronization method based on a raft consensus algorithm

Abstract

This application belongs to the field of distributed system technology, specifically disclosing a distributed cluster and state synchronization method based on the Raft consensus algorithm. The distributed cluster includes: a management node, multiple secondary clusters, and a primary cluster. Each secondary cluster consists of multiple nodes arranged according to the Raft consensus algorithm, with each secondary leader acting as the leader of its corresponding secondary cluster. Each primary cluster consists of secondary leaders arranged according to the Raft consensus algorithm, with each primary leader acting as the leader of its primary cluster. The management node manages the nodes in each level of the cluster and obtains the node state from any node within the cluster. This application, through the Raft consensus algorithm, decentralizes the node state synchronization process within the distributed cluster, allowing nodes to automatically negotiate and complete the state synchronization process. This enables efficient management of nodes in the distributed cluster, ensuring high availability and consistency of node state data.

Description

Technical Field

[0001] This application belongs to the field of distributed system technology, and more specifically, relates to a distributed cluster and state synchronization method based on the Raft consensus algorithm. Background Technology

[0002] In the era of the Internet of Things, cloud computing has become one of the major trends in the field of information technology, providing enterprises and individuals with powerful computing, storage, and application service capabilities. As the needs of enterprises and individuals continue to grow, traditional data processing and storage methods are gradually becoming inadequate. The emergence of data centers allows users to flexibly expand computing resources to adapt to ever-changing demands.

[0003] The core foundation of cloud computing is the data center, a collection of large-scale computing and storage resources. Through virtualization technology, computing, storage, and network resources are aggregated to meet the needs of different users and applications. Distributed cluster management technology is the foundation of cloud computing data centers. It manages servers within the data center as individual nodes, integrating them into a large distributed resource cluster to dynamically provide services to the outside world. To ensure the availability of external services, corresponding scheduling and management strategies are typically designed for the nodes in the cluster. Therefore, a management node is needed to monitor all nodes in the cluster and execute corresponding strategies based on changes in the cluster node status. Currently, there are two main methods for obtaining node status information: active polling by the management node and heartbeat detection.

[0004] Active polling. This method collects the status information of nodes in the cluster by periodically and actively polling the cluster through the management node. The biggest drawback of this method is that it requires continuous polling of each child node at regular intervals, resulting in significant system overhead. If the interval is too long, the cluster nodes have low sensitivity, and the obtained node status information may be outdated. If the interval is too short, a large number of socket connections need to be created repeatedly. When the number of nodes in the cluster is large, creating a corresponding socket connection for each node and constantly creating and deleting socket connections has a significant impact on the entire distributed system and may even affect system performance.

[0005] Heartbeat detection. While the result of heartbeat detection is the same as polling, the implementation differs. Heartbeat detection requires the management node to publish a specific communication interface, allowing cluster nodes to periodically report their operational status to the management node. Although this method reduces the pressure on the management node compared to polling, if one node crashes for any reason, the data will time out, requiring attempts to reconnect. The connection will only be closed after multiple unsuccessful attempts, a process that consumes time. This method becomes inefficient if many nodes crash. Furthermore, when the cluster has a large number of nodes, the periodic sending of heartbeat packets can put significant pressure on the network, leading to delayed or lost heartbeat packets and incorrect judgments about the cluster node status by the management node.

[0006] When the number of server nodes in a data center is large, whether the nodes in the cluster periodically report heartbeats to the management node or the management node periodically actively obtains the status of the nodes in the cluster, a large amount of network I / O will be generated in the entire system in a short period of time. This will put great pressure on the network throughput of the system and may even affect the performance of other aspects of the system.

[0007] In distributed systems, the CAP theorem emphasizes that a distributed system cannot simultaneously satisfy all three properties: Consistency, Availability, and Partition Tolerance; it can only satisfy at most two. Therefore, each distributed system, depending on its architecture, will tend towards CP, AP, or CA. However, for a distributed system, partition tolerance (P) must be guaranteed; otherwise, it violates the semantics of "distributed." Thus, distributed systems are generally classified into two types:

[0008] CP: Emphasizes the correctness of data in distributed systems, but due to the need to ensure strict consistency of data between different nodes in the cluster, the availability of the system may be sacrificed.

[0009] AP: Emphasizing system availability means that compromises must be made on data consistency.

[0010] For distributed systems, how to balance consistency and availability while ensuring partition tolerance is a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0011] In view of the shortcomings of the existing technology, the purpose of this application is to ensure both consistency and availability in distributed systems while guaranteeing partition tolerance.

[0012] To achieve the above objectives, in a first aspect, this application provides a distributed cluster based on the Raft consensus algorithm, comprising: a management node, multiple secondary clusters, and a primary cluster;

[0013] A secondary cluster consists of multiple nodes arranged according to the Raft consensus algorithm. Each node represents a server. The secondary leader is a node in the secondary cluster and acts as the leader of the corresponding secondary cluster. Nodes in the secondary cluster other than the secondary leader act as followers of the secondary leader in their respective clusters.

[0014] The primary cluster is composed of secondary leaders according to the Raft consensus algorithm. The primary leader is a node in the primary cluster and acts as the leader of the primary cluster. Nodes in the primary cluster other than the primary leader act as followers of the primary leader in their respective clusters.

[0015] The management node is used to manage the nodes in the cluster at each level and to obtain the node status from any node in the cluster.

[0016] Understandably, according to the requirements of the Raft consensus algorithm, all servers in the data center are aggregated to form a distributed cluster, with each server being a node in the cluster. According to the Raft algorithm, each node will play one of three roles: leader, follower, or candidate. However, to ensure the efficiency of real-time status acquisition for nodes in the cluster, this application further divides the leader role into primary leaders and secondary leaders, dividing the entire distributed cluster into multiple blocks. Each block consists of a subset of nodes in the cluster, with the secondary leaders acting as leaders within each block. Each secondary leader is responsible for the real-time status acquisition and synchronization of its respective subset of nodes. Then, a primary leader is elected from among all the secondary leaders in the cluster. The primary and secondary leaders then form a Raft cluster. The primary leader is responsible for receiving status update requests from management nodes outside the cluster and synchronizing these requests to the secondary leaders. This is the distributed cluster architecture of this application.

[0017] When a management node performs an operation on a node in the cluster, changing the node's running state, the secondary leader will detect the inconsistency between the node's current state and the state information stored in its own state machine from the heartbeats periodically sent by the followers, thus perceiving the change in the node's state. Subsequently, the secondary leader will send a state synchronization request to the primary leader, which will then execute a two-phase commit task to synchronize the node's running state.

[0018] In one possible implementation, the management node is used to send a status acquisition request to the primary leader to obtain the cluster's status information.

[0019] Understandably, the management node operates independently of the data center distributed cluster and is primarily responsible for monitoring, operating, and managing the nodes within the cluster. When the management node needs to obtain the running status of all nodes in the cluster, it needs to send a request to each node to retrieve the cluster state information stored in the state machine of any given node. Since only the first-level leader is the first node to actually apply the synchronization request to its state machine after the two-phase commit task of each state synchronization request is completed, in order to improve the real-time consistency of cluster state information, in this application, the management node needs to send a state retrieval request to the first-level leader to obtain the state information of the entire cluster.

[0020] In one possible implementation, the management node is used to send a cluster configuration file to the nodes in the cluster. The cluster configuration file indicates the cluster to which the node belongs and the nodes included in the cluster.

[0021] In one possible implementation, the first-level leader receives status update requests sent by the management node and synchronizes the requests to the second-level leader.

[0022] Secondly, this application also provides a distributed cluster state synchronization method based on the Raft consensus algorithm, applied to the distributed cluster described in the first aspect or any possible implementation of the first aspect, the method comprising:

[0023] The first-level leader receives client requests and writes the requests as new log entries to the write-ahead log.

[0024] The first-level leader sends the new log entry to the second-level leader;

[0025] The secondary leader sends the new log entry to the followers of the corresponding secondary cluster;

[0026] After receiving confirmation from a majority of followers in the corresponding secondary cluster, the secondary leader sends a confirmation message to the primary leader.

[0027] After receiving confirmation from a majority of second-level leaders, the first-level leader commits the new log entry to its own state machine and marks the new log entry as committed (and sends a committed message back to the client).

[0028] Leaders at all levels notify their followers to submit new log entries to their own state machines.

[0029] In one possible implementation, it also includes: during the initialization phase, the followers of the secondary cluster send their own follower state information to the secondary leader of their respective cluster to trigger the secondary leader to make a state change.

[0030] The secondary leader whose state change has been triggered sends a log synchronization request to the primary leader, and the log synchronization request carries the follower state information;

[0031] Leaders at all levels send log synchronization requests to followers;

[0032] After receiving confirmation from a majority of followers in the corresponding secondary cluster, the secondary leader sends a confirmation message to the primary leader.

[0033] After receiving confirmation from a majority of second-level leaders, the first-level leader submits the follower status information to its own state machine and marks the follower status information as submitted.

[0034] Leaders at all levels notify their followers to submit the followers' status information to their own state machine.

[0035] Understandably, each node's state machine should store the real-time operational status information of all nodes within the entire data center. During cluster initialization, the node state machines are empty because the status information has not yet been synchronized. During initialization, nodes in the cluster determine their own cluster and the other nodes within it based on the cluster configuration file. Nodes then elect a leader for their cluster according to the Raft algorithm. In the secondary cluster, follower nodes periodically send their operational status to the secondary leader via heartbeats. Upon receiving the follower's status information, the secondary leader compares it with the status information in its own state machine. Since all nodes' state machines are empty during initialization, receiving the follower's status information triggers a state change. The secondary leader generates a log synchronization request and sends it to the primary leader in the primary cluster. The primary leader then executes a two-phase commit process according to the Raft algorithm. In the first phase, upon receiving the synchronization request, the primary leader writes the request to its own write-ahead log and then sends the synchronization request to all followers in the cluster, i.e., all secondary nodes. Upon receiving a synchronization request from the primary leader, the secondary leader further distributes the request to all follower nodes in its secondary cluster. In the second phase, followers in the secondary cluster write the request to their own write-ahead logs and send a confirmation message to the secondary leader. If the secondary leader receives confirmation from a majority of followers, it writes the request to its own write-ahead log and then sends a confirmation message back to the primary leader. Once the primary leader receives confirmation from a majority of secondary leaders, it commits the log synchronization request and returns a success message to the secondary leader that triggered the request. This completes the primary leader's two-phase commit process. Furthermore, if a state update request fails to synchronize within the cluster, the primary leader sends a failure message to the secondary leader that triggered the request. Upon receiving the failure message, the secondary leader does not update the node's state and waits for the next follower heartbeat before initiating the state synchronization request again. This process repeats until initialization is complete, at which point the state machines of all nodes in the cluster store the real-time state information of all nodes within the cluster.

[0036] In one possible implementation, the secondary cluster also includes: in the event of a failure or deletion of the secondary leader, the secondary cluster elects a new secondary leader using the Raft consensus algorithm;

[0037] After receiving the heartbeat message from the primary leader, the new secondary leader sends a membership change request to the primary leader, requesting to join the primary cluster.

[0038] The first-level leader sends the member change request to the second-level leader;

[0039] After receiving confirmation from a majority of second-level leaders, a first-level leader changes its own member configuration information based on the member change request.

[0040] The first-level leader will synchronize the changed member configuration information to the followers (second-level leaders).

[0041] Understandably, when a secondary leader in a cluster fails and cannot provide service, the cluster will automatically elect a new secondary leader according to the Raft algorithm. After the new secondary leader takes office, it cannot join the primary cluster because it lacks member information from the primary leader and must wait for the primary leader to send a heartbeat to learn about the primary leader's information. Subsequently, the newly appointed secondary leader sends a member change request to the primary leader. Upon receiving the request, the primary leader, following the Raft algorithm, synchronizes the cluster configuration change request to all other secondary leaders. The other secondary leaders update their member configuration information and return confirmation messages to the primary leader. After receiving confirmation messages from a majority of secondary leaders, the primary leader updates its own member configuration information, completing the member change process.

[0042] One possible implementation also includes: in the event of a failure or deletion of the primary leader, the primary cluster elects a new primary leader using the Raft consensus algorithm.

[0043] Understandably, when a primary leader in a cluster fails and cannot provide service, its primary cluster will automatically elect a new primary leader according to the Raft algorithm. Furthermore, the primary leader also acts as the secondary leader in its secondary cluster; when it fails, its secondary cluster will also automatically elect a new secondary leader according to the Raft algorithm. Once the new secondary leader takes office, it will send a membership change request to the newly appointed primary leader, completing the membership change process as described above.

[0044] One possible implementation also includes: if the secondary leader does not receive the follower's status heartbeat information within a certain period of time, it determines that the follower has failed and updates the member configuration information;

[0045] Send the updated member configuration information to the followers in the corresponding secondary cluster.

[0046] Understandably, when a member changes in a distributed cluster, such as when a node is deleted, if the leader of the cluster where the deleted node is located does not receive the status heartbeat information of the deleted node within a certain period of time, it assumes that the node has failed and sends a member change request. Following the Raft algorithm process, it synchronizes the changed member configuration information to the followers in the cluster. After receiving confirmation information from a majority of nodes, the leader updates its own member configuration information and completes the member change process.

[0047] In one possible implementation, the following is also included: when a new node is added, the management node sends a membership change request to the secondary leader of the secondary cluster to which the new node belongs, and the request contains information about the newly added node;

[0048] The secondary leader sends a member change request to the followers in its secondary cluster;

[0049] After receiving confirmation from a majority of followers, the secondary leader changes its own member configuration information based on the member change request.

[0050] The second-level leader will synchronize the changed member configuration information to the followers;

[0051] The secondary leader synchronizes its state machine data and write-ahead logs to the new node.

[0052] Understandably, when adding a new node to a distributed cluster, the system administrator first needs to decide which cluster the new node should join. After determining the cluster, the system administrator sends a member change request to the leader in the cluster. The leader then synchronizes the changed member configuration information to other followers in the cluster according to the Raft algorithm, completing the cluster member change. Since the new node does not save write-ahead logs and has no data in its state machine, the secondary leader needs to copy the data saved in its state machine to the newly joined node, and also copy the log content in its own write-ahead log that has not yet been applied to the state machine to the new node. At this point, the data state of the new node is synchronized with the secondary leader. Then, when the secondary leader sends a heartbeat message, the unapplied write-ahead logs are applied to the state machine.

[0053] Overall, the technical solutions conceived in this application have the following beneficial effects compared with the prior art:

[0054] (1) The Raft consensus algorithm can decentralize the process of node state synchronization in the distributed cluster to the distributed cluster, allowing nodes in the cluster to automatically negotiate and complete the state synchronization process, thereby freeing up the cluster management node and allowing the management node to focus on resource management and scheduling, thus improving the overall performance of the distributed system; Based on the Raft consensus algorithm, nodes in the distributed cluster are managed efficiently, ensuring the high availability of the distributed cluster and the consistency of node state data.

[0055] (2) A scheme for constructing a hierarchical Raft distributed cluster is proposed, which logically divides the distributed cluster into two levels. Through this hierarchical approach, when the number of nodes in the distributed cluster is large, the bandwidth and performance pressure during synchronization within the Raft cluster can be effectively alleviated, the network overhead of the cluster leader can be saved, and the efficiency of the entire cluster state synchronization can be improved. Attached Figure Description

[0056] Figure 1 This is a diagram of the Raft distributed cluster system architecture provided in the embodiments of this application;

[0057] Figure 2 This is a flowchart of the cluster response client request provided in an embodiment of this application;

[0058] Figure 3 This is a flowchart of the cluster initialization process provided in an embodiment of this application;

[0059] Figure 4 This is a flowchart of the secondary leader fault handling process provided in the embodiments of this application;

[0060] Figure 5 This is a flowchart of the first-level leader fault handling process provided in the embodiments of this application;

[0061] Figure 6 This is a flowchart of the node deletion process provided in an embodiment of this application;

[0062] Figure 7 This is a flowchart of the newly added node provided in the embodiments of this application. Detailed Implementation

[0063] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0064] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0065] In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, for example, multiple processing units means two or more processing units, multiple elements means two or more elements, etc.

[0066] The embodiments of this application are described below with reference to the accompanying drawings.

[0067] Figure 1 This is a diagram of the Raft distributed cluster system architecture provided in the embodiments of this application, such as... Figure 1 As shown, this distributed cluster includes: a management node, multiple secondary clusters, and one primary cluster;

[0068] A secondary cluster consists of multiple nodes arranged according to the Raft consensus algorithm. Each node represents a server. The secondary leader is a node in the secondary cluster and acts as the leader of the corresponding secondary cluster. Nodes in the secondary cluster other than the secondary leader act as followers of the secondary leader in their respective clusters.

[0069] The primary cluster is composed of secondary leaders according to the Raft consensus algorithm. The primary leader is a node in the primary cluster and acts as the leader of the primary cluster. Nodes in the primary cluster other than the primary leader act as followers of the primary leader in their respective clusters.

[0070] The management node is used to manage the nodes in the cluster at each level and to obtain the node status from any node in the cluster.

[0071] When evaluating a system design, the evaluation criteria not only reflect its strengths but, more importantly, its lower bound. In the CAP theorem, emphasizing consistency (C) sacrifices availability (A), while prioritizing availability violates consistency (C), inevitably leading to either becoming the weakest link in the system. In reality, consistency (C) and availability (A) are not absolutely opposed; there is room for compromise. The key is to find a suitable balance that improves the overall lower bound of the distributed system. The Raft consensus algorithm is one such effective algorithm.

[0072] The Raft consensus algorithm is actually based on various algorithmic mechanisms that can improve availability A to the highest possible level while sacrificing consistency C as little as possible.

[0073] In terms of availability (A), the Raft algorithm guarantees that the system is stable and available when more than half of the nodes in the distributed system are alive. At the same time, the request time depends on the lower limit of the majority of nodes, rather than the lower limit of all nodes.

[0074] Regarding consistency (C), the standard Raft algorithm can guarantee eventual consistency. In various engineering implementations, the Raft algorithm can be slightly modified to ensure real-time consistency, thereby further enhancing consistency (C).

[0075] This application, based on the Raft consensus algorithm, establishes a distributed server cluster within a data center. Each server in the cluster synchronizes its operational state according to the Raft algorithm. The Raft algorithm guarantees eventual consistency of the state data of each node in the cluster, allowing each node to store the operational state data of the entire system, ultimately achieving consensus on the operational state of all servers in the cluster. This decentralizes the node state synchronization process within the distributed cluster, eliminating reliance on an external management node. The management node only needs to periodically retrieve the state of all nodes in the cluster from any node, significantly reducing network I / O during state synchronization and alleviating the pressure on the management node. Furthermore, this application employs a hierarchical division of the Raft cluster, reducing the number of nodes in each level. When the number of nodes in the distributed cluster is large, this accelerates the data synchronization rate within each level, thereby improving the overall system performance and stability.

[0076] Therefore, the Raft consensus algorithm can decentralize the node state synchronization process within the distributed cluster, allowing nodes to automatically negotiate and complete the state synchronization process. This frees up the cluster management node, enabling it to focus on resource management and scheduling, thus improving the overall performance of the distributed system. Based on the Raft consensus algorithm, nodes in the distributed cluster can be managed efficiently, ensuring high availability and consistency of node state data. Furthermore, this application proposes a scheme for constructing a hierarchical Raft distributed cluster, logically dividing the distributed cluster into two levels. This hierarchical approach effectively alleviates the bandwidth and performance pressure during synchronization within the Raft cluster when the number of nodes in the distributed cluster is large, saving network overhead for the cluster leader and thus improving the efficiency of the entire cluster state synchronization.

[0077] Figure 2 This is a flowchart of the cluster response client request provided in an embodiment of this application, such as... Figure 2 As shown, this application provides a distributed cluster state synchronization method based on the Raft consensus algorithm, which is applied to the above-mentioned distributed cluster based on the Raft consensus algorithm. The method includes the following steps S101 to S106.

[0078] Step S101: The first-level leader receives the client request and writes the request as a new log entry into the write-ahead log.

[0079] Step S102: The first-level leader sends the new log entry to the second-level leader;

[0080] Step S103: The secondary leader sends the new log entry to the followers of the corresponding secondary cluster;

[0081] Step S104: After receiving confirmation from a majority of followers in the corresponding secondary cluster, the secondary leader sends a confirmation message to the primary leader.

[0082] Step S105: After receiving confirmation from a majority of second-level leaders, the first-level leader submits the new log entry to its own state machine and marks the new log entry as committed (and sends a committed message to the client).

[0083] In step S106, leaders at all levels notify followers to submit new log entries to their own state machines.

[0084] Figure 3 This is a flowchart of the cluster initialization process provided in an embodiment of this application, such as... Figure 3 As shown, during the initialization phase, the write-ahead logs and state machine data of all nodes are empty, and the followers in the secondary cluster send their own state information to the secondary leader in the form of heartbeats.

[0085] After receiving the status information from the followers, the secondary leader triggers a status change request, generates a log synchronization request, and sends the log synchronization request to the primary leader in the primary cluster.

[0086] After receiving a synchronization request, the first-level leader writes the request to the write-ahead log file and then sends the synchronization request to all second-level leaders.

[0087] After receiving the synchronization request, the secondary leader further distributes the request to all followers in the secondary cluster.

[0088] After receiving a synchronization request, the follower writes the request to the write-ahead log file and returns an ACK message to the secondary leader.

[0089] If a secondary leader receives a response to the consent (confirmation) request from a majority of its followers, it writes the request to a write-ahead log file and sends a response to the consent (confirmation) request to the primary leader; otherwise, it sends a response to the dissenting request to the primary leader.

[0090] If the primary leader receives a response from a majority of secondary leaders agreeing to the request, it returns a success message to the secondary leader that triggered the synchronization request; otherwise, it returns a failure message, and the two-phase commit process ends.

[0091] If the request is successful, the first-level leader submits the state information to (applies to) its own state machine; otherwise, it does not update the state machine data.

[0092] When a Level 1 leader sends a heartbeat message to a Level 2 leader, it carries the latest committed write-ahead log index. After receiving the heartbeat message, the Level 2 leader commits (applies) the write-ahead log to its respective state machine based on the index information.

[0093] The second-level leader sends a heartbeat message to the followers, carrying the latest committed write-ahead log index. After receiving the heartbeat information, the followers commit (apply) the write-ahead log to their respective state machines according to the index information, and one state update process is completed.

[0094] Repeat the above process until the initialization process is complete. At this point, the state machines of all nodes in the cluster have stored the real-time state information of all nodes in the cluster.

[0095] Figure 4 This is a flowchart of the secondary leader fault handling process provided in the embodiments of this application, such as... Figure 4 As shown, when the secondary leader in a sub-cluster (secondary cluster) fails and cannot provide services, the nodes in its cluster automatically begin a new secondary leader election according to the Raft algorithm process.

[0096] After a new secondary leader takes office, it does not yet have member information in the primary cluster and cannot join the primary cluster. It must wait to receive heartbeat information sent by the primary leader.

[0097] Upon receiving the heartbeat information from the primary leader, the newly appointed secondary leader sends a membership change request to the primary leader, requesting to join the primary cluster.

[0098] After receiving a member change request, the first-level leader writes the request to the write-ahead log and sends the member change request to all second-level leaders.

[0099] After receiving the member's change request, other secondary leaders will write the request to the write-ahead log and reply with a confirmation message to the primary leader.

[0100] Once the first-level leader receives confirmation messages from most second-level leaders, it submits the write-ahead log and updates its own member configuration information.

[0101] Once most secondary leaders receive the heartbeat information from the primary leaders and update their respective member configuration information, the new secondary leader joins the primary cluster.

[0102] Figure 5 This is a flowchart of the first-level leader fault handling process provided in the embodiments of this application, such as... Figure 5 As shown, when a primary leader in a cluster fails and is unable to provide services, its primary cluster will automatically begin a new primary leader election according to the Raft algorithm process.

[0103] Once a new Level 1 leader takes office, it begins receiving requests from management nodes and other Level 2 leaders.

[0104] In addition, the secondary cluster to which the failed primary leader belongs will also begin electing a new secondary leader according to the Raft algorithm process.

[0105] Once a new Level 2 leader takes office, they will send a member change request to the new Level 1 leader following the steps described above to update the member configuration information.

[0106] Once all nodes in the primary cluster have updated their member configuration information, the fault handling process ends.

[0107] Figure 6 This is a flowchart of the node deletion process provided in an embodiment of this application, such as... Figure 6 As shown, if the node to be deleted is a regular follower node, and the secondary leader in its secondary cluster does not receive the node's status heartbeat information within a certain period of time, it considers the node to be invalid. Therefore, it will update the node's status in its own state machine and update the member configuration information. Then, it will send the updated member configuration information to other followers in the cluster to complete the node deletion process.

[0108] If the node to be deleted is a secondary leader, the process is the same as for secondary leader failures described above.

[0109] If the node to be deleted is a first-level leader, the process is the same as for first-level leader failures described above.

[0110] Figure 7 This is a flowchart of the newly added node provided in the embodiments of this application, such as... Figure 7 As shown, when adding a new node to a cluster, the system administrator first needs to decide which secondary cluster the new node should be added to.

[0111] Once the cluster is established, the system administrator sends a membership change request to the secondary leader of the cluster, which includes information about the newly added nodes.

[0112] The leader synchronizes the changed member configuration information to other followers in the cluster according to the request process of the Raft consensus algorithm, thus completing the cluster member change.

[0113] Since the newly joined node has not yet saved the write-ahead log and there is no data in its state machine, the secondary leader needs to copy the data in its own state machine to the newly joined node, and also copy the write-ahead log content that has not yet been applied to the state machine to the newly joined node.

[0114] Afterwards, when a newly added node waits for the secondary leader to send a heartbeat message, it writes the unapplied (uncommitted) write-ahead log into its own state machine, thus completing the process of adding a node.

[0115] The method steps in this application embodiment can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can reside in an ASIC.

[0116] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0117] It is understood that the various numerical designations used in the embodiments of this application are merely for the convenience of description and are not intended to limit the scope of the embodiments of this application.

[0118] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A distributed cluster based on the Raft consensus algorithm, characterized in that, include: Management node, multiple secondary clusters and one primary cluster; A secondary cluster consists of multiple nodes arranged according to the Raft consensus algorithm. Each node represents a server. The secondary leader is a node in the secondary cluster and acts as the leader of the corresponding secondary cluster. Nodes in the secondary cluster other than the secondary leader act as followers of the secondary leader in their respective clusters. The primary cluster is composed of secondary leaders according to the Raft consensus algorithm. The primary leader is a node in the primary cluster and acts as the leader of the primary cluster. Nodes in the primary cluster other than the primary leader act as followers of the primary leader in their respective clusters. The management node is used to manage the nodes in the cluster at each level and to obtain the node status from any node in the cluster; The distributed cluster is used for: The first-level leader receives client requests and writes the requests as new log entries to the write-ahead log. The first-level leader sends the new log entry to the second-level leader; The secondary leader sends the new log entry to the followers of the corresponding secondary cluster; After receiving confirmation from a majority of followers in the corresponding secondary cluster, the secondary leader sends a confirmation message to the primary leader. After receiving confirmation from a majority of the second-level leaders, the first-level leader commits the new log entry to its own state machine and marks the new log entry as committed; Leaders at all levels instruct their followers to submit new log entries to their own state machines; The distributed cluster is also used for: During the initialization phase, the followers of the secondary cluster send their follower status information to the secondary leader of their respective cluster to trigger the secondary leader to make a state change. The secondary leader whose state change has been triggered sends a log synchronization request to the primary leader, and the log synchronization request carries the follower state information; Leaders at all levels send log synchronization requests to followers; After receiving confirmation from a majority of followers in the corresponding secondary cluster, the secondary leader sends a confirmation message to the primary leader. After receiving confirmation from a majority of second-level leaders, the first-level leader submits the follower status information to its own state machine and marks the follower status information as submitted. Leaders at all levels instruct their followers to submit the followers' status information to their own state machines. The distributed cluster is also used for: If the secondary leader discovers from the heartbeats sent by the followers that the current state of the node is inconsistent with the state information stored in its own state machine, the secondary leader sends a state synchronization request to the primary leader.

2. The distributed cluster based on the Raft consensus algorithm according to claim 1, characterized in that, The management node is used to send a status retrieval request to the primary leader in order to obtain the cluster's status information.

3. The distributed cluster based on the Raft consensus algorithm according to claim 1, characterized in that, The management node is used to send cluster configuration files to the nodes in the cluster. The cluster configuration files indicate the cluster to which the node belongs and the nodes included in the cluster.

4. The distributed cluster based on the Raft consensus algorithm according to claim 1, characterized in that, The primary leader is used to receive status update requests sent by the management node and synchronize the requests to the secondary leader.

5. A distributed cluster state synchronization method based on the Raft consensus algorithm, characterized in that, Applied to a distributed cluster based on the Raft consensus algorithm as described in any one of claims 1-4, the method includes: The first-level leader receives client requests and writes the requests as new log entries to the write-ahead log. The first-level leader sends the new log entry to the second-level leader; The secondary leader sends the new log entry to the followers of the corresponding secondary cluster; After receiving confirmation from a majority of followers in the corresponding secondary cluster, the secondary leader sends a confirmation message to the primary leader. After receiving confirmation from a majority of the second-level leaders, the first-level leader commits the new log entry to its own state machine and marks the new log entry as committed; Leaders at all levels instruct their followers to submit new log entries to their own state machines; Also includes: During the initialization phase, the followers of the secondary cluster send their follower status information to the secondary leader of their respective cluster to trigger the secondary leader to make a state change. The secondary leader whose state change has been triggered sends a log synchronization request to the primary leader, and the log synchronization request carries the follower state information; Leaders at all levels send log synchronization requests to followers; After receiving confirmation from a majority of followers in the corresponding secondary cluster, the secondary leader sends a confirmation message to the primary leader. After receiving confirmation from a majority of second-level leaders, the first-level leader submits the follower status information to its own state machine and marks the follower status information as submitted. Leaders at all levels instruct their followers to submit the followers' status information to their own state machines. Also includes: If the secondary leader discovers from the heartbeats sent by the followers that the current state of the node is inconsistent with the state information stored in its own state machine, the secondary leader sends a state synchronization request to the primary leader.

6. The distributed cluster state synchronization method based on the Raft consensus algorithm according to claim 5, characterized in that, Also includes: In the event of a failure or deletion of the secondary leader, the secondary cluster elects a new secondary leader using the Raft consensus algorithm; After receiving the heartbeat message from the primary leader, the new secondary leader sends a membership change request to the primary leader, requesting to join the primary cluster. The first-level leader sends the member change request to the second-level leader; After receiving confirmation from a majority of second-level leaders, a first-level leader changes its own member configuration information based on the member change request. The first-level leader will synchronize the changed member configuration information to the followers.

7. The distributed cluster state synchronization method based on the Raft consensus algorithm according to claim 6, characterized in that, Also includes: In the event of a failure or deletion of the primary leader, the primary cluster elects a new primary leader using the Raft consensus algorithm.

8. The distributed cluster state synchronization method based on the Raft consensus algorithm according to claim 5, characterized in that, Also includes: If a secondary leader does not receive status heartbeat information from a follower within a certain period of time, it determines that the follower has become invalid and updates the member configuration information. Send the updated member configuration information to the followers in the corresponding secondary cluster.

9. The distributed cluster state synchronization method based on the Raft consensus algorithm according to claim 5, characterized in that, Also includes: When a new node is added, the management node sends a membership change request to the secondary leader of the secondary cluster to which the new node belongs. The request includes information about the newly added node. The secondary leader sends a member change request to the followers in its secondary cluster; After receiving confirmation from a majority of followers, the secondary leader changes its own member configuration information based on the member change request. The second-level leader will synchronize the changed member configuration information to the followers; The secondary leader synchronizes its state machine data and write-ahead logs to the new node.

Images