A method and device for large-scale data consistency based on a multi-level tree structure cluster.

Abstract

This invention discloses a large-scale data consistency method and device based on a multi-level tree-structured cluster. The method includes constructing a multi-level tree-structured cluster, assigning an i-level cluster as its corresponding subtree to each follower in the i-1 level cluster, with the leader in the i-level cluster acting as a follower in the i-1 level cluster; the i-level leader triggering its own data collection based on a heartbeat detection message and feeding back the collected data to the i-1 level leader; the i-1 level leader performing a consistency comparison on the collected data, and if inconsistent, constructing consistent data and sending the synchronization log of the constructed data to the i-level leader, which then verifies and performs log synchronization; and finally, the i-level leader sending log replication messages to the i-level followers in its subtree, which also verify and perform log synchronization. This invention exhibits low election performance overhead and is adaptable to large-scale distributed systems.

Description

Technical Field

[0001] This invention relates to the field of distributed systems technology, and in particular to a method and device for large-scale data consistency based on a multi-level tree structure cluster. Background Technology

[0002] With the rapid development of cloud computing and big data technologies, enterprises and users are increasingly demanding higher data processing capabilities and system availability. Against this backdrop, distributed systems and microservice architectures have become mainstream technologies. However, data consistency and high availability in distributed systems (especially large-scale systems) remain critical issues that urgently need to be addressed.

[0003] Traditional distributed systems typically employ master-slave replication or distributed consensus algorithms (such as Paxos and Raft) to ensure data consistency. Among these, the Raft algorithm has gained widespread attention due to its simplicity and ease of understanding. Raft ensures data consistency among multiple nodes in a distributed system through leader election, log replication, and security mechanisms.

[0004] However, current consensus algorithms still have some shortcomings, taking the Raft algorithm as an example:

[0005] Firstly, the election process may cause performance issues: The election process of the Raft algorithm may cause performance issues because when a network partition or node crash occurs, a new leader needs to be re-elected, which not only affects the availability of the system, but also leads to data inconsistency problems.

[0006] Secondly, it is not suitable for large-scale distributed systems: In large-scale distributed systems, the Raft algorithm's election process involves more nodes, and each election requires more message passing and computational resources. The election process may lead to excessive network load, which will reduce the overall performance of the system. Simultaneously, the leader needs to send heartbeat and log replication messages to a large number of followers. As the number of nodes increases, network traffic also increases significantly, potentially leading to network congestion and performance degradation. If the bandwidth of a large-scale distributed system is insufficient, message transmission speed will slow down, affecting the system's throughput. Therefore, it is not suitable for large-scale distributed systems.

[0007] Therefore, how to provide a large-scale data consistency method and device based on a multi-level tree structure cluster that is suitable for distributed systems and has high availability is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0008] In response to the aforementioned research status and existing problems, this invention provides a large-scale data consistency method and device based on a multi-level tree structure cluster, which has low election performance loss and the ability to adapt to large-scale distributed systems.

[0009] This invention provides a large-scale data consistency method based on a multi-level tree structure cluster, comprising the following steps:

[0010] Construct a multi-level tree-structured cluster, which includes: a first-level cluster and an i-level cluster, where i∈[2,N] and N is a positive integer greater than 2; the consensus node states in each cluster include: leader, follower, and candidate; assign an i-level cluster as its corresponding subtree to the follower in the i-1 level cluster, and the leader in the i-level cluster is the follower in the i-1 level cluster.

[0011] The i-1 level leader sends a heartbeat detection message to the i-level leader in its corresponding subtree based on the polling heartbeat duration; the i-level leader triggers its own data collection based on the heartbeat detection message and feeds back the collected data to the i-1 level leader.

[0012] The i-1 level leader performs a consistency comparison on the collected data. If there is no consistency, a consistent data structure is constructed, and the synchronization log of the constructed data is sent to the i-level leader. The i-level leader verifies the data and then performs log synchronization.

[0013] The i-th level leader sends a log replication message to the i-th level followers in its subtree, and the i-th level followers perform log synchronization after verification.

[0014] Preferably, the steps for constructing a multi-level tree-structured cluster include an election step:

[0015] The current-level follower is triggered to conduct a leader election when the current-level candidate does not receive a heartbeat detection message from the current-level leader.

[0016] Once a candidate of level i-1 becomes the leader of level i-1 through a voting mechanism, the remaining level i-1 consensus nodes update their consensus node status to follower. The level i-1 leader then sends the level i-1 election result notification to the level i leaders in its corresponding subtree. The level i leader then sends the level i-1 election result notification to the level i followers in its subtree. The level i followers continue to pass on the level i-1 election result notification downwards until the consensus node has no corresponding subtree.

[0017] Once a candidate of level i becomes the leader of level i through a voting mechanism, the remaining level i consensus nodes update their consensus node status to follower. The level i leader sends the level i election result notification to the corresponding level i-1 leader in its subtree. The level i leader continues to pass the level i election result notification upwards until the level i leader is reached. The level i-1 leader then sends the level i election result notification to the remaining level i-1 followers. The remaining level i-1 followers continue to pass the level i election result notification downwards until the consensus node has no corresponding subtree.

[0018] The consensus node that receives the election result notification adjusts its node information accordingly.

[0019] Preferably, the voting mechanism includes:

[0020] The current level candidate sends an election request message to other consensus nodes in the current level;

[0021] The current and other consensus nodes verify the status information of the current and candidates, and after successful verification, send voting information to the current-level candidate.

[0022] The current-level candidate accumulates the voting information. When the number of voting information meets the minimum number of consensus nodes, the current-level candidate updates its status to leader, and the remaining current-level consensus nodes update their status to follower.

[0023] Preferably, the election step further includes:

[0024] If the i-1 level leader still does not receive the i-level election result notification from the i-level leader in its corresponding subtree after a specified number of polling heartbeat durations, then the following steps are performed:

[0025] The i-1 level leader sends a liveness detection message to all consensus nodes in its corresponding subtree, and designates one consensus node as the i-level leader of its corresponding subtree from among the consensus nodes whose liveness verification is successful.

[0026] The designated consensus node updates its state to level i leader, sends the level i election result notification to the other consensus nodes in the subtree, and returns an update success message to the level i-1 leader; the other consensus nodes update their state to level i follower.

[0027] The i-1 level leader adjusts the node information based on the update success information, and the i-1 level leader continues to pass the update success information upwards until the first-level leader; and sends the i-1 level election result notification to the remaining i-1 level followers; the remaining i-1 level followers continue to pass the i-1 level election result notification downwards until the consensus node has no corresponding subtree;

[0028] The consensus node that receives the i-th level election result will notify and adjust its node information according to the i-th level election result.

[0029] Preferably, the step of designating a consensus node as the i-th level leader of its corresponding subtree from among the consensus nodes that have successfully passed liveness verification includes any one of the following:

[0030] Randomly assign: Select one consensus node from the consensus nodes that have successfully verified liveness as the i-th level leader of its corresponding subtree;

[0031] The comparison log specifies that the log status of the consensus nodes that have successfully verified their liveness is compared, and the consensus node with the latest log update time is designated as the i-th level leader of its corresponding subtree.

[0032] Preferably, the steps for building a multi-level tree-structured cluster include a log replication step:

[0033] The i-level leader receives the log replication message sent by the i-1 level leader, verifies it, performs log synchronization, and sends a log replication message to the i-level follower. The i-level follower verifies it, performs log synchronization, and replies with an i-level response message to the i-level leader.

[0034] The i-th level leader accumulates the i-th level response messages. When the number of i-th level response messages meets the minimum number of consensus nodes, the i-th level leader replies with an i-1 level response message to the i-1 level leader.

[0035] The i-1 level leader accumulates the i-1 level response messages. When the number of i-1 level response messages meets the minimum number of consensus nodes, the i-1 level leader updates the client request corresponding to the log to the state machine for execution.

[0036] Preferably, the minimum number of nodes is required to be greater than half the number of consensus nodes in the current level cluster.

[0037] Preferably, the step of sending the heartbeat detection message includes:

[0038] The i-level leader receives heartbeat detection messages sent by the i-1 level leader based on the polling heartbeat duration; the i-level leader sends heartbeat detection messages to the i-level followers based on the polling heartbeat duration.

[0039] If a consensus node does not receive a heartbeat detection message within the polling heartbeat duration, it triggers the generation of a candidate from the consensus nodes of the current level cluster.

[0040] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the large-scale data consistency method based on a multi-level tree structure cluster.

[0041] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the large-scale data consistency method based on a multi-level tree structure cluster.

[0042] This invention proposes a large-scale data consistency cluster scheme based on a multi-level tree structure governance, aiming to solve the performance and scalability problems faced by the traditional Raft algorithm in large-scale distributed systems. By transforming the original single-level Raft model into a multi-level cluster structure, it has the following advantages compared to existing technologies:

[0043] Adaptable to Large-Scale Distributed Systems: This invention expands the cluster size to powers of the previous limit through a multi-level tree structure. This scalability allows the invention to be applied to larger-scale distributed systems, meeting enterprise needs for high availability and large-scale data processing. This invention facilitates load balancing; in the multi-level structure, log replication and heartbeat message transmission are distributed across multiple leader nodes, rather than concentrated on a single node. This effectively distributes network load and improves the overall performance and stability of the system.

[0044] Improving data consistency: This invention ensures data consistency across multiple levels of the cluster through a multi-level log replication process. The i-1 level leader distributes log entries to the i-level leader, which then distributes them to its subordinate followers. This hierarchical synchronization mechanism not only improves the reliability of data transmission but also reduces the burden of individual network requests.

[0045] Enhanced system availability: This invention improves the system's partition tolerance through a multi-level structure and hierarchical election mechanism. Even in the event of network partitioning, each sub-cluster can still operate independently, ensuring that some functions of the system remain unaffected.

[0046] Reduced Election Performance Loss: This invention employs a hierarchical election mechanism, dividing the election process into multiple levels. For example, the first-level election is responsible for selecting the leader of the first-level cluster, while the second-level election selects a leader within each second-level cluster. This hierarchical election mechanism reduces the complexity of elections and network load in large-scale clusters, avoiding performance degradation caused by frequent elections. Rapid Fault Recovery: When a node in the i-th level cluster fails, the i-1-th level leader can quickly detect and appoint a new i-th level leader, ensuring the system resumes normal operation in a short time, improving system availability and response speed. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely embodiments of the present invention, and those skilled in the art can obtain other drawings based on the provided drawings without creative effort.

[0048] Figure 1 This is a structural principle diagram of a multi-level tree-structured cluster provided in an embodiment of the present invention;

[0049] Figure 2 A flowchart illustrating a large-scale data consistency method based on a multi-level tree-structured cluster, provided for embodiments of the present invention;

[0050] Figure 3 A flowchart of the first-level election process for a large-scale data consistency method based on a multi-level tree structure cluster, provided in an embodiment of the present invention;

[0051] Figure 4 A flowchart of a two-level election process for a large-scale data consistency method based on a multi-level tree structure cluster, provided in an embodiment of the present invention;

[0052] Figure 5 This is a flowchart illustrating the second-level election failure handling process of a large-scale data consistency method based on a multi-level tree structure cluster, provided as an embodiment of the present invention. Detailed Implementation

[0053] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0054] The first aspect of this invention discloses a large-scale data consistency method based on a multi-level tree structure cluster, comprising the following steps:

[0055] A multi-level tree-structured cluster is constructed, comprising: a first-level cluster and an i-level cluster, where i ∈ [2, N], and N is a positive integer greater than 2. The consensus node states in each cluster include: leader, follower, and candidate. Each follower in the i-1 level cluster is assigned an i-level cluster as its corresponding subtree, and the leader in the i-level cluster is a follower in the i-1 level cluster. The i-1 level leader sends a heartbeat detection message to the i-level leader in its corresponding subtree based on the polling heartbeat duration. The i-level leader triggers its own data collection based on the heartbeat detection message and feeds back the collected data to the i-1 level leader. The i-1 level leader performs a consistency comparison on the collected data; if inconsistent, it constructs consistent data and sends the synchronization log of the constructed data to the i-level leader. The i-level leader verifies the data and then performs log synchronization. The i-level leader sends a log replication message to the i-level followers in its subtree, and the i-level followers verify the data and then perform log synchronization.

[0056] like Figure 1 As shown, an example is given based on a two-level tree structure. Multi-level logic is extended using a two-level structure. In the diagram, i takes the value 2.

[0057] like Figure 2 As shown, after each follower receives a committed log entry, it applies it to its own replicated state machine to ensure that the states of all nodes are consistent.

[0058] This embodiment uses the Raft algorithm as an example for illustration:

[0059] Raft is a consistency algorithm for managing replicated logs. The core concepts of the Raft algorithm include:

[0060] Leader: In the Raft algorithm, the leader is responsible for handling tasks such as client requests, log replication, and member changes. There is only one leader in a Raft cluster.

[0061] Followers: Followers are other nodes in a Raft cluster that receive log entries from the leader and replicate them in their own logs. Followers do not initiate requests; they respond to requests from the leader.

[0062] Candidate: A candidate is a node in a Raft cluster. During the election process, a node will become a candidate and participate in the leader election.

[0063] Log: The Raft algorithm uses a log to record client requests. The leader sends client requests as log entries to the followers, who then copy these log entries into their own logs.

[0064] Log replication: The leader sends log entries to the followers, who then append these entries to their own logs. Once a majority of nodes have acknowledged receipt of the log entries, the leader commits the log entries and applies them to the state machine.

[0065] Heartbeat: The leader periodically sends heartbeat messages to its followers to maintain its leadership. If a follower does not receive a heartbeat message for a period of time, it assumes that the leader has malfunctioned and initiates an election.

[0066] Election: When a follower believes the leader has failed, it becomes a candidate and initiates an election. Candidates send vote requests to other nodes, and the candidate that receives a majority of votes becomes the new leader.

[0067] Security: The Raft algorithm ensures system security through a series of mechanisms, such as limiting there to only one leader at a time and ensuring log consistency.

[0068] CAP: The CAP theorem (Consistency, Availability, Partition tolerance) is a fundamental concept in distributed systems. It describes that a distributed system can only satisfy two of the three properties when faced with network partitions.

[0069] Consistency: Consistency means that all nodes in a distributed system see the same data at the same time. When one node modifies the data, other nodes can immediately see this modification. This consistency model is often called strong consistency.

[0070] Availability: Availability refers to the ability of a distributed system to provide services at any time, even if some nodes fail. When one node fails, other nodes can still continue to process requests.

[0071] Partition tolerance: Partition tolerance refers to the ability of a distributed system to continue providing services even when a network partition occurs (i.e., a network connection interruption prevents communication between nodes). A distributed system should be able to maintain normal operation even in the event of a network partition.

[0072] Figure 1 There are primary clusters and secondary clusters. Each secondary cluster has a primary cluster node as the master node.

[0073] This multi-level tree structure makes the cluster size a power of the previous Raft algorithm limit. Compared to distributing network requests across each first-level node, this structure can support larger-scale data consistency clusters. Assuming that the previous Raft algorithm could only guarantee normal operation for 32 nodes, the modified structure can support 32*32 (1024) nodes for normal operation.

[0074] In the data consistency process, after receiving the log replication message sent by the level i leader, the level i follower continues to propagate downwards until the consensus node has no corresponding subtree. The level i-1 leader performs a consistency comparison on the collected data and sends the synchronization log to the level i leader, including synchronization log fragments or installation snapshots. The level i leader also sends heartbeat detection messages to the level i followers in its corresponding subtree based on the polling heartbeat duration. As long as the current consensus node, as the leader, has its corresponding subtree, it will send heartbeat detection messages to the followers in its corresponding subtree.

[0075] In one embodiment, the steps for constructing a multi-level tree-structured cluster include an election step:

[0076] The current-level follower is triggered to conduct a leader election when the current-level candidate does not receive a heartbeat detection message from the current-level leader.

[0077] Once a candidate at level i-1 becomes the leader at level i-1 through a voting mechanism, the remaining level i-1 consensus nodes update their consensus node state to follower. The level i-1 leader sends the level i-1 election result notification to the level i leaders in its corresponding subtree; the level i leader sends the level i-1 election result notification to the level i followers in its subtree; the level i followers continue to propagate the level i-1 election result notification downwards until the consensus node has no corresponding subtree. It is understandable that, as... Figure 3 As shown, an operation to notify secondary followers has been added on top of the primary election process.

[0078] Once a candidate of level i becomes the leader of level i through a voting mechanism, the remaining level i consensus nodes update their consensus node state to follower. The level i leader sends the level i election result notification to the corresponding level i-1 leader in its subtree, and the level i leader continues to propagate the level i election result notification upwards until the first-level leader. The level i-1 leader sends the level i election result notification to the remaining level i-1 followers, and the remaining level i-1 followers continue to propagate the level i election result notification downwards until the consensus node has no corresponding subtree. It is understandable that, as... Figure 4 As shown, the secondary election adds an operation to notify the primary leader node based on the election rules. Then, the primary leader node notifies other primary follower nodes, and the primary follower nodes then notify the secondary follower nodes they lead.

[0079] Consensus nodes that receive the election result notification adjust their node information accordingly.

[0080] In this embodiment, the voting mechanism includes:

[0081] The current level candidate sends an election request message to other consensus nodes in the current level;

[0082] The current and other consensus nodes verify the current and candidate status information, such as whether the current candidate's log directory is more complete than its own log directory. After verification, voting information is sent to the current level candidate.

[0083] The current-level candidate accumulates voting information. When the number of voting information meets the minimum number of consensus nodes, the current-level candidate updates its status to leader, and the remaining current-level consensus nodes update their status to follower.

[0084] In this embodiment, the minimum number of consensus nodes is required to be greater than half the number of consensus nodes in the current level cluster.

[0085] In this embodiment, the election process further includes a secondary election failure handling step:

[0086] If the i-1 level leader has not received the i-th level election result notification from the i-th level leader in its corresponding subtree after a specified number of polling heartbeat cycles, then the following steps are performed:

[0087] The i-1 level leader sends a liveness detection message to all consensus nodes in its corresponding subtree and designates one consensus node from among the consensus nodes whose liveness verification is successful as the i-1 level leader of its corresponding subtree.

[0088] The designated consensus node updates its state to level i leader, sends the level i election result notification to the other consensus nodes in the subtree, and returns an update success message to the level i-1 leader; the other consensus nodes update their state to level i follower.

[0089] The i-1 level leader adjusts the node information based on the successful update information, and continues to pass the successful update information upwards until the level 1 leader; and sends the i-1 level election result notification to the remaining i-1 level followers; the remaining i-1 level followers continue to pass the i-1 level election result notification downwards until the consensus node has no corresponding subtree.

[0090] Consensus nodes that receive the i-th election results adjust their node information according to the i-th election results notification.

[0091] It is understandable that, such as Figure 5As shown, when the election of the secondary cluster fails (such as when the number of cluster nodes drops below half), in order to ensure that the primary cluster can work normally (election and log replication), a leader will actively probe the survival information of other nodes in the cluster after not receiving the primary leader information from the secondary cluster within a limited number of heartbeats, and designate a node as the secondary leader node. After the secondary leader node is designated, the cluster is adjusted by the two leaders to reach a state of data consistency.

[0092] It should be noted that the condition for designating a secondary cluster leader is that a single cluster node must not meet the majority requirement. Therefore, the primary leader node forces the designation of a secondary cluster node as the secondary leader, thus ensuring data consistency.

[0093] In this embodiment, the step of designating a consensus node as the i-th level leader of its corresponding subtree from among the consensus nodes that have successfully passed liveness verification includes any one of the following:

[0094] Random assignment: Select one consensus node from the consensus nodes that have successfully passed the liveness verification as the i-th level leader of its corresponding subtree;

[0095] Compare logs: Compare the log states of consensus nodes that have successfully verified their liveness, and designate the consensus node with the latest log update time as the i-th level leader of its corresponding subtree.

[0096] In one embodiment, the steps for building a multi-level tree-structured cluster include a log replication step:

[0097] The i-level leader receives the log replication message sent by the i-1 level leader, verifies it, performs log synchronization, and sends a log replication message to the i-level follower. The i-level follower verifies it, performs log synchronization, and replies with an i-level response message to the i-level leader.

[0098] The i-level leader accumulates i-level response messages. When the number of i-level response messages meets the minimum number of consensus nodes, the i-level leader replies with an i-1 level response message to the i-1 level leader.

[0099] The i-1 level leader accumulates i-1 level response messages. When the number of i-1 level response messages meets the minimum number of nodes required for consensus, the i-1 level leader updates the client request corresponding to the log to the state machine for execution.

[0100] In this embodiment, the minimum number of nodes required includes: greater than half the number of consensus nodes in the current level cluster.

[0101] In one embodiment, the step of sending a heartbeat detection message includes:

[0102] The i-level leader receives heartbeat detection messages sent by the i-1 level leader based on the polling heartbeat duration; the i-level leader sends heartbeat detection messages to the i-level followers based on the polling heartbeat duration.

[0103] If a consensus node does not receive a heartbeat detection message within the polling heartbeat duration, it triggers the generation of a candidate from the consensus nodes of the current level cluster.

[0104] A second aspect of the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of a large-scale data consistency method based on a multi-level tree structure cluster according to the first aspect of the present invention.

[0105] A third aspect of the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of a large-scale data consistency method based on a multi-level tree structure cluster according to the first aspect of the present invention.

[0106] The above provides a detailed description of a large-scale data consistency method and device based on a multi-level tree structure cluster provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

[0107] In this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, without necessarily requiring or implying any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A large-scale data consistency method based on a multi-level tree structure cluster, characterized in that, Includes the following steps: Construct a multi-level tree-structured cluster, which includes: a first-level cluster and an i-level cluster, where i∈[2,N] and N is a positive integer greater than 2; the consensus node states in each cluster include: leader, follower, and candidate; assign an i-level cluster as its corresponding subtree to the follower in the i-1 level cluster, and the leader in the i-level cluster is the follower in the i-1 level cluster. The i-1 level leader sends a heartbeat detection message to the i-level leader in its corresponding subtree based on the polling heartbeat duration; the i-level leader triggers its own data collection based on the heartbeat detection message and feeds back the collected data to the i-1 level leader. The i-1 level leader performs a consistency comparison on the collected data. If there is no consistency, a consistent data structure is constructed, and the synchronization log of the constructed data is sent to the i-level leader. The i-level leader verifies the data and then performs log synchronization. The i-th level leader sends a log replication message to the i-th level followers in its subtree, and the i-th level followers perform log synchronization after verification.

2. The large-scale data consistency method based on a multi-level tree structure cluster according to claim 1, characterized in that, The steps for building a multi-level tree-structured cluster include an election step: The current-level follower is triggered to conduct a leader election when the current-level candidate does not receive a heartbeat detection message from the current-level leader. Once a candidate of level i-1 becomes the leader of level i-1 through a voting mechanism, the remaining level i-1 consensus nodes update their consensus node status to follower. The level i-1 leader then sends the level i-1 election result notification to the level i leaders in its corresponding subtree. The level i leader then sends the level i-1 election result notification to the level i followers in its subtree. The level i followers continue to pass on the level i-1 election result notification downwards until the consensus node has no corresponding subtree. Once a candidate of level i becomes the leader of level i through a voting mechanism, the remaining level i consensus nodes update their consensus node status to follower. The level i leader sends the level i election result notification to the corresponding level i-1 leader in its subtree. The level i leader continues to pass the level i election result notification upwards until the level i leader is reached. The level i-1 leader then sends the level i election result notification to the remaining level i-1 followers. The remaining level i-1 followers continue to pass the level i election result notification downwards until the consensus node has no corresponding subtree. The consensus node that receives the election result notification adjusts its node information accordingly.

3. The large-scale data consistency method based on a multi-level tree structure cluster according to claim 2, characterized in that, The voting mechanism includes: The current level candidate sends an election request message to other consensus nodes in the current level; The current and other consensus nodes verify the status information of the current and candidates, and after successful verification, send voting information to the current-level candidate. The current-level candidate accumulates the voting information. When the number of voting information meets the minimum number of consensus nodes, the current-level candidate updates its status to leader, and the remaining current-level consensus nodes update their status to follower.

4. A large-scale data consistency method based on a multi-level tree structure cluster according to claim 2, characterized in that, The election process also includes: If the i-1 level leader still does not receive the i-level election result notification from the i-level leader in its corresponding subtree after a specified number of polling heartbeat durations, then the following steps are performed: The i-1 level leader sends a liveness detection message to all consensus nodes in its corresponding subtree, and designates one consensus node as the i-level leader of its corresponding subtree from among the consensus nodes whose liveness verification is successful. The designated consensus node updates its state to level i leader, sends the level i election result notification to the other consensus nodes in the subtree, and returns an update success message to the level i-1 leader; the other consensus nodes update their state to level i follower. The i-1 level leader adjusts the node information based on the update success information, and the i-1 level leader continues to pass the update success information upwards until the first-level leader; and sends the i-1 level election result notification to the remaining i-1 level followers; the remaining i-1 level followers continue to pass the i-1 level election result notification downwards until the consensus node has no corresponding subtree; The consensus node that receives the i-th level election result will notify and adjust its node information according to the i-th level election result.

5. A large-scale data consistency method based on a multi-level tree structure cluster according to claim 4, characterized in that, The step of designating a consensus node as the i-th level leader of its corresponding subtree from among the consensus nodes that have successfully passed liveness verification includes any one of the following: Randomly assign: Select one consensus node from the consensus nodes that have successfully verified liveness as the i-th level leader of its corresponding subtree; The comparison log specifies that the log status of the consensus nodes that have successfully verified their liveness is compared, and the consensus node with the latest log update time is designated as the i-th level leader of its corresponding subtree.

6. A large-scale data consistency method based on a multi-level tree structure cluster according to claim 1, characterized in that, The steps to build a multi-level tree-structured cluster include log replication: The i-level leader receives the log replication message sent by the i-1 level leader, verifies it, performs log synchronization, and sends a log replication message to the i-level follower. The i-level follower verifies it, performs log synchronization, and replies with an i-level response message to the i-level leader. The i-th level leader accumulates the i-th level response messages. When the number of i-th level response messages meets the minimum number of consensus nodes, the i-th level leader replies with an i-1 level response message to the i-1 level leader. The i-1 level leader accumulates the i-1 level response messages. When the number of i-1 level response messages meets the minimum number of consensus nodes, the i-1 level leader updates the client request corresponding to the log to the state machine for execution.

7. A large-scale data consistency method based on a multi-level tree structure cluster according to claim 6, characterized in that, The minimum number of nodes required includes: greater than half the number of consensus nodes in the current level cluster.

8. A large-scale data consistency method based on a multi-level tree structure cluster according to claim 1, characterized in that, The step of sending the heartbeat detection message includes: The i-level leader receives heartbeat detection messages sent by the i-1 level leader based on the polling heartbeat duration; the i-level leader sends heartbeat detection messages to the i-level followers based on the polling heartbeat duration. If a consensus node does not receive a heartbeat detection message within the polling heartbeat duration, it triggers the generation of a candidate from the consensus nodes of the current level cluster.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of a large-scale data consistency method based on a multi-level tree structure cluster as described in any one of claims 1-8.

10. A non-transitory computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the steps of a large-scale data consistency method based on a multi-level tree structure cluster as described in any one of claims 1-8.

Images