Automatic cluster recombination method, distributed system and computer equipment

Abstract

The invention relates to an automatic cluster recombination method, a distributed system and computer equipment. Determining a plurality of independent network partitions after the distributed system is subjected to network partitioning; for each independent network partition, determining first existence information of each cluster node in each independent network partition; determining a target local leader node and a target follower node from the cluster nodes of the independent network partitions according to a preset consensus recombination strategy based on the first existence information; establishing a first local consensus cluster corresponding to the independent network partition according to a following relationship between the target local leader node and the target follower node; according to the invention, automatic reorganization of cluster nodes during network partitioning is realized, so that each independent network partition can still maintain local consensus and service availability under the condition that global intercommunication cannot be realized, service is provided to the greatest extent, the availability is ensured, and the service continuity and the overall availability of the system after network partitioning are remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of distributed systems, and particularly relates to a cluster automatic reorganization method, a distributed system and a computer device. BACKGROUND

[0002] Nowadays, Internet applications have deeply integrated into daily life. To cope with the demand for large data size, low delay, high throughput and strong scalability, various data-intensive systems widely adopt native distributed database architecture to improve system scalability and partition fault tolerance. However, with the explosive growth of business data and the increasing complexity of scenarios, network partition and other distributed faults occur frequently, which seriously threatens system reliability. When facing network partition, the traditional scheme usually follows the CAP (Consistency, Availability, Partition tolerance) theory, prioritizes consistency and actively sacrifices availability, resulting in some nodes being unavailable and affecting user experience.

[0003] In view of the problem that the distributed system cannot guarantee availability when network partition occurs in the related technology, no effective solution has been proposed at present. SUMMARY

[0004] Therefore, it is necessary to provide a cluster automatic reorganization method, a distributed system and a computer device in view of the above technical problems.

[0005] In a first aspect, the present application provides a cluster automatic reorganization method applied to a distributed system, and the method comprises the following steps.

[0006] determining a plurality of independent network partitions after the distributed system occurs network partition;

[0007] determining first existence information of each cluster node in each independent network partition for each independent network partition;

[0008] determining a target local leader node and a target follower node from the cluster nodes of the independent network partition according to a preset consensus reorganization strategy based on the first existence information;

[0009] establishing a first local consensus cluster corresponding to the independent network partition according to a follower relationship between the target local leader node and the target follower node.

[0010] In one of the embodiments, the preset consensus reorganization strategy comprises a first preset consensus reorganization strategy; and the determining of the target local leader node and the target follower node from the cluster nodes of the independent network partition according to the preset consensus reorganization strategy based on the first existence information comprises:

[0011] if there is a global leader node in the cluster nodes of the independent network partition, and the state of the global leader node is valid, determining the global leader node as a target local leader node, and determining all the cluster nodes of the independent network partition except the global leader node as target follower nodes;

[0012] if all the cluster nodes of the independent network partition are candidate nodes, determining a target local leader node and a target follower node from the candidate nodes of the independent network partition based on the first existence information according to the first preset consensus reorganization strategy.

[0013] In one of the embodiments, the first existence information includes a first belonging cluster identifier, a first log entry index, a first total number of cluster nodes, and a first leader node priority; and the determining of the target local leader node and the target follower node from the candidate nodes of the independent network partition based on the first existence information according to the first preset consensus reorganization strategy includes:

[0014] determining a first candidate local leader node from the candidate nodes of the independent network partition based on the first log entry index;

[0015] if there is one first candidate local leader node, determining the first candidate local leader as a target local leader node;

[0016] if there are multiple first candidate local leader nodes, determining a target local leader node satisfying a first preset preferred condition from the multiple first candidate local leader nodes based on the first leader node priority and the first belonging cluster identifier;

[0017] determining all the candidate nodes of the independent network partition except the target local leader node as target follower nodes.

[0018] In one of the embodiments, the preset consensus reorganization strategy includes a second preset consensus reorganization strategy; and the method further includes:

[0019] if it is detected that there is at least one network recovery area in the distributed system, determining at least two local consensus clusters to be reorganized in the network recovery area;

[0020] determining a target reorganization leader node and a target reorganization follower node from the at least two local consensus clusters to be reorganized according to the second preset consensus reorganization strategy;

[0021] According to a following relationship between the target reorganization leader node and the target reorganization follower node, a target reorganization consensus cluster corresponding to the network recovery area is established.

[0022] In one embodiment, the target reorganization leader node and the target reorganization follower node are determined from the at least two local consensus clusters to be reorganized according to the second preset consensus reorganization strategy, including:

[0023] Second existence information corresponding to each of the target reorganization leader nodes in each of the local consensus clusters to be reorganized is determined, and the second existence information includes a second cluster identifier, a second log entry index, a second total number of cluster nodes, and a second leader node priority.

[0024] If the second total number of cluster nodes is different, the target reorganization leader node is determined as the target reorganization leader node that meets the first reorganization condition, and all cluster nodes in the network recovery area except the target reorganization leader node are determined as target reorganization follower nodes.

[0025] If the second total number of cluster nodes is the same, the target reorganization leader node is determined as the target reorganization leader node that meets the second reorganization condition, and all cluster nodes in the network recovery area except the target reorganization leader node are determined as target reorganization follower nodes.

[0026] In one embodiment, the target reorganization leader node that meets the first reorganization condition is determined as the target reorganization leader node, including:

[0027] Based on the second log entry index, a first candidate reorganization leader node is determined from the at least two target reorganization leader nodes.

[0028] If there is one first candidate reorganization leader node, the first candidate reorganization leader node is determined as the target reorganization leader node.

[0029] If there are multiple first candidate reorganization leader nodes, the target reorganization leader node is determined from the multiple first candidate reorganization leader nodes based on the second total number of cluster nodes, the second leader node priority, and the second cluster identifier.

[0030] In one embodiment, the target reorganization leader node is determined from the multiple first candidate reorganization leader nodes based on the second total number of cluster nodes, the second leader node priority, and the second cluster identifier, including:

[0031] determining, based on the second total number of cluster nodes, a first candidate reorganization leader node with the highest second total number of cluster nodes from the plurality of first candidate reorganization leader nodes as a second candidate reorganization leader node;

[0032] if there is one second candidate reorganization leader node, determining the second candidate reorganization leader node as a target reorganization leader node;

[0033] if there are a plurality of second candidate reorganization leader nodes, determining a target reorganization leader node satisfying a second preset preference condition from the plurality of second candidate reorganization leader nodes based on the second leader node priority and the second cluster identification.

[0034] In one of the embodiments, the method further comprises:

[0035] determining, based on the second log entry index, a third candidate reorganization leader node from the at least two to-be-reorganized leader nodes;

[0036] if there is one third candidate reorganization leader node, determining the third candidate reorganization leader node as a target reorganization leader node;

[0037] if there are a plurality of third candidate reorganization leader nodes, determining a target reorganization leader node satisfying a second preset preference condition from the plurality of third candidate reorganization leader nodes based on the second leader node priority and the second cluster identification.

[0038] In one of the embodiments, the method further comprises:

[0039] for each cluster node after the network partition of the distributed system, when the cluster node joins a target consensus cluster where a target leader node is located in a follow state according to the preset consensus reorganization strategy, deleting all newly added log entries added after the network partition of the cluster node, and synchronizing consensus log entries submitted by the target leader node.

[0040] In one of the embodiments, before the cluster node joins the target consensus cluster where the target leader node is located in the follow state according to the preset consensus reorganization strategy, the method further comprises:

[0041] storing all newly added log entries added by the cluster node after the network partition to a target storage area according to a preset storage rule;

[0042] The preset storage rule includes taking the last log entry index corresponding to the cluster node as a storage index, archiving all newly added log entries after the cluster node itself is subjected to network partition, generating log archive data, and storing the log archive data to the target storage area according to the storage index.

[0043] In one of the embodiments, when the distributed system is not subjected to network partition, the distributed system includes a plurality of cluster nodes; the plurality of cluster nodes include a global leader node and at least one global follower node following the global leader node; the method further includes:

[0044] periodically broadcasting third existence information corresponding to the global leader node to each of the global follower nodes;

[0045] If there is at least one global follower node that does not receive the third existence information, it is determined that the distributed system is subjected to network partition.

[0046] In one of the embodiments, the method further includes:

[0047] After it is determined that the distributed system is subjected to network partition, for each of the cluster nodes, based on a preset network awareness strategy, a connected cluster node that is bi-directionally reachable with the cluster node is determined from the plurality of cluster nodes;

[0048] According to each of the cluster nodes and the connected cluster node that is bi-directionally reachable with each of the cluster nodes, a plurality of independent network partitions of the distributed system subjected to network partition are determined; any two cluster nodes in each of the independent network partitions are bi-directionally reachable.

[0049] In a second aspect, the present application provides a distributed system for performing steps of the method as described above.

[0050] In a third aspect, the present application provides a computer device including a memory and a processor, the memory stores a computer program, and the processor implements steps of the method as described above when executing the computer program.

[0051] In a fourth aspect, the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement steps of the method as described above.

[0052] In a fifth aspect, the present application provides a computer program product, which includes a computer program, and the computer program is executed by a processor to implement steps of the method as described above.

[0053] The cluster automatic reorganization method, the distributed system and the computer device have the following advantages. After network partition occurs in the distributed system, a plurality of independent network partitions are identified, and first existence information of cluster nodes in each independent network partition is acquired. Based on the first existence information, the system can dynamically elect target local leader nodes and corresponding target follower nodes from the cluster nodes of each independent network partition according to a preset consensus reorganization strategy. Then, a first local consensus cluster that adapts to the current network state is established according to the following relationship between the target local leader nodes and the target follower nodes, that is, a following relationship. The cluster automatic reorganization method, the distributed system and the computer device realize automatic reorganization of cluster nodes when network partition occurs, so that each independent network partition can still maintain local consensus and service availability in the case that global intercommunication is unavailable, thereby providing services to the maximum extent, guaranteeing availability, and significantly improving service continuity and overall availability of the system after network partition occurs. BRIEF DESCRIPTION OF DRAWINGS

[0054] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the drawings needed to be used in the description of the embodiments of the present application or the related art will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other related drawings can be obtained by those skilled in the art without any creative effort on the basis of these drawings.

[0055] Figure 1 A flowchart of a cluster automatic reorganization method in an embodiment;

[0056] Figure 2 A flowchart of a target local leader node and a target follower node determination step in an embodiment;

[0057] Figure 3 A schematic diagram of a global consensus cluster in an embodiment when network partition does not occur in the distributed system;

[0058] Figure 4 A schematic diagram of each independent network partition in an embodiment in the initial stage of network partition;

[0059] Figure 5 A schematic diagram of a first local consensus cluster corresponding to each independent network partition in an embodiment;

[0060] Figure 6 A flowchart of a first preset consensus reorganization strategy in an embodiment;

[0061] Figure 7 A flowchart of a cluster automatic reorganization method in an embodiment in a network recovery stage;

[0062] Figure 8 A flowchart of a second preset consensus reorganization strategy in an embodiment;

[0063] Figure 9 A schematic diagram of cluster automatic reorganization in a local network recovery phase in an embodiment;

[0064] Figure 10 A schematic diagram of cluster automatic reorganization in a global network recovery phase in an embodiment;

[0065] Figure 11 A schematic diagram of archiving of a new log entry for node D in an embodiment;

[0066] Figure 12 An internal structure diagram of a computer device in an embodiment. DETAILED DESCRIPTION

[0067] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not used to limit the present application.

[0068] Today's Internet applications have become an important part of people's lives. In order to meet the data-intensive application requirements of various scenarios with large data volume, delay, throughput and high scalability requirements, more and more database systems implement native distributed cluster management methods to achieve higher scalability and partition fault tolerance. However, with the continuous development of business data volume and business individual needs, the scenarios faced by distributed database systems are becoming more and more complex. Network partitioning, as a representative of distributed system failure, has brought various challenges to the reliability of distributed systems.

[0069] When facing network partitioning, the traditional scheme usually follows the CAP theory, prioritizes consistency and actively sacrifices availability, resulting in some nodes being unavailable and affecting user experience. In addition, the scheme in the related art that relies on centralized coordination services such as ZooKeeper and etcd requires additional deployment and maintenance of independent cluster instances, which not only increases the complexity of software package management and server resource configuration, making the deployment process cumbersome, but also introduces significant software and hardware overhead and operation and maintenance burden. More critically, such centralized architecture has a single point of failure risk. Once the coordination service cluster is abnormal, the entire distributed system will be paralyzed. At the same time, its normal operation is highly dependent on the stability of the control plane network, further increasing the dependence on network conditions and reducing the robustness and autonomy of the system.

[0070] To solve the above technical problems, the application provides a cluster automatic reorganization method, which aims to realize automatic reorganization of cluster nodes after network partition of a distributed system, so that each independent network partition can maintain local consensus and service availability under the condition of being unable to globally interconnect, thereby providing maximum service and ensuring availability, and significantly improving service continuity and overall availability of the system after network partition.

[0071] In one embodiment, as shown in Figure 1 , Figure 1 is a flowchart of the cluster automatic reorganization method in one embodiment; the cluster automatic reorganization method is applied to a distributed system and includes the following steps:

[0072] Step S101, determining a plurality of independent network partitions after network partition of the distributed system.

[0073] The distributed system is a system composed of a plurality of physically dispersed and logically cooperative cluster nodes, and each cluster node communicates, exchanges data and cooperates through a network to jointly complete a specific computing or service task. The distributed system can be but is not limited to a distributed database system, which is not specifically limited here.

[0074] It should be noted that each cluster node in the distributed system is embedded with an extended Raft cluster instance, which is used to provide high-availability and scalable service capabilities and support for continuing to provide services through local reorganization after network partition. It can be understood that by embedding the extended Raft cluster instance into the nodes of the distributed system, automatic distributed consensus and cluster awareness and management capabilities can be achieved, which significantly reduces the deployment complexity, hardware and software environment requirements and operation and maintenance costs of the cluster, and provides stronger scalability for the distributed system.

[0075] The network partition refers to the fact that due to network failure, excessively high delay or connection interruption, etc., part of the nodes in the distributed system cannot normally communicate with each other, so that the originally connected network is divided into a plurality of independent network partitions that run independently and are isolated from each other. Within each independent network partition, nodes can still communicate and cooperate with each other, but nodes across the partitions are in an unreachable state.

[0076] In one exemplary embodiment, network partitioning can be determined, but is not limited to, by combining network reachability detection results with node state information: when a node is unable to communicate with a majority of nodes in the cluster via heartbeats or voting requests within a preset period, and candidate nodes fail to complete leader election due to insufficient votes, and do not receive information about the existence of a valid leader for an extended period, a network partition can be determined. This embodiment, combining network reachability detection and consensus state analysis, can accurately identify network splitting scenarios, avoid misjudgments caused by momentary delays or single points of failure, and improve the reliability and accuracy of partition detection.

[0077] An independent network partition refers to a set of isolated sub-networks formed when the original system was divided in the initial stage of a network partition due to the interruption of network connectivity between some nodes. These sub-networks are mutually isolated and cannot communicate directly with each other. It is understandable that any two nodes within each independent network partition can still maintain network reachability and exchange messages and coordinate work normally, but they cannot communicate with nodes outside the partition.

[0078] In an exemplary embodiment, the method for determining multiple independent network partitions after a network partition occurs in a distributed system may include, but is not limited to, the following: After determining that a network partition has occurred in the distributed system, for each cluster node, based on a preset network awareness strategy, determine the connected cluster nodes that are bidirectionally reachable from the multiple cluster nodes; based on each cluster node and the connected cluster nodes that are bidirectionally reachable from each cluster node, determine multiple independent network partitions after the network partition occurs in the distributed system; wherein, any two cluster nodes in each independent network partition are bidirectionally reachable from each other. The preset network awareness strategy may include, but is not limited to, detecting network connectivity by having nodes broadcast their own presence information. Each node receives presence information from other nodes, determines their reachability and activity status, and identifies the currently connectable set of nodes based on bidirectional communication confirmation, thereby forming independent network partitions. The form of broadcasting presence information needs to be determined based on the network deployment environment corresponding to the cluster nodes, and is not specifically limited here.

[0079] Step S102: For each independent network partition, determine the first existence information of each cluster node in each independent network partition.

[0080] In the process of each cluster node, a Raft instance thread is embedded. In an embodiment, taking a cluster node as a distributed database node as an example, for each distributed database node running in the cluster, a Raft instance thread is started in the node process, and all the Raft instances corresponding to the nodes are in a global consensus cluster (i.e., a global Raft group) in normal operation; the instance node in the distributed system is bound to the node in the global consensus cluster (i.e., the global Raft group), and the node list and leader of the entire cluster are taken as the node configuration set and the global leader node of the global consensus cluster, and the node existence information is propagated to the entire cluster by replicating the node-specific information in the global consensus cluster, and the node control is performed through the node existence information, so as to realize the cluster control process such as leader election, perception of cluster information, and broadcast of cluster control events; through the above-mentioned manner, an embedded cluster management and distributed consensus system can be constructed.

[0081] The first existence information at least includes a first belonging cluster identifier, a first log entry index, a first belonging cluster node total number, a first leader node priority, and first term information.

[0082] The first belonging cluster identifier is a globally unique identifier of the consensus cluster to which the cluster node belongs. The first belonging cluster identifier can be but is not limited to in the form of {node ID}_{random UUID}, for example, the first belonging cluster identifier can be A_abcd.

[0083] The first log entry index is an index number of the highest log entry that has been successfully replicated by a majority of nodes in the cluster and reached an agreement; it can be understood that the first log entry index is a monotonically increasing integer; all log entries with an index less than or equal to the first log entry index are considered to have reached distributed consistency, and their contents cannot be changed and can be safely and sequentially applied by the state machine of each node. Each committed log entry represents a state change decision that is recognized by the cluster.

[0084] The first belonging cluster node total number is the total number of cluster nodes in the consensus cluster to which the cluster node belongs; in an exemplary embodiment, the first belonging cluster node total number is greater than or equal to 1. For example, in the initial stage of network partitioning of a distributed system, if the cluster nodes of an independent network partition are all candidate nodes, each candidate node is considered as a local consensus cluster with a first belonging cluster node total number of 1.

[0085] The first leader node priority is the priority weight of the cluster node to become a leader. The first term information is the term number of the cluster in which the cluster node is currently located; the first term information is a globally monotonically increasing logical clock, and the term number is incremented by 1 each time a leader election occurs.

[0086] In step S103, based on the first existence information of each cluster node, a target local leader node and a target follower node are determined from the cluster nodes in the independent network partition according to a preset consensus reorganization strategy.

[0087] The preset consensus reorganization strategy can be a set of methods for automatically reorganizing the cluster nodes according to the existence information of each cluster node when the network is partitioned. In an exemplary embodiment, the preset consensus reorganization strategy is designed based on the Raft algorithm extension logic and is fixed in the system configuration, supporting group joining decision making according to the existence information.

[0088] The target local leader node refers to a cluster node that assumes the role of a leader in the independent network partition after the network partition occurs, and is determined by the preset consensus reorganization strategy. The target follower node refers to a cluster node that is in the same independent network partition as the target local leader node and follows the target local leader node. The "follow" refers to acknowledging the leadership of the target local leader node and actively responding to the instructions of the target local leader node to achieve state synchronization and cooperation.

[0089] In step S104, a first local consensus cluster corresponding to the independent network partition is established according to the follow-up relationship between the target local leader node and the target follower node.

[0090] It should be noted that in the independent network partition, by establishing an explicit master-slave follow-up relationship between the target follower node and the target local leader node, a first local consensus cluster with complete consensus capability is formed, which can guarantee the availability of local services based on the first local consensus cluster before the global network is restored, effectively improving the fault tolerance and business continuity of the system in the network abnormal scenario.

[0091] In this embodiment, after the network partition occurs in the distributed system, multiple independent network partitions are identified, and the first existence information of the cluster nodes in each independent network partition is obtained. Based on the first existence information, the system can dynamically elect a target local leader node and corresponding target follower nodes from the cluster nodes in each independent network partition according to the preset consensus reorganization strategy, and then establish a first local consensus cluster that adapts to the current network state according to the follow-up relationship between the target local leader node and the target follower node, realizing automatic reorganization of the cluster nodes when the network is partitioned. Each independent network partition can still maintain local consensus and service availability even if it cannot be globally interconnected, thereby providing maximum service and ensuring availability, significantly improving the service continuity and overall availability of the system after the network partition occurs.

[0092] In one embodiment, as Figure 2as shown, Figure 2 A flowchart of a determination procedure for a target local leader node and a target follower node in an embodiment; based on each first existence information, a target local leader node and a target follower node are determined from the cluster nodes in the independent network partition according to a preset consensus reorganization strategy, including the following steps:

[0093] Step S201, if there is a global leader node in the cluster nodes in the independent network partition, and the state of the global leader node is valid, the global leader node is determined as the target local leader node, and all cluster nodes in the independent network partition except the global leader node are determined as the target follower nodes.

[0094] Wherein, in the case that the distributed system does not occur network partition, the distributed system includes a plurality of cluster nodes; the plurality of cluster nodes includes a global leader node and at least one global follower node following the global leader node. It can be understood that the global leader node and the at least one global follower node together constitute a global consensus cluster in the global network. Wherein, the global leader node refers to a cluster node that assumes a leader role when the distributed system does not occur network partition.

[0095] Exemplarily, after the distributed system occurs network partition, if it is detected that there is a global leader node in the cluster nodes in the independent network partition, and the state of the global leader node is valid (such as the term is not expired, and the self is not lost), the global leader node is determined as the target local leader node, and all cluster nodes in the independent network partition except the global leader node are determined as the target follower nodes; then according to the following relationship between the target local leader node and the target follower node, a first local consensus cluster corresponding to the current independent network partition is established.

[0096] Step S202, if all the cluster nodes in the independent network partition are candidate nodes, then based on each first existence information, a target local leader node and a target follower node are determined from the candidate nodes in the independent network partition according to a first preset consensus reorganization strategy.

[0097] It should be noted that when the distributed system occurs network partition, the global follower node which is unreachable from the global leader node actively updates its own state from "follower" to "candidate", and then converts to a candidate node.

[0098] The preset consensus reorganization strategy includes a first preset consensus reorganization strategy. It should be noted that the first preset consensus reorganization strategy is applied to scenarios in the early stages of network partitioning and before recovery. It aims to integrate the initial presence information reported or broadcast by each node after an independent network partition occurs, and to conduct multi-dimensional collaborative decision-making to ensure the safe and efficient election of a representative and consistent target local leader node and target follower nodes within a local area, thereby establishing a stable first local consensus cluster.

[0099] Understandably, in the initial stage of network partitioning in a distributed system, for each independent network partition lacking a global leader node, all cluster nodes within that partition are candidate nodes. At this point, each candidate node considers itself a local consensus cluster with only one node. Furthermore, based on the first existence information corresponding to each cluster node (i.e., candidate node) in the independent network partition, a target local leader node and target follower nodes are determined from the candidate nodes according to a first preset consensus reorganization strategy. This facilitates automatic reorganization of cluster nodes and the establishment of the first local consensus cluster corresponding to the independent network partition.

[0100] For example, a global consensus cluster in a distributed system where no network partitioning occurs, such as Figure 3 As shown, the global consensus cluster consists of 8 nodes; the cluster majority (Quorum, i.e., more than half) of the distributed system is 5; in this embodiment, the first cluster identifier is generated using {Node ID}_{Random UUID}, where the random UUID is represented by 4 lowercase letters. Node A is elected as the global leader node, and nodes B to H are all global followers; the number after ":" in each node represents the known log entry index (CommitIndex, CID) of the node. For example, "4" in "A:4" for node A means that node A's known log entry index CID is 4; "3" in "G:3" for node G means that node G's known log entry index CID is 3; node G's known log entry index CID of 3 may be due to network latency preventing the receipt of the latest CID. During normal operation, all nodes in the global consensus cluster can communicate with each other. Figure 3In this diagram, "ID:A_abcd" represents the cluster identifier corresponding to the global consensus cluster, namely Major:A_abcd(8); "CID:4" indicates that the log entry index CID corresponding to the global consensus cluster, namely Major:A_abcd(8), is 4; and "#N:8" indicates that the total number of nodes corresponding to the global consensus cluster, namely Major:A_abcd(8), is 8. It should be noted that the global consensus cluster belongs to the majority consensus cluster because its total number of nodes exceeds the cluster majority (Quorum=5) of the distributed system.

[0101] Furthermore, in the initial stage of distributed system network partitioning, the entire network was divided into three independent network partitions that were unreachable from each other, such as... Figure 4 As shown; for an independent network partition where the global leader node A is located, if the global leader node A is in a valid state (e.g., its term has not expired, and it is not disconnected), then the global leader node A is determined as the target local leader node, and nodes B and C are determined as target follower nodes; based on the following relationship between the target local leader node (i.e., node A) and the target follower nodes (i.e., nodes B and C), the first local consensus cluster Minor:A_abcd(3) corresponding to the independent network partition is established. It should be noted that the first local consensus cluster Minor:A_abcd(3) belongs to the minority consensus cluster because its total number of nodes is less than the cluster majority (Quorum=5) of the distributed system.

[0102] For the other two independent network partitions, in the independent network partitions where nodes D and E reside, nodes D and E are bidirectionally reachable. In this case, node D considers itself a local consensus cluster with only one node (which can also be understood as a minority consensus cluster). In the first existence information corresponding to node D, "ID:D_bcde" represents the cluster identifier of the local consensus cluster to which node D currently belongs; "CID:4" represents that the log entry index CID of the local consensus cluster to which node D currently belongs is 4; and "#N:1" represents that the total number of nodes in the local consensus cluster to which node D currently belongs is 1. Similarly, node E considers itself a local consensus cluster with only one node (which can also be understood as a minority consensus cluster). In the first existence information corresponding to node E, "ID:E_cdef" represents the cluster identifier of the local consensus cluster to which node E currently belongs; "CID:4" represents that the log entry index CID of the local consensus cluster to which node E currently belongs is 4; and "#N:1" represents that the total number of nodes in the local consensus cluster to which node E currently belongs is 1.

[0103] At this time, based on the first existence information corresponding to node D and the first existence information corresponding to node E, the target local leader node is determined as node E and the target follower node is determined as node D from node D and node E according to the first preset consensus reorganization strategy; further, node D is added to the local consensus cluster where node E is located to establish the first local consensus cluster Minor:E_cdef(2) of the independent network partition, as shown in Figure 5 It should be noted that the first local consensus cluster Minor:E_cdef(2) also belongs to the minority consensus cluster.

[0104] Similarly, for the independent network partition where node F, node G and node H are located, node F, node G and node H each regard itself as a local consensus cluster (which can also be understood as a minority consensus cluster) of only one node; based on the first existence information corresponding to node F, node G and node H respectively, the target local leader node is determined as node H and the target follower nodes are determined as node F and node G from node F, node G and node H according to the first preset consensus reorganization strategy; further, node F and node G are added to the local consensus cluster where node H is located to establish the first local consensus cluster Minor:H_fghi(3) of the independent network partition, as shown in Figure 5 It should be noted that the first local consensus cluster Minor:H_fghi(3) also belongs to the minority consensus cluster.

[0105] It should be noted that the cluster majority (Quorum) of the distributed system in the embodiment is 5, so the local consensus cluster with a total number of cluster nodes less than the cluster majority (i.e. 5) can be defined as a minority consensus cluster, i.e. a Minor consensus cluster. The local consensus cluster with a total number of cluster nodes greater than or equal to the cluster majority (i.e. 5) can be defined as a majority consensus cluster, i.e. a Major consensus cluster.

[0106] In this embodiment, after network partition occurs, if there is a global leader node in the cluster nodes of the independent network partition, and the state of the global leader node is valid (such as the term is not expired, and the global leader node is not lost), the global leader node is directly taken as a target local leader node, and the rest of the reachable nodes are taken as target follower nodes, a following relationship is quickly rebuilt, and a first local consensus cluster is formed. If the independent network partition does not contain a global leader node, a new target local leader node is dynamically elected from the current multiple candidate nodes based on the first existence information reported by each node according to the first preset consensus reorganization strategy, and other nodes in the independent network partition are determined as target follower nodes, a new following relationship is established, and a corresponding first local consensus cluster is formed. Through the above method, the global leader node is preferentially retained during network partition, so as to reduce state switching overhead, and when the global leader node is unreachable, local election is automatically triggered based on the first preset consensus reorganization strategy, so that the entire global consensus cluster can be split into a plurality of first local consensus clusters, thereby maximizing service provision and ensuring availability.

[0107] In one embodiment, as shown in Figure 6 Figure 6 is a flowchart of the first preset consensus reorganization strategy in one embodiment; based on the first existence information, a target local leader node and a target follower node are determined from the candidate nodes of the independent network partition according to the first preset consensus reorganization strategy, including the following steps:

[0108] Step S601, a first candidate local leader node is determined from the candidate nodes of the independent network partition based on a first log entry index.

[0109] The first existence information includes a first belonging cluster identifier, a first log entry index, a first belonging cluster node total number, and a first leader node priority.

[0110] It should be noted that when the distributed system network partition is in the initial stage and has not been restored, for the independent network partition that does not contain a global leader node, each cluster node in the independent network partition has become a candidate node because it cannot communicate with the global leader node; at this time, since the following relationship between the leader and the follower has not been formed in the independent network partition, each candidate node regards itself as a local consensus cluster with only one node, and the first belonging cluster node total number corresponding to each cluster node is 1. In this case, the first preset consensus reorganization strategy will preferentially determine a first candidate local leader node from the candidate nodes of the independent network partition according to the first log entry index corresponding to each candidate node; if there are multiple first candidate local leader nodes, the first belonging cluster identifier and the first leader node priority are further used for decision-making to screen the final target local leader node from the multiple first candidate local leader nodes.​

[0111] It should be noted that the first log entry index is a monotonically increasing integer, and therefore, the method of determining the first candidate local leader node from the candidate nodes of the independent network partition based on the first log entry index can be: determining the candidate node corresponding to the highest index value of the first log entry index as the first candidate local leader node, so as to ensure that the node has the latest log state, maximally retains the committed data, and guarantees the data integrity and consistency of the distributed system in the local consensus reconstruction process.

[0112] Step S602, if there is a first candidate local leader node, determining the first candidate local leader as the target local leader node.

[0113] Step S603, if there are multiple first candidate local leader nodes, determining the target local leader node satisfying the first preset preferred condition from the multiple first candidate local leader nodes based on the first leader node priority and the first belonging cluster identifier.

[0114] The first preset preferred condition can be that the first leader node priority is the highest and the lexicographic order of the first belonging cluster identifier is the largest, or the first leader node priority is the highest and the lexicographic order of the first belonging cluster identifier is the smallest. It should be noted that the first preset preferred condition can be flexibly set according to actual system requirements, and the core is to ensure that the target local leader node is uniquely determined in the multiple first candidate local leader nodes through the hierarchical comparison mechanism. The specific strategy can be configured according to the performance, stability or business scenario requirements, and is not specifically limited here.

[0115] Step S604, determining all candidate nodes except the target local leader node in the independent network partition as target follower nodes.

[0116] In an exemplary embodiment, the target local leader node is determined based on the first log entry index, and the target follower nodes are determined based on the second log entry index. Figure 4For example, considering the independent network partitions where nodes D and E reside, since the first log entry index CID for both nodes D and E is 4, nodes D and E are identified as the first candidate local leader nodes. Further, assume that the first leader nodes for nodes D and E have the same priority. Based on the first cluster identifier "D_bcde" for node D and "E_cdef" for node E, the lexicographical order of the first cluster identifier for node D is less than that for node E. If the first preset preference condition is that the first leader node has the highest priority and the first cluster identifier has the largest lexicographical order, then node E satisfies the first preset preference condition. Therefore, node E is identified as the target local leader node, and node D is identified as the target follower node, thus establishing the corresponding first local consensus cluster.

[0117] For the independent network partitions where nodes F, G, and H reside, the first log entry index CID for nodes F and H is 4, and for node G it is 3. The candidate nodes corresponding to the first log entry index with the highest index value are nodes F and H. Therefore, nodes F and H are determined as the first candidate local leader nodes. Further, it is assumed that the first leader nodes corresponding to nodes F and H have the same priority. Based on the first cluster identifier "F_defg" for node F and "H_fghi" for node H, the lexicographical order of the first cluster identifier for node F is less than that for node H. If the first preset preference condition is that the first leader node has the highest priority and the lexicographical order of its first cluster identifier is the largest, then node H satisfies the first preset preference condition. Therefore, node H is determined as the target local leader node, and nodes F and G are determined as target follower nodes, thus establishing the corresponding first local consensus cluster.

[0118] In this embodiment, one or more first candidate local leader nodes are determined based on the first log entry index. If only one first candidate local leader node exists, it is directly determined as the target local leader node. If multiple first candidate local leader nodes exist, final arbitration is further performed according to the priority of the first leader node and the first cluster identifier, based on the first preset preference conditions, to ensure that a unique target local leader node is selected. Through the above-mentioned hierarchical screening mechanism, local leader election can be deterministically completed even without a global leader node, achieving fast, stable, and consistent node reorganization, effectively improving the availability and stability of the distributed system during network partitioning.

[0119] In one embodiment, such as Figure 7 As shown,Figure 7 This is a flowchart illustrating an automatic cluster reassembly method during the network recovery phase in one embodiment; the preset consensus reassembly strategy includes a second preset consensus reassembly strategy; the automatic cluster reassembly method further includes the following steps:

[0120] Step S701: If at least one network recovery zone is detected in the distributed system, at least two local consensus clusters to be reorganized are identified within the network recovery zone.

[0121] The network recovery zone refers to the area in a distributed system where, after a network partition occurs, network connectivity is re-established between at least two previously isolated network zones as the network fault is repaired. The term "network zone" refers to a logical sub-cluster that is isolated from each other during the network recovery process. It can be an independent network partition in the initial stage or an intermediate network recovery zone that has been partially reassembled during the recovery process; its specific form depends on the actual progress and topology changes of the network recovery, and is not specifically limited here.

[0122] It should be noted that the number of network recovery zones depends on the actual topology and connectivity of the network recovery, and is not specifically limited here. For example, if one network recovery zone is detected in the distributed system, it can be determined that the system has achieved global network recovery; if two or more isolated network recovery zones are detected in the distributed system, it indicates that the system is in the local recovery stage, and the connectivity between different areas has not yet fully converged.

[0123] The term "local consensus cluster to be reorganized" refers to a local consensus cluster in the network recovery zone that has not yet completed cluster reorganization. It is understood that this embodiment supports multi-stage, gradual network recovery scenarios. The local consensus cluster to be reorganized can be a first local consensus cluster formed in the early stage of network partitioning, an intermediate local consensus cluster formed after partial reorganization of multiple first local consensus clusters, a new intermediate local consensus cluster formed after partial reorganization of multiple intermediate local consensus clusters, or a new intermediate local consensus cluster formed after partial reorganization of first local consensus clusters and intermediate local consensus clusters. No specific limitations are made here.

[0124] In one exemplary embodiment, the method for detecting the existence of a network recovery zone in a distributed system can be implemented by cluster nodes automatically sensing changes in network status. Specifically, when nodes from at least two different network zones resume bidirectional communication and mutually confirm network reachability, the system determines that connectivity has been established between at least two different network zones, thereby forming a corresponding network recovery zone.

[0125] Step S702, according to the second preset consensus reorganization strategy, determining a target reorganization leader node and a target reorganization follower node from the at least two local consensus clusters to be reorganized.

[0126] It should be noted that the second preset consensus reorganization strategy is suitable for a network recovery scenario and can adapt to multiple network recovery modes. The network recovery mode can include, but is not limited to, a multi-stage gradual recovery. It can be understood that the multi-stage gradual recovery refers to gradual recovery of network connectivity, and multiple network partitions are merged in batches.

[0127] The target reorganization leader node refers to a cluster node that assumes a leader role in a network recovery zone during network recovery and is determined by the second preset consensus reorganization strategy. The target reorganization follower node refers to a cluster node that is in the same network recovery zone as the target reorganization leader node and follows the target reorganization leader node. The "follow" refers to acknowledging the leadership of the target reorganization leader node and actively responding to the instructions of the target reorganization leader node to achieve state synchronization and cooperation.

[0128] Step S703, establishing a target reorganization consensus cluster corresponding to the network recovery zone according to the follow-up relationship between the target reorganization leader node and the target reorganization follower node.

[0129] It should be noted that in the network recovery zone, by establishing an explicit master-slave follow-up relationship between the target reorganization leader node and the target reorganization follower node, a target reorganization consensus cluster with complete consensus capability is formed. After network recovery, the local states can be quickly unified to ensure data consistency and continuously provide stable and usable services, thereby ensuring service continuity and overall reliability of the system in the recovery phase.

[0130] In this embodiment, by identifying multiple local consensus clusters to be reorganized in the network recovery zone, a unique target reorganization leader node is elected based on the second preset consensus reorganization strategy, and the remaining nodes are determined as target reorganization follower nodes. Then, a master-slave follow-up relationship is established between the target reorganization leader node and the target reorganization follower node to form a unified target reorganization consensus cluster with complete consensus capability. This process realizes automatic reorganization of multiple independent evolving local consensus clusters to be reorganized after network recovery, and ensures service continuity and overall reliability of the system in the recovery phase.

[0131] In one embodiment, as shown in Figure 8 Figure 8 ​A flowchart of a second preset consensus reorganization strategy in an embodiment; according to the second preset consensus reorganization strategy, a target reorganization leader node and a target reorganization follower node are determined from at least two local consensus clusters to be reorganized, including the following steps:

[0132] In step S801, second existence information corresponding to each of the reorganization leader nodes in each of the local consensus clusters to be reorganized is determined.

[0133] It can be understood that the local consensus cluster to be reorganized includes the reorganization leader node and at least one reorganization follower node following the reorganization leader node.

[0134] The method for determining the second existence information corresponding to each of the reorganization leader nodes in each of the local consensus clusters to be reorganized can be that when any reorganization leader node detects that network connectivity with nodes in other local consensus clusters to be reorganized is restored, the reorganization leader node actively broadcasts its own second existence information, so that each reorganization leader node can obtain the second existence information of the other reorganization leader nodes, thereby providing a basis for subsequent role decision based on the second preset consensus reorganization strategy.

[0135] The second existence information includes a second cluster identifier, a second log entry index, a second total number of cluster nodes, and a second leader node priority. The second cluster identifier refers to a globally unique identifier of the consensus cluster to which the cluster node belongs. The second log entry index refers to the index number of the highest log entry that has been successfully replicated by a majority of nodes in the cluster and reached an agreement; it can be understood that the second log entry index is a monotonically increasing integer. The second total number of cluster nodes refers to the total number of cluster nodes in the consensus cluster to which the cluster node belongs. The second leader node priority refers to the priority weight of the cluster node as a leader.

[0136] In step S802, if the second total number of cluster nodes is not the same, the reorganization leader node satisfying the first reorganization condition is determined as the target reorganization leader node, and all cluster nodes in the network recovery area except the target reorganization leader node are determined as the target reorganization follower node.

[0137] The first reorganization condition is used as a key criterion for determining the target reorganization leader node when the second total number of cluster nodes of each local consensus cluster to be reorganized in the network recovery area is inconsistent. It should be noted that the first reorganization condition follows the principle of the highest log entry index, the principle of majority priority, and the principle of the highest leader priority.

[0138] Step S803, if the total number of second belonging cluster nodes is the same, determining the leader node meeting the second reorganization condition as a target reorganization leader node, and determining all cluster nodes in the network recovery area except the target reorganization leader node as target reorganization follower nodes.

[0139] The second reorganization condition is used as a key criterion for determining the target reorganization leader node when the total number of second belonging cluster nodes of each reorganization local consensus cluster in the network recovery area is consistent. It should be noted that the second reorganization condition follows the principle of the highest log entry index and the principle of the highest leader priority.

[0140] In this embodiment, the system makes a reorganization role decision based on the first reorganization condition when the total number of second belonging cluster nodes is not equal during the network recovery process, which can ensure the majority dominance and guarantee data consistency and security. When the total number of second belonging cluster nodes is the same, the system makes a reorganization role decision based on the second reorganization condition according to the log entry index and the leader priority, which can ensure the uniqueness and rationality of the election. Based on this, the orderly integration of multiple reorganization local consensus clusters and the rapid reconstruction of global consensus are realized.

[0141] In one embodiment, determining the leader node meeting the first reorganization condition as a target reorganization leader node includes the following steps:

[0142] Step 1, determining a first candidate reorganization leader node from the at least two reorganization leader nodes based on the second log entry index.

[0143] It should be noted that the second log entry index is a monotonically increasing integer, so the method of determining a first candidate reorganization leader node from the at least two reorganization leader nodes based on the second log entry index can be: determining the reorganization leader node corresponding to the second log entry index with the highest index value as the first candidate reorganization leader node.

[0144] Step 2, if there is a first candidate reorganization leader node, determining the first candidate reorganization leader node as a target reorganization leader node.

[0145] Step 3, if there are multiple first candidate reorganization leader nodes, determining a target reorganization leader node from the multiple first candidate reorganization leader nodes based on the total number of second belonging cluster nodes, the second leader node priority, and the second belonging cluster identifier.

[0146] In an exemplary embodiment, step 3 of determining a target reorganization leader node from the multiple first candidate reorganization leader nodes based on the total number of second belonging cluster nodes, the second leader node priority, and the second belonging cluster identifier includes the following steps:

[0147] Step 3.1, based on the total number of second belonging cluster nodes, determining the first candidate reorganization leader node with the highest total number of second belonging cluster nodes from the plurality of first candidate reorganization leader nodes as a second candidate reorganization leader node.

[0148] It should be noted that the first candidate reorganization leader node with more second belonging cluster nodes is preferentially selected as the second candidate reorganization leader node, which can ensure data integrity and system representation, thereby ensuring the security and consistency of the reorganization process.

[0149] Step 3.2, if there is one second candidate reorganization leader node, determining the second candidate reorganization leader node as the target reorganization leader node.

[0150] Step 3.3, if there are multiple second candidate reorganization leader nodes, based on the second leader node priority and the second belonging cluster identifier, determining the target reorganization leader node that meets the second preset preferred condition from the multiple second candidate reorganization leader nodes.

[0151] The second preset preferred condition can be that the second leader node priority is the highest and the lexicographic order of the second belonging cluster identifier is the largest, or the second leader node priority is the highest and the lexicographic order of the second belonging cluster identifier is the smallest. It should be noted that the second preset preferred condition can be flexibly set according to actual system requirements, and the core is to ensure the unique determination of the target reorganization leader node in multiple second candidate local leader nodes through a hierarchical comparison mechanism. The specific strategy can be configured according to performance, stability or business scenario requirements, which is not specifically limited here.

[0152] In an exemplary embodiment, referring to Figure 9After the local network of the distributed system is recovered, it is determined that there are two local consensus clusters to be reorganized in the network recovery area, which are Minor:E_cdef(2) and Minor:H_fghi(3) respectively. Both of the two local consensus clusters to be reorganized belong to minority consensus clusters. It can be understood that the five nodes from node D to node H in the network recovery area are bidirectionally connected with each other; the second existence information corresponding to node E and node H respectively can be accepted by the remaining four nodes except themselves. After receiving the second existence information of node H, node D and node E find that the total number of nodes of Minor:H_fghi(3) is greater than the total number of nodes of Minor:E_cdef(2); node H is determined as the target reorganization leader node; node D, node E, node F and node G are determined as the target reorganization follower nodes; according to the following relationship between the target reorganization leader node and the target reorganization follower nodes, the target reorganization consensus cluster corresponding to the network recovery area is established, and Major:H_fghi(5) is obtained. It should be noted that after node D and node E choose to join the Minor:H_fghi(3) where node H is located; the number of nodes corresponding to Minor:H_fghi(3) increases to 5, reaching the cluster majority (Quorum=5), and therefore is promoted to the majority consensus cluster, i.e. the Major consensus cluster, to obtain Major:H_fghi(5). In addition, node D and node E will discard the log submitted in Minor:E_cdef(2) and store and copy the log of Major:H_fghi(5) instead.

[0153] Further, in combination with Figure 9 and Figure 10After the global network of the distributed system is recovered, it is determined that there are two local consensus clusters to be reorganized in the network recovery area, which are Minor: A abcd (3) and Major: H fghi (5) respectively. It can be understood that the eight nodes A to H in the network recovery area are bidirectionally connected with each other; the second existence information corresponding to the node A and the node H can be accepted by the remaining seven nodes except themselves. After receiving the second existence information of the node H, the node A, the node B and the node C find that the total number of nodes of Major: H fghi (5) is greater than that of Minor: A abcd (3); the node H is determined as the target reorganization leader node; the node A, the node B, the node C, the node D, the node E, the node F and the node G are determined as the target reorganization follower nodes; according to the following relationship between the target reorganization leader node and the target reorganization follower nodes, the target reorganization consensus cluster corresponding to the network recovery area is established, and Major: H fghi (8) is obtained. In addition, the node A, the node B and the node C will discard the log submitted in Minor: A abcd (3) and store and copy the log of Major: H fghi (8) instead.

[0154] In the embodiment, when the total number of nodes in the second belonging cluster is not equal, the reorganization role decision is made based on the first reorganization condition, which can ensure the majority dominance and guarantee the data consistency and security.

[0155] In one embodiment, the target reorganization leader node is determined as the target reorganization leader node which satisfies the second reorganization condition, including the following steps:

[0156] Step 1, based on the second log entry index, a third candidate reorganization leader node is determined from at least two reorganization leader nodes.

[0157] It should be noted that the second log entry index is a monotonically increasing integer, therefore, the method of determining the third candidate reorganization leader node from at least two reorganization leader nodes based on the second log entry index can be: the reorganization leader node corresponding to the highest index value of the second log entry index is determined as the third candidate reorganization leader node.

[0158] Step 2, if there is a third candidate reorganization leader node, the third candidate reorganization leader node is determined as the target reorganization leader node.

[0159] Step 3, if there are multiple third candidate reorganization leader nodes, the target reorganization leader node satisfying the second preset preferred condition is determined from the multiple third candidate reorganization leader nodes based on the second leader node priority and the second belonging cluster identifier.

[0160] The second preset preferred condition can be that the second leader node has the highest priority and the largest dictionary order of the second belonging cluster identifier. The second preset preferred condition can also be that the second leader node has the highest priority and the smallest dictionary order of the second belonging cluster identifier. It should be noted that the second preset preferred condition can be flexibly set according to actual system requirements, and the core is to ensure that the target reorganization leader node is uniquely determined in the plurality of second candidate local leader nodes through the hierarchical comparison mechanism. The specific strategy can be configured according to performance, stability or business scenario requirements, and is not specifically limited here.

[0161] Exemplarily, when the total number of second belonging cluster nodes is equal, based on the second log entry index, the second log entry index corresponding to the highest index value is determined as the third candidate reorganization leader node. If there is one third candidate reorganization leader node, the third candidate reorganization leader node is determined as the target reorganization leader node, and all cluster nodes in the network recovery area except the target reorganization leader node are determined as target reorganization follower nodes. Then, according to the following relationship between the target reorganization leader node and the target reorganization follower node, the target reorganization consensus cluster corresponding to the network recovery area is established. If there are a plurality of third candidate reorganization leader nodes, the target reorganization leader node that meets the second preset preferred condition is determined from the plurality of third candidate reorganization leader nodes based on the second leader node priority and the second belonging cluster identifier. For example, the third candidate reorganization leader node with the highest second leader node priority and the largest dictionary order of the second belonging cluster identifier is selected from the plurality of third candidate reorganization leader nodes as the target reorganization leader node.

[0162] In this embodiment, when the total number of second belonging cluster nodes is equal, the role decision of reorganization is made based on the second reorganization condition according to the log entry index and the leader priority, which can ensure the uniqueness and rationality of election.

[0163] In one embodiment, the cluster automatic reorganization method further includes the following steps:

[0164] For each cluster node after network partition of the distributed system, when the cluster node joins the target consensus cluster where the target leader node is located in the following state according to the preset consensus reorganization strategy, all newly added log entries since the network partition are deleted, and the consensus log entries submitted by the target leader node are synchronized.

[0165] The target leader node is a target local leader node or a target reorganization leader node; if the target leader node is a target local leader node, a target consensus cluster where the target leader node is located is determined as a first local consensus cluster; if the target leader node is a target reorganization leader node, a target consensus cluster where the target leader node is located is determined as a target reorganization consensus cluster.

[0166] The new log entry can be a to-be-submitted log record written by a local client under the condition that a cluster node does not reach consensus with other nodes after network partition of a distributed system. In an exemplary embodiment, the new log entry can be used to record a write operation that is attempted to be executed but does not reach consensus during network partition. For example, due to network partition, the new log entry is not confirmed by the target leader node, and therefore does not have global consistency. Further, the new log entry conflicts with a consensus log entry submitted by the target leader node, and needs to be deleted or synchronously overwritten after joining the target consensus cluster because it is not confirmed by a majority of nodes.

[0167] The consensus log entry can be a log entry initiated and successfully submitted by the target leader node in the target consensus cluster, representing the latest data state that has reached consensus in the cluster. The consensus log entry is used to maintain data consistency and service continuity. In an exemplary embodiment, the consensus log entry can be obtained by the target leader node using the Raft standard log replication mechanism to broadcast and confirm that a majority of nodes are written before submission. Further, the consensus log entry is used as a basis for the target follower node to synchronize data and overwrite the local new log entry to ensure consistency.

[0168] In an exemplary embodiment, when a cluster node joins a target consensus cluster corresponding to a target leader node in a follow state according to a preset consensus reorganization strategy, all new log entries added since network partition can be located and cleared by recording a network partition timestamp or a log index range; further, the consensus log entry submitted by the replication target leader node is stored, so as to ensure that the node data state is consistent with the target consensus cluster and has the latest valid data to continue to provide services.

[0169] In this embodiment, by deleting the new log entries added since network partition when a cluster node joins a target consensus cluster, and synchronizing the consensus log entry submitted by the target leader node, the node can clear local data that does not reach consensus during reorganization, and keep consistent with the overall state of the cluster, so as to ensure data consistency and service continuity after network partition is restored.

[0170] In one embodiment, before the cluster node joins the target consensus cluster where the target leader node is located according to the preset consensus reorganization strategy to follow the state, it further includes:

[0171] The cluster node stores all the newly added log entries added after the network partition of the cluster node itself in the target storage area according to a preset storage rule.

[0172] The preset storage rule includes taking the last log entry index corresponding to the cluster node as a storage index, archiving all the newly added log entries added after the network partition of the cluster node itself to generate log archive data, and storing the log archive data in the target storage area according to the storage index. In one exemplary embodiment, the method of archiving all the newly added log entries added after the network partition of the cluster node itself can be implemented by using a trie data structure, but is not limited thereto.

[0173] Exemplarily, when the cluster node switches the cluster ownership (such as cross-cluster reorganization), the cluster node triggers a log snapshot operation, and the snapshot index can be set as the log entry index CID of the consensus cluster to which the cluster node originally belongs. Further, each cluster node can store the logs of all the consensus clusters joined after the network partition in a trie data structure, and clean up the logs of other minority consensus clusters (Minor consensus clusters) after entering the majority consensus cluster (Major consensus cluster). It should be noted that the minority consensus cluster (Minor consensus cluster) cannot form a majority, and this type of cluster does not have the ability to submit new logs independently in the recovery phase, and is mainly used to maintain local availability or wait for merging with the majority consensus cluster (Major consensus cluster).

[0174] It should be noted that in the traditional technology based on the native Raft protocol, nodes in different consensus clusters (i.e. Raft groups) cannot directly join the same cluster, and the nodes can usually only belong to a single consensus group, and cross-group merging or migration requires manual intervention or complete state reset. Based on the cluster automatic reorganization method described in any of the above embodiments, the node is allowed to dynamically join another consensus cluster when the network state changes, provided that the node already belongs to a certain consensus cluster (Raft group). The specific implementation is as follows: when the node needs to switch to a new consensus cluster, a separate Raft instance thread is started inside the node; after the node completes state synchronization in the new consensus cluster and runs stably, the original Raft instance thread is closed, and the associated old Raft group log data is cleaned up; the smooth migration of the node between multiple consensus domains is achieved, the self-adaptation capability of the system in complex network recovery scenarios is improved, and the resource occupation is effectively reduced by releasing the invalid threads and log storage in time, thereby saving the memory and disk space of the node.

[0175] In an example embodiment, the node adds a log entry using a dictionary tree data structure, so that the node can temporarily store the log data generated or received during different Raft groups, avoiding frequent data copying and migration. These log data are stored in isolation according to the Raft group to which they belong. After all nodes re-enter the same Raft group and enter a stable state, the system periodically merges and cleans up the redundant logs, and finally forms a unified and continuous log sequence.

[0176] Taking node D as an example, the log entry archiving process is as shown in Figure 11 : initially (as shown in Figure 3 ), all nodes are in the same consensus domain, and the log entry stored by node D is consistent with that of node A, identified as ID: A abcd, Range [1:4], indicating that this part of the log is based on the CID range [1:4] of Raft group A.

[0177] When network partition occurs (as shown in Figure 4 ), the independent network partition where node D is located is disconnected from the original global consensus set, and it regards itself as an independent Raft group, starts a local consensus process, and generates a new log. At this time, the system creates a new dictionary tree branch for it, records the log entry ID: D bcde, Range [5:5], indicating that this log is generated in the Raft group with node D as the local leader, and depends on the CID range [5:5].

[0178] Subsequently, node D and node E in the same independent network partition complete cluster reorganization (as shown in Figure 5 ), join the Raft group led by node E, continue to append logs, and add new log entry ID: E cdef, Range [5:6], stored in the corresponding branch.

[0179] Further, when the partition is restored to connectivity with another Raft group led by node H (as shown in Figure 9 ), node D joins the Raft group of node H, synchronizes and records new log entry ID: H fghi, Range [4:8].

[0180] Finally, when the global network is restored and the system enters a stable state, node D will perform cross-group merging and deduplication on all log entries based on the log range and CID dependency relationship in each branch of the dictionary tree, and integrate them into a complete and continuous log sequence. The final retained main path log is ID: H fghi, Range [1:8], covering all valid operations from the initial to the latest state.

[0181] Wherein, ID:X represents the Raft group identity to which the log entry belongs, Range[x1:x2] represents the CID interval relied on to form the Raft group consensus, used to identify the context and attribution period of log generation.

[0182] In this embodiment, by archiving and structurally storing the newly added log entries, the generation context of each log segment can be accurately traced after the network partition is restored, the overlapping and conflicting intervals are identified, and then the logs from different local consensus stages are de-duplicated, sorted and integrated in the global logical order, finally forming a continuous, complete and consistent log sequence, avoiding the performance overhead brought by large-scale log replication, and improving the recovery efficiency and storage resource utilization.

[0183] In one embodiment, when the network partition does not occur in the distributed system, the distributed system includes a plurality of cluster nodes; the plurality of cluster nodes include a global leader node and at least one global follower node following the global leader node; the cluster automatic reorganization method further includes the following steps:

[0184] Step 1, periodically broadcast the third existence information corresponding to the global leader node to each global follower node.

[0185] It can be understood that the global leader node and the at least one global follower node jointly constitute a global consensus cluster within the global network. Among them, the global leader node refers to the cluster node that assumes the role of leader when the network partition does not occur in the distributed system. The third existence information at least includes a third belonging cluster identity, a third log entry index, a third belonging cluster node total number, a third leader node priority and a third term information.

[0186] Step 2, if at least one global follower node does not receive the third existence information, it is determined that the network partition occurs in the distributed system.

[0187] Exemplarily, the global leader node periodically broadcasts the third existence information to each global follower node; if the global follower node does not receive the third existence information within a preset timeout time, it is determined that the network partition occurs in the distributed system.

[0188] In this embodiment, by periodically broadcasting the third existence information from the global leader node to each global follower node, and determining that the network partition occurs when at least one global follower node does not receive the third existence information, the cluster state is synchronized by broadcasting the existence information and the network partition determination mechanism is triggered, which can enhance the network partition detection capability, improve the network abnormal response speed of the system, and lay a foundation for quickly establishing a local consensus cluster.

[0189] In one embodiment, the cluster automatic reorganization method further includes the following steps:

[0190] Step 1, after determining that the distributed system has network partition, for each cluster node, based on a preset network awareness strategy, determine a connected cluster node from multiple cluster nodes that is network bidirectional reachable with the cluster node.

[0191] Step 2, according to each cluster node and the connected cluster node that is network bidirectional reachable with each cluster node, determine multiple independent network partitions of the distributed system after network partition.

[0192] Among them, any two cluster nodes in each independent network partition are network bidirectional reachable.

[0193] Among them, the preset network awareness strategy can be but is not limited to network connectivity detection through node broadcasting of own existence information. Each node judges its reachability and active state by receiving existence information from other nodes, and identifies the current connectable node set based on bidirectional communication confirmation, and then divides to form independent network partitions. Among them, the broadcast form of existence information needs to be selected according to the network deployment environment corresponding to the cluster node, which is not limited here.

[0194] In an exemplary embodiment, for the scenario that nodes are not in the same local area network or each server is not configured with a network consensus mechanism, the broadcast form of existence information can be: initiating a Transmission Control Protocol (TCP) connection request to each node in the known node list one by one, judging the reachability of the opposite node based on whether the connection establishment is successful or not, and exchanging node existence information after connection establishment, so as to realize reliable broadcast of existence information and bidirectional connectivity verification, and then identify the connected cluster node from multiple cluster nodes that is network bidirectional reachable with the cluster node. Among them, the network consensus mechanism can include but is not limited to: multiple cluster nodes are preconfigured with the same multicast address and port number.

[0195] In another exemplary embodiment, for the scenario that is in the same local area network and is configured with a network consensus mechanism, the broadcast form of existence information can be: registering and sending User Datagram Protocol (UDP) multicast messages on the predefined multicast address and port to realize fast batch broadcast of node existence information. Each node listens to the multicast message, receives and parses the existence information from other nodes, and verifies the network reachability in combination with the bidirectional communication confirmation mechanism, so as to efficiently determine the connected cluster node from multiple cluster nodes that is network bidirectional reachable with the cluster node. Among them, the network consensus mechanism can include but is not limited to: multiple cluster nodes are preconfigured with the same multicast address and port number.

[0196] In this embodiment, after network partition occurs in the distributed system, the nodes in the interconnected cluster that are bidirectionally reachable with each node are identified based on a preset network perception strategy, and multiple independent network partitions are determined accordingly. In each independent network partition, any two nodes are bidirectionally reachable, which provides a network topology basis for subsequent independent consensus reconstruction, election of target local leader nodes, and establishment of a first local consensus cluster.

[0197] In one specific embodiment, the upper-layer application can autonomously decide whether to provide services to the outside according to whether the node is in the minority consensus cluster, i.e., the Minor consensus cluster, and in combination with actual requirements of the application for consistency and availability. Specifically, if the application can tolerate potential data rollback risks, the application can choose to continue to provide read-write services to prioritize system availability; if the application requires strong consistency and data security, the application can choose to suspend services to prioritize data consistency. Based on this, the system realizes dynamic trade-off support for CAP characteristics, and in abnormal scenarios such as network partition, can maintain service continuity for risk-tolerant businesses and provide consistency protection control capabilities for critical applications, significantly improving the flexibility, adaptability, and controllability of the distributed system in complex network environments.

[0198] In one specific embodiment, when a node decides to join another Raft group, the node attempts to establish a TCP connection with the leader node of the other Raft group and join the Raft group through an RPC request, ensuring that the behavior of joining the Raft group is performed on the premise of bidirectional connectivity.

[0199] It should be noted that the cluster automatic reconstruction method according to any of the above embodiments can split into several small clusters to independently provide services when network partition occurs in the distributed system, thereby improving the availability of the cluster when network partition occurs, and can combine the nodes into a large cluster after the network partition is restored, so that the distributed system has the ability to automatically recover normal operation. Thus, the service capability of the distributed system can be maximized, the degree and time of service degradation when a network fault occurs can be reduced, and a better elastic storage system and fault tolerance and disaster recovery capability can be provided for the upper-layer application.

[0200] At the same time, by embedding the extended Raft instance into the nodes of the distributed system, automatic distributed consensus and cluster perception and management capabilities are realized, which can significantly reduce the deployment complexity, hardware and software environment requirements, and operation and maintenance costs of the cluster, and provide stronger scalability for the distributed system.

[0201] It should be understood that although the steps in the flowcharts related to the embodiments described above are shown in sequence according to the arrows, the steps are not necessarily executed in sequence according to the arrows. Unless explicitly stated herein, the execution of the steps is not strictly limited in sequence, and the steps can be executed in other sequences. Moreover, at least some of the steps in the flowcharts related to the embodiments described above can include multiple steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence of the steps or stages is not necessarily sequential, but can be executed alternately or alternately with at least part of other steps or steps or stages in other steps.

[0202] Based on the same inventive concept, the embodiments of the present application also provide a distributed system for implementing the cluster automatic reorganization method described above. The distributed system is used to implement the steps in the method embodiments described above, and specific limitations can be referred to the limitations of the cluster automatic reorganization method described above, which will not be repeated here.

[0203] In an exemplary embodiment, a computer device, which can be a server, is provided, and an internal structure diagram of the computer device can be as shown in Figure 12 The computer device includes a processor, a memory, an input / output interface (I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The database of the computer device is used to store cluster automatic reorganization related data. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals through network connection. The computer program is executed by the processor to implement a cluster automatic reorganization method.

[0204] Those skilled in the art can understand that Figure 12 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0205] In an embodiment, a computer device is also provided, comprising a memory and a processor, the memory storing a computer program, and the processor implementing the steps in the above method embodiments when executing the computer program.

[0206] In an embodiment, a computer readable storage medium is provided, storing a computer program, and the computer program implementing the steps in the above method embodiments when executed by a processor.

[0207] In an embodiment, a computer program product is provided, comprising a computer program, and the computer program implementing the steps in the above method embodiments when executed by a processor.

[0208] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant regulations.

[0209] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiment methods. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. The non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. The volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, the RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., without being limited thereto.

[0210] The technical features of the above embodiments can be combined in any manner. To make the description concise, not all possible combinations of the technical features in the above embodiments are described, but as long as the combinations of the technical features do not exist contradictions, they should be considered as the scope of the present application.

[0211] The above-described embodiments are merely illustrative of several embodiments of the present application, which are described in more detail and in a specific manner, but should not be construed as limiting the scope of the patent of the present application. It should be noted that, for those of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A method for automatic cluster reassembly, characterized in that, Applied to distributed systems, the method includes: Identify multiple independent network partitions after a network partition occurs in the distributed system; For each of the independent network partitions, determine the first existence information of each cluster node in each of the independent network partitions; Based on the first existence information, the target local leader node and the target follower node are determined from the cluster nodes of the independent network partition according to the preset consensus reorganization strategy; Based on the following relationship between the target local leader node and the target follower node, a first local consensus cluster corresponding to the independent network partition is established.

2. The method according to claim 1, characterized in that, The preset consensus reorganization strategy includes a first preset consensus reorganization strategy; the step of determining the target local leader node and the target follower node from the cluster nodes of the independent network partition according to the preset consensus reorganization strategy based on each of the first existence information includes: If a global leader node exists in the cluster nodes of the independent network partition, and the state of the global leader node is valid, then the global leader node is determined as the target local leader node, and all the cluster nodes in the independent network partition except for the global leader node are determined as target follower nodes. If all cluster nodes of the independent network partition are candidate nodes, then based on the first existence information, the target local leader node and the target follower node are determined from the candidate nodes of the independent network partition according to the first preset consensus reorganization strategy.

3. The method according to claim 2, characterized in that, The first existence information includes the first cluster identifier, the first log entry index, the total number of nodes in the first cluster, and the priority of the first leader node; the step of determining the target local leader node and the target follower node from the candidate nodes of the independent network partition based on each of the first existence information and according to the first preset consensus reorganization strategy includes: Based on the first log entry index, a first candidate local leader node is determined from the candidate nodes of the independent network partition; If a first candidate local leader node exists, then the first candidate local leader is determined as the target local leader node; If there are multiple first candidate local leader nodes, then based on the priority of the first leader node and the first cluster identifier, a target local leader node that meets the first preset preference condition is determined from the multiple first candidate local leader nodes. All candidate nodes in the independent network partition, except for the target local leader node, are identified as target follower nodes.

4. The method according to claim 1, characterized in that, The preset consensus reorganization strategy includes a second preset consensus reorganization strategy; the method further includes: If at least one network recovery zone is detected in the distributed system, at least two local consensus clusters to be reassembled within the network recovery zone are identified. According to the second preset consensus reorganization strategy, the target reorganization leader node and the target reorganization follower node are determined from at least two of the local consensus clusters to be reorganized; Based on the following relationship between the target reorganization leader node and the target reorganization follower nodes, a target reorganization consensus cluster corresponding to the network recovery zone is established.

5. The method according to claim 4, characterized in that, The step of determining the target reorganization leader node and the target reorganization follower node from at least two of the local consensus clusters to be reorganized according to the second preset consensus reorganization strategy includes: Determine the second existence information corresponding to the leader node to be reorganized in each of the local consensus clusters to be reorganized; the second existence information includes the second cluster identifier, the second log entry index, the total number of nodes in the second cluster, and the priority of the second leader node. If the total number of nodes in each of the second clusters is different, the leader node to be reorganized that meets the first reorganization condition is determined as the target reorganization leader node, and all cluster nodes in the network recovery area except the target reorganization leader node are determined as target reorganization follower nodes. If the total number of nodes in each of the second-to-be-reorganized clusters is the same, then the leader node to be reorganized that meets the second reorganization condition is determined as the target reorganization leader node, and all cluster nodes in the network recovery zone other than the target reorganization leader node are determined as target reorganization follower nodes.

6. The method according to claim 5, characterized in that, The step of determining the leader node to be reorganized that meets the first reorganization condition as the target reorganization leader node includes: Based on the second log entry index, a first candidate reorganization leader node is determined from at least two leader nodes to be reorganized; If there exists a first candidate reorganization leader node, then the first candidate reorganization leader node is determined as the target reorganization leader node; If there are multiple first candidate reorganization leader nodes, the target reorganization leader node is determined from the multiple first candidate reorganization leader nodes based on the total number of nodes in the second affiliated cluster, the priority of the second leader node, and the identifier of the second affiliated cluster.

7. The method according to claim 6, characterized in that, The step of determining the target reorganization leader node from multiple first candidate reorganization leader nodes based on the total number of nodes in the second affiliated cluster, the priority of the second leader node, and the identifier of the second affiliated cluster includes: Based on the total number of nodes in the second affiliated cluster, the first candidate reorganization leader node with the highest total number of nodes in the second affiliated cluster among the multiple first candidate reorganization leader nodes is determined as the second candidate reorganization leader node; If a second candidate reorganization leader node exists, then the second candidate reorganization leader node is determined as the target reorganization leader node; If there are multiple second candidate reorganization leader nodes, then based on the priority of the second leader node and the second cluster identifier, the target reorganization leader node that meets the second preset preference condition is determined from the multiple second candidate reorganization leader nodes.

8. The method according to claim 5, characterized in that, The step of determining the leader node to be reorganized that meets the second reorganization condition as the target reorganization leader node includes: Based on the second log entry index, a third candidate reorganization leader node is determined from at least two of the proposed reorganization leader nodes; If a third candidate reorganization leader node exists, then the third candidate reorganization leader node is determined as the target reorganization leader node; If there are multiple third candidate reorganization leader nodes, then based on the priority of the second leader node and the second cluster identifier, a target reorganization leader node that meets the second preset preference condition is determined from the multiple third candidate reorganization leader nodes.

9. The method according to claim 1, characterized in that, The method further includes: For each cluster node after a network partition occurs in the distributed system, when the cluster node joins the target consensus cluster where the corresponding target leader node is located in a follow state according to the preset consensus reorganization strategy, it deletes all new log entries added since the network partition occurred and synchronizes the consensus log entries submitted by the target leader node.

10. The method according to claim 9, characterized in that, Before the cluster nodes join the target consensus cluster where the corresponding target leader node resides in a follow state according to the preset consensus reorganization strategy, the method further includes: All new log entries added by the cluster node after the network partition occurred are stored in the target storage area according to the preset storage rules. The preset storage rules include using the index of the last log entry corresponding to the cluster node as the storage index, archiving all new log entries added after the network partition occurs, generating log archive data, and storing the log archive data in the target storage area according to the storage index.

11. The method according to claim 1, characterized in that, In the absence of network partitioning in the distributed system, the distributed system comprises multiple cluster nodes; the multiple cluster nodes include a global leader node and at least one global follower node following the global leader node; the method further includes: The third existence information corresponding to the global leader node is periodically broadcast to each of the global follower nodes; If at least one of the global follower nodes has not received the third presence information, then the distributed system is determined to have experienced a network partition.

12. The method according to claim 1, characterized in that, The method further includes: After determining that a network partition has occurred in the distributed system, for each cluster node, based on a preset network awareness strategy, a connected cluster node that is bidirectionally reachable from the multiple cluster nodes is identified. Based on each cluster node and the connected cluster nodes that are bidirectionally reachable from each cluster node, multiple independent network partitions are determined after the network partitioning of the distributed system; any two cluster nodes in each independent network partition are bidirectionally reachable from each other.

13. A distributed system, characterized in that, The distributed system is used to perform the steps of the method according to any one of claims 1 to 12.

14. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 12.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 12.

16. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 12.

Images