Node failure detection and resolution in distributed databases

Abstract

To provide methods and systems to detect and resolve failure in a distributed database system.SOLUTION: A first node in a distributed database system can detect an interruption in communication with at least one other node in the distributed database system. This indicates a network failure. In response to detection of this failure, the first node starts a failure resolution protocol. This invokes coordinated broadcasts of respective lists of suspicious nodes among neighbor nodes. Each node compares its own list of suspicious nodes with its neighbors' lists of suspicious nodes to determine which nodes are still directly connected to each other. Each node determines the largest group of these directly connected nodes and whether or not it exists in that group. If a node does not exist in that group, it fails itself to resolve the network failure.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of priority under 35 U.S.C. §119(e) to U.S. Application No. 62 / 800,009, entitled “Node Failure Detection and Resolution,” filed February 1, 2019, the disclosure of which is incorporated herein by reference in its entirety. [Background technology]

[0002] Data and metadata in a distributed database are stored across multiple nodes that communicate with each other. However, communication interruptions may occur between the nodes. For example, a node in a distributed database system may find itself in an inconsistent state and crash or fail. In another example, a virtual machine or process running on a node in a distributed database system may crash or fail. In yet another example, a communication link between a first node and a second node in a distributed database system may fail. For example, a network (e.g., a local area network, a wide area network, an Ethernet, etc.) connecting two or more nodes in a distributed database system may fail, interrupting communication between the nodes. Summary of the Invention

[0003] Described herein is a distributed database system. The distributed database system may include a plurality of nodes. Each node in the plurality of nodes may include a corresponding processor and a corresponding memory. Each node in the plurality of nodes may be connected to all other nodes in the plurality of nodes. A processor of a first node in the plurality of nodes may be configured to resolve a failure in the distributed database system by identifying a suspect node in the plurality of nodes, broadcasting a list of the first suspect node to adjacent nodes in the plurality of nodes, receiving a list of a second suspect node from at least one other adjacent node, determining whether the first node is within a winning fully connected component of the distributed database based on the connectivity information, allowing the first node to continue operation in response to determining that the first node is within the winning fully connected component of the plurality of nodes, and taking down the first node to resolve the failure in response to determining that the first node is not within the winning fully connected component of the plurality of nodes. The suspect node may be a node in the plurality of nodes that is no longer connected to the first node as a result of the failure in the distributed database system. The list of first suspect nodes can include the suspect node. The neighboring nodes can be nodes in the plurality of nodes that remain directly connected to the first node after the network failure. The winning all-connected component can include more than half of the nodes in the plurality of nodes, and each node in the winning all-connected component nodes is directly connected to each other node in the winning all-connected component nodes.

[0004] Described herein is a method for resolving failures in a distributed database. The distributed database can include a plurality of nodes, where each node in the plurality of nodes can be directly connected to each other node in the plurality of nodes. The method can include detecting, at a first node in the plurality of nodes, an interruption in communication with a second node in the plurality of nodes, initiating a cooperative broadcast among adjacent nodes in the plurality of nodes of a list of respective suspect nodes in response to detecting the interruption, determining connectivity information based on the lists of respective suspect nodes, and resolving the failure based at least in part on the connectivity information. The adjacent nodes can be nodes in the plurality of nodes that remain directly connected to the first node. The list of suspect nodes of the first node includes the second node.

[0005] Described herein is a method for resolving failures in a distributed database, where the distributed database may include a plurality of nodes, each of which may be connected to each of the other nodes of the plurality of nodes.The method includes, in response to detecting a failure, determining at a first node in the plurality of nodes whether the first node is connected to at least half of the nodes in the plurality of nodes; in response to determining that the first node is directly connected to less than half of the nodes in the plurality of nodes, shutting down the first node to at least partially resolve the failure; in response to determining that the first node is directly connected to at least half of the nodes in the plurality of nodes, broadcasting a list of first suspect nodes to neighboring nodes in the plurality of nodes; receiving a list of second suspect nodes from at least one of the neighboring nodes; determining whether the list of first suspect nodes matches the list of second suspect nodes; and in response to determining that the list of first suspect nodes matches the list of second suspect nodes, maintaining the first node operational in at least partially resolving the failure; The method may include broadcasting a first updated list of suspect nodes to neighboring nodes based on the first list of suspect nodes and the second list of suspect nodes in response to determining that the first node is not in the winning all-connected component of the plurality of nodes, receiving at least one second updated list of suspect nodes from at least one of the neighboring nodes, determining connectivity information of the plurality of nodes based at least in part on the first updated list of suspect nodes and the second updated list of suspect nodes, determining a winning all-connected component of the distributed database based on the connectivity information, determining whether the first node is in the winning all-connected component, allowing the first node to continue operation to at least partially resolve the failure in response to determining that the first node is not in the winning all-connected component of the plurality of nodes, and taking the first node out of service to at least partially resolve the failure in response to determining that the first node is not in the winning all-connected component of the plurality of nodes. The first list of suspect nodes may include nodes that are not directly connected to the first node. The neighboring nodes may be nodes that remain directly connected to the first node after the failure.The winning all-connected component includes more than half of the nodes in the plurality of nodes, and every node in the winning all-connected component is directly connected to every other node in the winning all-connected component.

[0006] All combinations of the above and further concepts are discussed in more detail below (provided such concepts are not mutually inconsistent) and are part of the inventive subject matter disclosed herein. In particular, all combinations of claimed subject matter at the end of this disclosure are part of the inventive subject matter disclosed herein. Terms used herein that may also appear in any disclosures incorporated by reference shall be given the meaning most consistent with the specific concepts disclosed herein.

[0007] As will be appreciated by those skilled in the art, the drawings are primarily for illustrative purposes and are not intended to limit the scope of the inventive subject matter described herein. The drawings are not necessarily to scale, and in some cases, various aspects of the inventive subject matter disclosed herein may be exaggerated or enlarged in the drawings to facilitate an understanding of different features. In the drawings, like reference characters typically refer to like features (e.g., functionally similar and / or structurally similar elements). [Brief description of the drawings]

[0008] [Figure 1] FIG. 2 illustrates a process for resolving a network failure and restoring full connectivity between nodes in a distributed database. [Diagram 2] FIG. 1 illustrates an example of a typical case of network failure in a distributed database system, where a network partition event splits the distributed database system into two disjoint groups of fully connected nodes. [Diagram 3] FIG. 2 illustrates an exemplary distributed database system of three nodes with partial connections that can be resolved by the process shown in FIG. 1. [Figure 4]FIG. 2 illustrates an exemplary distributed database system of five nodes with two link failures that can be resolved by the process shown in FIG. 1. [Diagram 5] FIG. 2 illustrates an exemplary distributed database system of five nodes with four link failures that can be resolved by the process shown in FIG. 1. [Figure 6] FIG. 6 is a diagram illustrating an example of a case of partial connection illustrated in FIG. 5. [Figure 7] FIG. 2 illustrates an exemplary distributed database system of five nodes with three link failures that can be resolved by the process shown in FIG. 1. [Figure 8] FIG. 2 illustrates an example distributed database system of five nodes with five link failures that can be resolved by the process shown in FIG. 1. [Figure 9] FIG. 2 illustrates an exemplary case of a special partial connection having a unidirectional link failure that can be resolved by the process illustrated in FIG. 1. [Figure 10] FIG. 2 illustrates an exemplary case of a network failure during a membership change that can be resolved by the process illustrated in FIG. 1. [Figure 11] FIG. 2 is a diagram illustrating an example of the process shown in FIG. [Figure 12] 1 is a flow chart illustrating an enhanced process for resolving a network failure. [Figure 13] FIG. 2 illustrates membership changes in a distributed database system when a network partition separates the new node and the entry node from the remaining nodes in the distributed database system. [Figure 14] FIG. 14 illustrates a variation of the scenario of FIG. 13 when a network partition isolates the new node and the entry node from the remaining nodes in the distributed database system. [Figure 15]A diagram illustrating membership changes in a distributed database system when a network partition separates a new node, an entry node, and some peers from the remaining peers in the distributed database system. [Figure 16] FIG. 2 illustrates membership changes in a distributed database system when a network partition separates an entry node from the remaining nodes in the distributed database system. [Figure 17] A diagram illustrating membership changes in a distributed database system when a network partition separates a new node, an entry node, and some peers from the remaining nodes in the distributed database system. [Figure 18] FIG. 2 is another diagram of membership changes in a distributed database system when a network partition isolates the new node, the entry node, and some peers from the remaining nodes in the distributed database system. [Figure 19] FIG. 2 illustrates nodes exchanging failure detection messages to resolve network failure events. [Figure 20] FIG. 1 illustrates how to handle a node failure while exchanging failure detection messages. [Figure 21] FIG. 1 illustrates handling node failures while performing failover. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] A distributed database system includes multiple nodes that store fragments of data and / or metadata for the distributed database. All nodes in a distributed database system are directly connected to each other so that they can communicate with each other. However, one or more nodes in a distributed database system may experience communication interruptions due to network failures. These communication interruptions may be due to failure of a communication link between two or more nodes or failure of one or more nodes. These failures may be resolved by identifying the nodes that are still directly connected to each other, determining the largest group of directly connected nodes, and disabling nodes that are not part of that group, as described in more detail below.

[0010] Distributed Database System A distributed database system can include two types of nodes: transaction engine (TE) nodes, which provide user access to the distributed database, and storage manager (SM) nodes, which maintain respective disk archives of the entire distributed database. Each storage manager node typically stores a copy of the entire distributed database, but a single transaction engine node may contain only the portion of the distributed database necessary to support transactions currently being executed on that transaction engine node.

[0011] Each node in the distributed database system has its own processor, memory, and communication interface(s) and can communicate directly with every other node in the distributed database system over the database system network. Communication between any two nodes can include the transmission of serialized messages. The serialized messages can follow the Transmission Control Protocol (TCP) or any other suitable messaging protocol.

[0012] Each node in the distributed database system has a unique identifier (e.g., a lexicographical ID) and stores a list of all other nodes in the distributed database system in order of their unique identifiers. Each node uses this list to track the status of all transaction engine nodes and storage manager nodes in the distributed database system. In addition, each node may track all database transactions and the location of all database records (i.e., which node stores which data fragments). The nodes may store this node and transaction information in their respective copies of a master catalog, which contains metadata about the distributed database system and is replicated to all nodes in the database. When a new node joins the distributed database system, it receives a copy of the master catalog from another node, called an entry node.

[0013] Tracking database transactions and the location of database fragments helps a distributed database system maintain atomicity, consistency, isolation, and durability, commonly known as the ACID properties, to ensure the accuracy, completeness, and integrity of the data in a distributed database.

[0014] Network faults and fault detection Each node in the distributed database system sends a "heartbeat" message to every other node in the distributed database system at frequent intervals. For example, each node sends a heartbeat message to every other node every second or every few seconds. (Optionally, a node that receives a heartbeat message may send an acknowledgment message to the node that sent the heartbeat message.) In the absence of communication interruptions, every node in the distributed database system continues to send heartbeat messages directly to, and receive heartbeats directly from, every other node in the distributed database system. However, network failures may interrupt such communication. A node that detects an interruption in communication (e.g., one that has not received a heartbeat message from other nodes within a predetermined time) initiates a failure resolution protocol to resolve the network failure.

[0015] Resolving network issues In the failure resolution procedures presented here, nodes in a distributed database regroup themselves in response to a network failure and fail if they are not part of the largest fully connected group of nodes of majority size that is furthest in lexicographical ID ranking. If the largest fully connected group contains less than half the nodes in the distributed database system, all nodes may fail themselves. By failing disconnected or partially connected nodes, it is less likely that part or all of the database may become invalid. These failure resolution procedures can be performed in a leaderless manner and without blocking or aborting ongoing database transactions.

[0016] FIG. 1 illustrates a process 100 for resolving a network failure. Any node in a distributed database system may initiate the process 100 in response to detecting a network failure (e.g., failure to receive heartbeat messages from other nodes within a predetermined period of time). At 102, a first node detects the network failure and initiates the fault resolution process 100 by creating a list of "suspect nodes," i.e., nodes that the first node suspects to have failed. For example, the first node's suspect list is a list of nodes that satisfy one or both of the following conditions: (a) the first node has not received heartbeat messages from them within a predetermined timeout interval (e.g., pingTimeout seconds), and (b) the operating system has closed the connection(s) between the first node and the other node(s). At this point, if the first node's suspect list includes all other nodes in the distributed database system, the first node may take itself down to at least partially resolve the network failure.

[0017] At 104, the first node (i.e., the node that initiated process 100) broadcasts its suspect node list to its neighboring nodes, which are nodes with which the first node can still directly communicate after the network failure. (In the absence of a network failure, in a distributed database, every node is a neighbor of every other node.) The neighboring nodes receive this suspect list and broadcast their own suspect lists to their neighbors. The neighboring nodes' suspect lists may be the same as or different from the first node's suspect list, depending on the nature of the network failure.

[0018] At 106, the first node receives suspect lists from its neighboring nodes and uses them and its own suspect lists to build a connection graph. The connection graph indicates which nodes in the distributed database system the first node is actually directly connected to (i.e., which nodes are actually neighbors of the first node). The other nodes also build connection graphs. Depending on the nature of the network failure, these connection graphs may be the same as the first node's connection graph or may be different. Similarly, each connection graph may complement the corresponding node's suspect list.

[0019] Each node uses its connectivity graph to identify groups of nodes that remain directly connected to each other after the network failure. Each group of directly connected nodes is called an "all-connected component." Within an all-connected component, each node continues to communicate with all other nodes in the all-connected component after the network failure. Once each node has identified an all-connected component in the distributed database system, it determines (110) whether it is part of a "winning all-connected component." If it is not part of an all-connected component, each node shuts itself down (112) to resolve the network failure. If it is part of a winning all-connected component, it continues to operate (114).

[0020] The winning all connected component may include all data in the database, but is not required to do so (e.g., it does not have to include a storage manager node). This procedure does not take into account the types of nodes that form the winning all connected component. (However, in some cases, this process can be modified to pay attention to the types of nodes in an all connected component when determining the winning all connected component.) If the winning all connected component does not include all data in the distributed database, the user may intervene to ensure proper operation.

[0021] Each node may determine whether it is part of a winning all-connected component as follows: First, each node may determine whether it is part of the all-connected component based on its connectivity graph. If not, it shuts itself down. However, if the node is part of the all-connected component (or possibly more than one all-connected component), the node determines the size of its all-connected component(s) based on its connectivity graph. If the node determines (based on its connectivity graph and the information each node stores about other nodes in the distributed database system) that it is not part of the largest all-connected component, it shuts itself down (112). If the node is part of the largest all-connected component, and that all-connected component includes more than half of the total number of nodes in the distributed database system before the network failure, the node remains operational (114). This all-connected component is called the "winning all-connected component" because it includes all operational nodes in the distributed database system at the end of the fault resolution process 100.

[0022] If a node determines that there are two or more all-connected components of the same size, each with more than half of the nodes in the distributed database, and larger than all other all-connected components, the node performs a tie-breaking process to identify a winning all-connected component. The tie-breaking process may include sorting the nodes in each all-connected component in order of the nodes' unique identifiers. Once the unique identifiers are sorted, the node selects a winning all-connected component based on the lexicographical order of the unique identifiers. For example, the node may select the all-connected component with the smallest node ID following a common prefix as the winning all-connected component.

[0023] Technical advantages over other troubleshooting processes The failure resolution process shown in Figure 1 has several differences and advantages over other processes for resolving failures in a distributed database. First, unlike blocking processes, the failure resolution process shown in Figure 1 removes one or more nodes in the distributed database after a network failure to restore full connectivity. Blocking is undesirable because it may roll back updates made to data in the distributed database. Unlike other methodologies, the process described herein does not include any kind of blocking mechanism.

[0024] Moreover, the failure resolution process shown in FIG. 1 does not require or use a leader node. Conversely, other methodologies for resolving failures in distributed databases implement a strong leadership model. Essentially, the methodologies use a leader node to make failure resolution decisions. Unlike this leader-based methodology, the process described herein does not have a leader node to make failure resolution decisions. Instead, as described above with respect to FIG. 1, any node can initiate the failure resolution process, and each node determines, as part of the process, whether it should shut itself down or remain operational without instructions from a leader node.

[0025] Unlike blocking and leader-based failure resolution processes, the non-blocking leader-less failure resolution process described herein can address partially connected network failures in a consistent manner. In partially connected network failures, network partitions in a distributed database system may cause a node or set of nodes to only communicate with a subset of the nodes in the distributed database system. To address the partially connected case, other processes apply a rotating leader model, making leaders and informers use explicit message acknowledgments. In some cases, leadership may continually shift between nodes experiencing communication interruptions, delaying (possibly indefinitely) the resolution of the network failure.

[0026] Various cases of network failure Process 100 does not allow two or more disjoint node groups (i.e., different fully connected components) to continue operation after a network failure event. To avoid trivial solutions (e.g., taking down all nodes), process 100 allows a single node group to continue operation when possible.

[0027] Furthermore, if a user chooses to shut down more than half of the surviving nodes in a distributed database system, the process 100 does not necessarily have to take down the remaining nodes. The process 100 can also handle slow links (i.e., communication paths between two or more nodes that have a slow connection) in addition to link failures. In other words, the process 100 handles slow links and link failures in the same manner.

[0028] 2-10 illustrate various types of network faults that can be resolved using the process 100 of FIG.

[0029] Case A: Figure 2 illustrates a typical failure case in which a network partition event splits the distributed database system 200 into two or more disjoint groups of fully connected nodes. As seen on the left side of Figure 2, the distributed database system 200 includes three transaction engine nodes TE1, TE2, and TE3, and two storage manager nodes SM1 and SM2, which are all interconnected. These nodes communicate with each other via respective communication links 212a-212j, TE1 communicates with TE2 via link 212a, TE2 communicates with TE3 via link 212d, TE3 communicates with SM2 via link 212e, SM2 communicates with SM1 via link 212f, TE2 communicates with SM1 via link 212b, TE2 communicates with SM1 via link 212c, TE1 communicates with SM2 via link 212h, TE3 communicates with SM1 via link 212j, and TE2 communicates with SM2 via link 212i.

[0030] In the center of FIG. 2, a network partition splits the chorus into two disjoint groups of nodes (two fully connected components 202′ and 202″). (A chorus or chorus group is the set of all nodes in a distributed database system.) In this example, the first fully connected component 202′ includes {TE1, TE2, SM1} and the second fully connected component 202″ includes {TE3, SM2}. The process 100 then determines that the first fully connected component 202′ is the winning fully connected component 204′ because it is larger than the second fully connected component 202″ and includes more than half of the nodes in the distributed database 200. The nodes {TE1, TE2, SM1} in the winning fully connected component 204′ continue to operate, while nodes TE3 and SM2 shut themselves down in response to realizing that they are not in the winning fully connected component 204′.

[0031] Case B: Figures 3-8 show various examples of partial connectivity. In each of these examples, a network partition or (bidirectional) link failure(s) causes a node or set of nodes to communicate with only a subset of other nodes in the distributed database system. In the partial connectivity cases, the connections between nodes do not satisfy the transitive property, i.e., for example, node TE1 may be able to communicate directly with node TE2, and node TE2 can communicate directly with node SM1, but node SM1 cannot communicate directly with node TE1.

[0032] Example B1: Figure 3 shows a distributed database system 300 with three nodes TE1, TE2, and SM1. As seen in Figure 3, the three nodes TE1, TE2, and SM1 form a chorus group, where TE1 communicates with TE2 via link 212a, TE2 communicates with SM1 via link 212b, and SM1 communicates with TE1 via link 212c. Due to the failure of link 212a between TE1 and TE2 (or a network partition between the data centers of TE1 and TE2, assuming that TE1 and TE2 are in different data centers), two fully connected components 202'{SM1, TE1} and 202''{SM1, TE2} are created with partial connections of nodes TE1 and TE2. Because the fully connected components 202' and 202'' are of the same size and have more than half the number of nodes that were up before the link failure, these nodes perform a tie-breaking process, such as the lexicographical ordering discussed above, to determine the winning fully connected component 204'. In FIG. 3, {SM1, TE1} is the winning all-connected component 204' (as determined by a tie-breaking process such as lexicographical order), so SM1 and TE1 continue to run and TE2 shuts itself down.

[0033] Example B2: FIG. 4 shows a distributed database system 400 with a chorus group of five nodes TE1, TE2, TE3, SM1, and SM2. In this example, two link failures occur, one between SM1 and SM2 (link 212f) and another between SM2 and TE3 (link 212e). These failures create fully connected components 402′{TE1, TE2, TE3, SM1} and 402″{TE1, TE2, SM2}, where node SM2 is partially connected to the other nodes and the other nodes remain directly connected to each other. The first fully connected component 402′{TE1, TE2, TE3, SM1} is the winning fully connected component 404′ because it contains more than half the nodes and is larger than the other winning fully connected components 402″. Node SM2 fails and the other nodes continue to operate.

[0034] Example B3: Figure 5 shows a five-node distributed database system 500 with a chorus group of five nodes TE1, TE2, TE3, SM1, and SM2 suffering from four link failures. In this example, four link failures occur between TE1 and SM1 (link 212c), between TE1 and SM2 (link 212h), between TE2 and TE3 (link 212g), and between TE3 and SM1 (link 212j). These failures result in several fully connected components, but only one has at least three nodes {TE2, SM1, SM2}, which is shown on the right. Nodes TE1 and TE3 remain partially connected to the distributed database, but cannot directly communicate with all other nodes in the distributed database. As a result, nodes TE1 and TE3 fail, leaving {TE2, SM1, SM2} as the winning fully connected component 404'.

[0035] Figure 6 illustrates why the partially connected case of Figure 5 cannot be addressed using the rotating leader model methodology. As shown in Figure 6, five nodes TE1, TE2, TE3, SM1, and SM2 form a group in step 1 (left). In step 1, all these nodes can communicate with each other without interruption. However, as shown in step 2 (right), communication links fail between TE1 and SM1 (link 212c), between TE1 and SM2 (link 212h), between TE2 and TE3 (link 212g), and between TE3 and SM1 (link 212j). These failures interrupt direct communication between TE1 and SM1, between TE1 and SM2, between TE2 and TE3, and between TE3 and SM1.

[0036] SM1 is the current leader just before the network failure. Based on the rotating leader methodology after the link failure, SM1 continues to assume that it is the leader because it receives heartbeat messages from TE2 and SM2. Due to the link failure between TE1 and SM1 (link 212c), TE1 rotates leadership to TE2 because TE1 does not receive heartbeat messages from SM1. Similarly, due to the link failure between TE3 and SM1 (link 212j), TE3 rotates leadership to TE1. Thus, SM1, TE2, and TE1 take leadership in succession (not necessarily in this order), but TE1 does not even know if SM1 is connected to SM2 because it is not connected to either SM1 or SM2. This rotating leadership makes it difficult to resolve the failure(s).

[0037] Conceptually, as noted above, it is difficult for a centralized leader-based solution to adequately handle partial connectivity cases, since the leader node may not be connected to all other nodes (and thus may not know the connectivity information of all other nodes). However, the leaderless failure resolution process described herein is an improvement over leader-based failure resolution methods, as it handles all of these partial connectivity cases in a reliable manner.

[0038] Example B4: FIG. 7 shows a distributed database system 700 with a chorus group of five nodes with three link failures, one between TE1 and SM1 (link 212c), another between TE2 and TE3 (link 212g), and another between SM1 and TE3 (link 212j). These failures generate fully connected components {TE1, TE2, SM2}, {TE2, SM1, SM2}, and {TE1, SM2, TE3}, and nodes TE1 and TE2 are partially connected. Each of these three fully connected components contains more than half the number of nodes in the chorus group before the link failures. Furthermore, these three fully connected majority groups are of the same size. Thus, the nodes perform a tie-breaking process, such as lexicographic ordering, to identify the winning fully connected component 704'. In this example, {TE2, SM1, SM2} is the winning fully connected component 704' (as determined by the tie-breaking process). Therefore, nodes TE1 and TE3 disable themselves to resolve the network failure.

[0039] Example B5: Figure 8 shows a distributed database system 800 with a chorus group of five nodes TE1, TE2, TE3, SM1, and SM2. In this example, five link failures occur between TE1 and SM1 (link 212c), between TE1 and SM2 (link 212h), between TE2 and TE3 (link 212g), between TE2 and SM2 (link 212i), and between TE3 and SM1 (link 212j). As can be seen from Figure 8, after these failures there are five fully connected node groups, each with a size of two nodes. This is less than half the number of nodes in the chorus group before the link failures. Therefore, since there is no fully connected majority group after the link failures, all the nodes shut themselves down.

[0040] Case C: Figure 9 illustrates a special case of partial connectivity where a (unidirectional) link failure(s) causes a node or set of nodes to communicate with a subset of other nodes in one direction but not in the other direction. As seen in Figure 9, three nodes TE1, TE2, and SM1 in a distributed database system 900 form a chorus group. However, a unidirectional link failure occurs between TE1 and TE2 (link 212a''), such that TE2 can send messages to TE1 (link 212a'), but TE1 cannot send messages to TE2 (link 212a''). This unidirectional link failure (as well as the bidirectional link failure between TE1 and TE2) creates fully connected components 902'{TE1, SM1} and 902''{TE2, SM1}. Since the two sets of fully connected components are of equal size and contain more than half the number of nodes that were up before the link failure (i.e., two out of the three total nodes), the nodes perform a tie-breaking process to determine a winning fully connected component 904'. In this example, {TE1, SM1} is the winning all-connected component 904' (as determined by the tie-breaking process above). Therefore, nodes TE1 and SM1 continue to run, and node TE2 takes itself out of service.

[0041] Case D: The process 100 also prevents the distributed database system from splitting into multiple majority groups due to a network failure occurring during a membership change. A membership change refers to a new node joining the distributed database system or an existing node of the distributed database system leaving the distributed database system. Figure 10 shows an example of case D. In this example, a chorus 1000 starts with three nodes TE1, SM1, and SM2. Two nodes TE2 and TE3 try to join the chorus 1000. While they are joining, a network partition occurs and the distributed database is separated into all connected components 1002'{TE2, TE3, TE1} and 1002''{SM1, SM2}. Both groups may continue to run because the members of group {TE2, TE3, TE1} consider themselves to be a majority because they are part of chorus {TE2, TE3, TE1, SM1, SM2}, and the members of group {SM1, SM2} consider themselves to be a majority because they are part of chorus {TE1, SM1, SM2}. Process 100 ensures that only one group continues to run. In other words, process 100 ensures that both {TE2, TE3, TE1} and {SM1, SM2} do not continue to run at the same time.

[0042] Collecting and sharing information about suspect nodes The failure resolution process disclosed herein (e.g., process 100) is a leaderless process. In response to a network failure event, each node identifies its own suspect list, exchanges connection information (its own and optionally that of the other nodes) with the other nodes, and then makes a failure resolution decision. The process causes the nodes to communicate and exchange connection information in such a way that at the end of the communication phase of the process, each node has sufficient connection information about the other nodes in its partition to arrive at the same failure resolution decision(s). Any new network failure event that occurs while the protocol is in progress causes all nodes to restart the protocol.

[0043] In general, the fault resolution process of the present invention may include two phases: Phase 1, in which each node collects information regarding the suspect lists / connections of other nodes, and Phase 2, in which each node makes a fault resolution decision (e.g., takes itself down) based on the information collected during Phase 1.

[0044] During phase 1, each node participates in at most two rounds of cooperative broadcasts. These cooperative broadcasts of suspect lists include an exchange of connectivity information / suspect lists between nodes in the partition. In case A above, each node performs one cooperative broadcast. In cases B and C above, each node performs two cooperative broadcasts. Two rounds of cooperative broadcasts are sufficient for all nodes to agree on group membership changes in cases A, B, and C.

[0045] To provide an intuitive understanding of this process, we first present below an unoptimized connection information exchange process, which involves (n-1) rounds of broadcasting, where n is the number of nodes in the chorus during phase 1. Then we present below an optimized version of the connection information exchange process, in which each node participates in at most 2 rounds of broadcasting, regardless of the number of nodes in the chorus.

[0046] For clarity and simplicity, we assume that there are no new network failure events, no new nodes joining, and no chorus member node failures while the connection information exchange process is in progress. However, the connection information exchange process described herein can be extended to all of these events. These assumptions and / or limitations are lifted in a later section after the core process is described.

[0047] Non-optimal distribution of suspected node list First, a chorus contains n all connected nodes. Suppose a network failure event occurs. Each node executes the following protocol to resolve the network failure event.

[0048] Each node maintains its own suspect list (which may be an empty list, which is possible if the node is fully connected to all other nodes (or at least thinks it is) after a network failure event).

[0049] Phase 1: Each node performs (n-1) rounds of cooperative broadcasting to gather information about other nodes' suspect lists / connections. In round 1, each node transmits its suspect list to its neighbors and waits until it receives the suspect lists of its neighbors. In rounds 2 through (n-1), each node transmits the suspect lists of other nodes received in the previous round to its neighbors and waits until it receives such information from its neighbors.

[0050] Phase 2: Each node now has received the connectivity information of all other nodes in its partition (since a chorus contains n nodes, a node will broadcast (n-1) rounds in the manner described above so that each node has the connectivity information of all other nodes in its partition). Each node prepares a connectivity graph for its partition and finds an all-connected component with the largest size (or largest clique) of the connectivity graph. If there are two or more such all-connected components, the node selects one all-connected component as the winning all-connected component, determined by a tie-breaking process (e.g., based on the lexicographical order of the unique identifiers of the nodes in the complete components). If the size of the winning all-connected component is at least (n / 2+1) and the node is a member of the winning all-connected component, the node decides to continue running (and terminates the protocol), otherwise the node shuts itself down.

[0051] Below is an optimization that ensures that nodes agree on membership changes after at most two rounds of broadcasting.

[0052] Optimization 1: This is an optimization applicable in case of the scenario covered by Case A (in the above section). It is based on the observation that if a network failure event partitions the database into disjoint groups of fully connected nodes, the suspect lists of all nodes in the groups / partitions will be the same. For example, consider Figure 2. In Figure 2, nodes TE1, TE2, and SM1 suspect TE3 and SM2, and nodes TE3 and SM2 suspect TE1, TE2, and SM1. If, after the first round of cooperative broadcasting during Phase 1, a node's suspect list matches the suspect lists of all its neighbors, the node can infer that (a) it is part of the fully connected component, and (b) it can determine the size of the fully connected component (which is equal to the size of the chorus minus the size of its own suspect list). Thus, all nodes can agree on the membership changes after the first round of broadcasting during Phase 1.

[0053] Optimization 2: This is an optimization that is mainly applicable to cases B and C above, and partially applicable to case A. In the unoptimized process, all nodes participate in (n-1) rounds of cooperative broadcasting. This makes each node aware of the connectivity information of all other nodes in its partition. However, does each node really need to know the connectivity information of all other nodes in its partition to arrive at an optimal failure resolution decision? Consider splitting the nodes into two categories based on their suspect list after a network failure event. Category (M) includes nodes that suspect less than n / 2 other nodes, and category (N) includes nodes that suspect more than n / 2 nodes. Nodes that suspect more than n / 2 can immediately shut themselves down rather than broadcasting their suspect list, because they cannot be part of the winning all-connected component.

[0054] For example, consider FIG. 11. After a network failure event, nodes TE2, SM1, and SM2 are in category (M), and nodes TE1 and TE3 are in category (N). Consider category (M). Does a node in category (M) need to know about the connectivity information of other nodes in category (M) to make optimal failure resolution decisions? Yes. The reason is that a node in category (M) can form a total connected component of size at least (n / 2+1) with other nodes in category (M), and knowing about the connectivity information of other nodes in category (M) helps it to (a) determine whether it is part of a total connected component of size at least (n / 2+1), (b) identify all total connected components of size at least (n / 2+1), and (c) determine whether it is part of a winning total connected component. Does a node in category (M) need to know about the connectivity information of nodes in category (N) to make optimal failure resolution decisions? No. This is because a node of category (M) will never form a total connected component of size at least (n / 2+1) with a node of category (N) because the node of category (N) suspects more than (n / 2) other nodes.

[0055] Now consider category (N). Does a node in category (N) need to know about the connectivity information of nodes in category (M) and category (N) to make optimal failure resolution decisions? No. The reason is that a node in category (N) suspects more than (n / 2) other nodes, and therefore does not form a total connected component of size at least (n / 2+1) with any other node(s). Having the connectivity information of all other nodes helps a node in category (N) know which other nodes will remain operational, but it does not change the fact that the node cannot form a total connected component of size at least (n / 2+1) with other nodes.

[0056] Thus, enough rounds of cooperative broadcasting to make each node in category (M) aware of the connectivity information of each other node in category (M) will be sufficient for all nodes in the distributed database system to agree on the optimal failure resolution outcome. Thus, as a modification to the unoptimized process, the optimized process begins by killing category (N) nodes before the start of phase 1, but at the same time keeping them as members of the chorus. In other words, category (M) nodes kill themselves before phase 1 begins, but keep category (N) nodes on their node list until phase 2. By keeping the killed nodes (category (N) nodes that can be killed before phase 1 begins) as members of the chorus until phase 2, correctness is ensured, i.e., the failure resolution outcome is a fully connected set with at least (n / 2+1) nodes, where n includes the nodes that were optimally killed before phase 1. (If we do not include category (N) nodes (or nodes of any type), the value of n (group size) and the size of the majority may change, making it more difficult to prove the accuracy of the results.)

[0057] Outages of nodes in category (N) do not affect connections between nodes in category (M) (i.e., no nodes in category (M) are disconnected due to outages of nodes in category (N)), because any two nodes in category (M) are either directly connected to each other or connected through other nodes in category (M). Therefore, outages of nodes in category (N) should not affect the optimality of the fault-resolution outcome.

[0058] Conceptually, the optimization essentially ensures that the nodes in category (M) reach a consensus on the outcome of the failure resolution and that the nodes in category (N) follow that outcome. With this optimization, each node that started phase 1 is connected to at least (n / 2) other nodes, so the diameter of the connectivity graph (i.e., the maximum distance between any two nodes in the connectivity graph) is at most 2. Therefore, only two rounds of broadcasting are required for each node that started phase 1 to learn about the connections of each other node that started phase 1. Each node in phase 1 is connected to at least n / 2 other nodes, so the diameter of the connectivity graph is at most 2, so any two nodes are at most one node apart.

[0059] Optimized distribution of suspected node list Consider a chorus containing n fully connected nodes. Suppose a network failure occurs. Each node executes the following protocol to resolve the network failure: Each node prepares its own suspect list (Note: this suspect list can be an empty list, which can happen if a node is (or thinks it is) fully connected to all other nodes after the network failure event).

[0060] Phase 0: Each node checks if it suspects more than (n-1 / 2) other nodes. If so, it aborts itself. (Other nodes may hear about this abort during Phase 1. If they do, they restart the protocol and start over from Phase 0.)

[0061] Phase 1, Round 1: Each node sends its suspect list to its neighbors and waits to receive its neighbors' suspect lists. As noted above, if one or more of a node's neighbors fail in Phase 0, the node may hear about those outages while waiting for its neighbors' suspect lists. Upon hearing about such outage(s), the node restarts the protocol and starts over from Phase 0. This causes other nodes to restart the protocol as well. Similarly, if a neighboring node restarts the protocol, the node starts over from Phase 0. Also, as noted above, the node does not initiate failover of any failed nodes at this stage (i.e., it keeps all nodes in its chorus for the purposes of determining the winning all-connected component). This is true for multiple rounds of Phase 0.

[0062] Each node checks whether its suspect list is the same as the suspect lists of all its neighbors. If the suspect list of a node matches the suspect lists of all its neighbors, this indicates that the node is fully connected to its neighbors. This scenario is covered by Case A above (e.g., Figure 2). Since each node that started Phase 1 is connected to at least (n / 2) other nodes, the size of the node's neighbor list can be at least (n / 2) (a node together with its neighbors forms a group containing at least (n / 2+1) nodes). The node decides to continue and terminates the protocol.

[0063] If a node's suspect list does not match the suspect list of at least one of its neighbors, this indicates that the node is not fully connected to all other nodes in its partition. This scenario is covered in Cases B and C above (e.g., Figures 3-9). Such a node cannot determine whether or not to continue operating based on the information received in Round 1. Therefore, it performs Round 2 of Phase 1.

[0064] Phase 1, Round 2: Each node transmits the suspect lists of other nodes received in Round 1 to its neighbors and waits until it receives such suspect lists of its neighbors' neighbors from its neighbors.

[0065] Phase 2: Each node has now received the connectivity information of all other nodes in its partition. Each node prepares the connectivity graph of its partition and finds the largest all-connected component (or a maximal clique of size at least (n / 2+1)) with at least (n / 2+1) nodes in the connectivity graph. If there are more than one all-connected components (e.g., Figure 7), the node selects one all-connected component as the winning all-connected component, determined by a tie-breaking process (e.g., lexicographical order), to make failure resolution deterministic. If the node is a member of the winning all-connected component, the node decides to continue running (and terminates the protocol), otherwise the node shuts itself down.

[0066] If a new network failure event occurs while the distributed database system is resolving a network event, the protocol forces the node to backtrack and re-examine its connections, taking into account the effect of the new network event, before making a failure resolution decision.

[0067] In addition to new network failure events, node outages (e.g., caused by a manual shutdown of a node) may occur while a node in the distributed database system is resolving a network failure. In response to a node outage, the protocol has the node restart from phase 0 while retaining the failed node as a member of the chorus until phase 2 (by not performing a failover to the failed node, the remaining nodes are prevented from removing the failed node from their node lists). As explained above, retaining the failed node as a member of the chorus until phase 2 ensures correctness, i.e., the outcome of failure resolution is a fully connected set with at least (n / 2+1) nodes, where n includes the failed node, and thus there can be only one such set that continues to operate after phase 2.

[0068] 12 is a flow chart illustrating an optimized process 1200 for resolving network faults. Since each node follows the same process, the flow chart shows the process 1200 from the perspective of a single node. The process 1200 will be described in detail in terms of stages.

[0069] Stage 0: Initial stage. At 1202, a node is fully connected to all other nodes in the chorus. If it detects a suspect node locally or remotely, the node moves to Stage 1.

[0070] In stage 1:1210, the node waits for one ping (heartbeat) cycle for another ping (heartbeat) timeout to occur, prepares its suspect list, incorporates any received suspect list messages, and then enters stage 2.

[0071] Stage 2: In 1220, the node checks whether it suspects more than (n-1 / 2) other nodes, where n is the number of nodes in the chorus. If so, in 1299, the node shuts itself down. If not, the node prepares its suspect list in stage 1 and then checks whether there are any new suspects. The node also checks whether any of its neighbors have restarted the protocol because they have detected a new suspect. Each node may assign a number, called protocolIterationNumber, for each iteration of process 1200 it performs. Each node sets this number in the suspect list messages it sends and compares its local protocolIterationNumber with the protocolIterationNumber in the suspect lists it receives from other nodes. If the node determines that its protocolIterationNumber is less than its neighbor's protocolIterationNumber, it determines that its neighbor has restarted the process and returns to stage 1. Otherwise, the node enters stage 3. (If a node's protocolIterationNumber is greater than its neighbor's protocolIterationNumber, then the node is restarting the protocol (perhaps because it has found a new victim), which should cause its neighbors to restart the protocol as well.)

[0072] Stage 3: At 1230, the node broadcasts its round 1 suspect list to its neighbors. While waiting for the round 1 suspect list message, the node may detect a new suspect or hear that one or more of its neighbors have detected a new suspect, at 1232. If so, the node stops waiting for further responses and returns to stage 1. At 1234, the node receives round 1 suspect list messages from all of its neighbors. If the node does not receive a timely response (e.g., within a predefined period) from any of its neighbors, the node marks such neighbors as suspects, at 1236, and returns to stage 1. If the node receives a round 1 suspect list with a protocolIterationNumber greater than its protocolIterationNumber, the node returns to the beginning of stage 1, at 1238. Once the node has received round 1 responses from all of its neighbors, it enters stage 4.

[0073] Stage 4: At 1240, if the node's suspect list matches the suspect lists of all its neighbors, the node determines that it is fully connected to its neighbors (e.g., as in FIG. 2). Since each node that started stage 3 is connected to at least (n / 2) other nodes, the size of the node's neighbor list can be at least (n / 2) (i.e., the node and its neighbors form a fully connected component or group that contains at least (n / 2+1) nodes). At 1201, the node decides to continue operation, removes the suspect node, and ends process 1200.

[0074] If a node's suspect list does not match the suspect list of at least one of its neighbors, then the node is not fully connected to all other nodes in its partition (e.g., as in Figures 3-9). Because the node cannot determine whether to continue operating or to shut down based on the information received in round 1, the node enters stage 5, which includes broadcasting a round 2 suspect list message at 1250.

[0075] Stage 5: At 1250, the node broadcasts its Round 2 suspect list to its neighbors, including its original suspect plus the suspects of its neighbors, and waits until it receives Round 2 suspect list messages from all its neighbors. The node may receive Round 2 suspect list messages from other nodes at any time after it broadcasts its Round 1 suspect list message at 1230. The node accumulates these Round 2 suspect list messages. At 1252, the node returns to Stage 1 if a new network failure occurs, or if the node receives a Round 1 message from another node, or if the node hears about an outage of another node. Upon returning to Stage 1, the node discards all accumulated Round 2 suspect list messages. However, if the other node returns and sends another message, the message is kept. The node distinguishes between these two types of messages based on the protocolIterationNumber in the Round 1 and Round 2 suspect list messages. In other words, a message based on the protocolIterationNumber includes the protocolIterationNumber and the round number.

[0076] At 1254, the node enters stage 6 when it receives the Round 2 suspect list messages from all of its neighbors. If a new network event occurs after the node broadcasts its Round 2 suspect list message, or if the node hears about an outage of another node, the fault resolution decision may become suboptimal. There are at least two possible cases: case (a) the node has already received the Round 2 message from the new suspect or failed node, and case (b) the node has not received the Round 2 message from the new suspect or failed node.

[0077] In case (a), the node may proceed to stage 6, perform fault resolution of the current network event, and then deal with the new network event by restarting the protocol, or return to stage 1 (without resolving the current network event) and restart process 1200 (which then resolves both the current and new network faults). In case (b), the node does not receive a round 2 message from either the new victim or the failed node, so the node returns to stage 1. However, there is no guarantee that other nodes will also return to stage 1 before completing stage 6 (because they may have received a round 2 message from the new victim or the failed node). In this case, the fault resolution outcome may be suboptimal (i.e., the surviving set will be smaller than it could have been, but there is still only one surviving set). However, the node's move to stage 1 does not block the progress of other nodes, since it has already sent its round 2 message.

[0078] Stage 6: In 1260, the node prepares a connectivity graph for its partition and finds the largest fully-connected component of the connectivity graph that is at least (n / 2+1) in size (or a largest clique of at least (n / 2+1) in size). If there are two or more such components, the node selects one of them as the winning fully-connected component, as determined by a tie-breaking process. If the node is a member of the winning fully-connected component, in 1201, the node decides to continue running and prunes nodes that are not part of the winning fully-connected component. Otherwise, in 1299, the node shuts itself down.

[0079] Protocol Iteration Number As discussed above, any node in the distributed database system may initiate a fault resolution protocol (e.g., process 1200 of FIG. 12) in response to detecting one or more suspect nodes. Also, any new network fault event occurring during the execution of the fault resolution protocol will trigger a restart of the protocol. To enable a node to detect whether a suspect list message it receives (round 1 or round 2) belongs to the current invocation of the protocol or the next invocation of a restart of the protocol (and, for a node that has restarted the protocol, the previous invocation of the protocol), the node associates a number, called protocolIterationNumber, with each invocation of the fault resolution protocol.

[0080] Each node maintains its own local protocolIterationNumber and sets this number in suspect list messages it sends, and each node compares its local protocolIterationNumber with the protocolIterationNumber in the received suspect list message. If the numbers match, the node infers that the received suspect list message corresponds to the current invocation of the protocol. If the protocolIterationNumber in the received suspect list message is greater than its own protocolIterationNumber, the node infers that the sender has initiated a protocol restart (and will therefore restart the protocol). Also, if the protocolIterationNumber in the received suspect list message is less than its own protocolIterationNumber, the node infers that the sender is still executing a previous iteration of the protocol and should therefore ignore the message.

[0081] Each node can maintain its own local protocolIterationNumber in the following way: (a) The ProtocolIterationNumber is set to zero on the first node during database initialization and during database restart. (b) The ProtocolIterationNumber is serialized as part of the Master Catalog, and a new node joining the distributed database system receives the current protocolIterationNumber from the Master Catalog Chairman when it fetches the Master Catalog. A new node receives the Master Catalog when it joins the distributed database system. Storing the current protocolIterationNumber in the Master Catalog makes the current protocolIterationNumber available to the new node. (c) If the node does not suspect any node, has detected a suspect node, has not received any suspect list messages from other nodes, and is invoking the fault resolution protocol, the node increments its protocolIterationNumber. (d) If a node does not suspect any nodes, has detected a suspect node, has received one or more suspect-list messages from other nodes, and has invoked the fault resolution protocol, the node sets its protocolIterationNumber to the maximum protocolIterationNumber in the received suspect-list messages. (e) If a node has not detected a suspect node, has received one or more suspect list messages from other nodes, and has invoked the fault resolution protocol, the node sets its own protocolIterationNumber to the largest protocolIterationNumber in the suspect list messages received from the other nodes. (f) If a node is running the failure resolution protocol, has not received a suspect list message with a protocolIterationNumber greater than its local number, and detects a new network failure event, the node increments its protocolIterationNumber (and restarts the protocol). (g) If a node is running the failure resolution protocol, receives a suspect list message with a protocolIterationNumber greater than its own local number, and detects a new network failure event, the node sets its local protocolIterationNumber to the number in the received suspect list message (and restarts the protocol). (h) If a node is running the failure resolution protocol and receives a suspect list message with a protocolIterationNumber greater than its own local number, and it has not detected any new network failure events, the node sets its local protocolIterationNumber to the number in the received suspect list message (and restarts the protocol). (i) If a node is running the fault resolution protocol and receives a suspect list message with a protocolIterationNumber less than its own local number, the node ignores the message.

[0082] These points can be summarised as follows: (A) ProtocolIterationNumber is set to zero on the first node during database initialization and during database restart. (B) The ProtocolIterationNumber is serialized as part of the Master Catalog, and any new node joining the database receives the current protocolIterationNumber from the Master Catalog Chairman when it fetches the Master Catalog. (C) When a node is invoking the fault resolution protocol (because it has detected a suspect node and / or received a suspect list message from another node), it checks whether it has received a suspect list message with a protocolIterationNumber greater than its local protocolIterationNumber. If so, it sets its local protocolIterationNumber to the maximum protocolIterationNumber in the received suspect list message(s), otherwise it increments its own protocolIterationNumber.

[0083] Dealing with Unidirectional Link Failures Unidirectional link failures such as case D (Figure 10) above can be resolved by treating them as bidirectional link failures (i.e., by making the nodes on both sides of the failed link suspect each other). For example, consider two nodes, node A and node B, in a distributed database system. Assume that node A can send messages to node B, but node B cannot send messages to node A. Node A can send ping messages to node B, but does not receive an acknowledgement message from node B, so node A begins to suspect node B. At this point, node B does not yet suspect node A. However, node A begins to suspect node B, so it stops sending ping messages to node B. This causes node B to suspect node A, and so the unidirectional link failure is converted into a bidirectional link failure.

[0084] In the process described herein, a node sends a MsgPing message (e.g., a ping message) and sets Node::lastPingTime for a particular node only if that node has acknowledged the previous MsgPing message. This ensures that a unidirectional link failure will cause nodes on both sides of the link to suspect each other. Thus, the above protocol can resolve unidirectional link failures or a mix of unidirectional and bidirectional link failures.

[0085] Chorus Membership Changes If a network failure event occurs while a new node (or set of new nodes) is joining the chorus, this process must ensure that the chorus does not split into multiple majority groups. In Figure 10, for example, a network partition causes the chorus to split into a majority group {SM1, SM2} and a minority group {TE1}. However, the minority group {TE1}, together with the new node {TE2, TE3}, forms a majority group {TE1, TE2, TE3}, resulting in two "majority" groups {TE1, TE2, TE3} and {SM1, SM2}.

[0086] One way to solve the problems associated with joining a new node into a chorus is to disable the new node(s) if a network failure event occurs while the new node(s) are joining the chorus. This prevents the minority set of nodes in the current chorus from forming a majority group with the new node(s). In FIG. 10, new nodes TE2 and TE3 (which are still in the process of joining the chorus) can be disabled, which will also disable TE1, leaving a single majority group {SM1, SM2} in the database. This process does not limit the number of nodes that can join the chorus at the same time. However, this process may impact the availability of the system (depending on the number of odd or even nodes in the current chorus, the number of nodes trying to join the chorus, the number of nodes in the chorus that are aware of the new node(s) at the time of the network failure, etc.) since the new node(s) are disabled and some nodes in the current chorus may be aware of the new node(s).

[0087] This process can also piggyback on the process for requesting fragments of data in a distributed database (originator sends available fragments, peers send acknowledgments to originator, originator sends complete data to requester) to get current chorus members to agree on a new node joining the chorus. This process includes the following modifications to the failure resolution process 1200 of Figure 12 in order for nodes to agree on chorus size during the node joining process:

[0088] Nodes exchange their complete connectivity information (i.e., their neighbor lists and their suspected node lists) during the Round 1 and Round 2 broadcasts. In response to receiving Round 1 / Round 2 messages, nodes compare their suspected node lists and neighbor lists with those of their neighbors. Nodes may not recognize anyj If a node notices that its neighbors know about it, it increases its chorus size to n j Then, the process restarts.

[0089] This process ensures correctness: if a network partition prevents the new node(s) from being included in the node lists of all nodes in the chorus, the new node(s) will take themselves down while the fault is resolved. Let n be the number of nodes in the chorus, and j Let be the number of nodes that are trying to join the chorus at the same time but cannot be included in the node list of all n nodes due to network partitions. j This node (new node) will fail itself during the process regardless of its partition. Therefore, at most n nodes check whether they are in the majority partition after round 1 to decide whether they should continue to run. The nodes in each partition are divided by chorus size s (n≦s≦n+n j ), and since after round 1 there are at most n nodes in the chorus, at most one partition can form the majority group, which ensures correctness.

[0090] However, what happens if all nodes in a partition add new node(s) to their node list after starting the failure resolution protocol? (Note that nodes prepare their suspect and neighbor lists during stage 1 at the start of the protocol and cache that information.) None of the nodes can detect that new node(s) have been added to their node list. As a result, the master catalog of the new node(s) may transition to a full state and the new node(s) may join the failure resolution process, resulting in multiple majority groups.

[0091] For example, consider this scenario: A chorus includes nodes A, B, and C, where A is the chairman / leader of a fragment of a distributed database (e.g., fragment "master catalog"). New nodes D and E try to join the chorus at the same time. Node A sends availability messages for D and E to B and C. B and C do not receive a ping message from A, suspect A, and start the protocol. Since B and C have not (yet) applied the available message from A, they start the protocol with chorus members {A,B,C}. Then B and C apply the available message and send an acknowledgement message to A, and then a network partition occurs. Since the master catalog for D and E is complete, A, D, E start the protocol with chorus members {A,B,C,D,E}. Both groups {A,D,E} and {B,C} think they can form the majority group.

[0092] The following extension prevents such a situation: after applying the available message (or, in case of the chairman node, after sending the master catalog to the new node), the node restarts the failure resolution protocol (if one is in progress), which causes the node to invalidate its cached suspect and neighbor lists and recompute them with a larger chorus size.

[0093] 13-18 show some exemplary failure scenarios and how they are handled by the failure resolution process of the present invention.

[0094] Scenario (A): A network partition occurs, isolating the new node and the entry node (originator of the master catalog) from the rest of the nodes.

[0095] In Figure 13, SM3 requests and receives the master catalog from TE1 (chairman of the master catalog), and before TE1 sends MsgObjectAvailable (e.g., a message informing the receiving node that the source node is joining the distributed database system) to SM1 and SM2, a network partition occurs. All nodes, including SM3, start the resolution protocol. SM3 and TE1 suspect nodes SM1 and SM2, and SM1 and SM2 suspect TE1 (SM1 and SM2 do not know about SM3). SM3 dies because it is still joining the chorus (it has not received a completion from TE1), TE1 dies (in phase 0) because it suspects two nodes in the chorus {SM1, SM2, SM3, TE1}, and SM1 and SM2 continue to run because they form the majority of the chorus {SM1, SM2, TE1}.

[0096] Scenario (B): A variation of scenario (A), where a network partition occurs, isolating the new node and the entry node (originator of the master catalog) from the rest of the nodes.

[0097] In Figure 14, SM3 requests and receives the master catalog from TE1 (chairman of the master catalog), SM1 receives MsgObjectAvailable from TE1, and a network partition occurs before SM2 receives MsgObjectAvailable from TE1. SM3 and TE1 suspect SM1 and SM2, SM1 suspects SM3 and TE1, and SM2 suspects TE1 (SM2 does not know about SM3). SM3 fails because it is still joining the chorus (it has not received final confirmation of joining from TE1), and TE1 and SM1 fail (in phase 0) because they suspect two nodes in the chorus {SM1, SM2, SM3, TE1}. SM2 does not fail in phase 0 because it initially suspects only TE1 (for n=3, this is less than n / 2 the number of nodes), and it sends a message to SM1 for round 1 of phase 1, but after hearing about SM1's failure, it restarts the resolution protocol and then fails itself.

[0098] Scenario (D): A network partition occurs, isolating the new node, the entry node, and some peers from the remaining peers.

[0099] In Figure 15, SM3 requests and receives the Master Catalog from TE1 (chairman of the Master Catalog), SM1 receives MsgObjectAvailable from TE1, and before SM2 receives MsgObjectAvailable from TE1, a network partition separates SM2 from the rest of the nodes. SM3 fails because it is still joining the chorus (has not received completion from TE1). SM2 fails because it is in the minority partition of the chorus {SM1, SM2, TE1}. TE1 and SM1 start the protocol, do not receive a response from SM3 (round 1), and eventually fail themselves after suspecting SM3.

[0100] Scenario (E): A network partition separates the entry node (chairman of the master catalog) from the rest of the nodes.

[0101] In Figure 16, SM3 requests and receives the master catalog from TE1 (chairman of the master catalog), SM1 and SM2 receive MsgObjectAvailable from TE1, and a network partition causes entry node TE1 to become isolated from the rest of the nodes. SM3 fails because it is still joining the chorus. TE1 fails because it is in the minority partition of chorus {SM1, SM2, SM3, TE1}. SM1 and SM2 start fault resolution procedures, do not receive a response from SM3, and eventually suspect SM3 before failing themselves.

[0102] Scenario (H): A network partition isolates the new node, the entry node, and some of the peers from the remaining nodes.

[0103] In Figure 17, SM4 requests and receives the master catalog from TE1 (chairman of the master catalog), SM1 and SM3 receive MsgObjectAvailable from TE1, and a network partition occurs isolating SM2 from the rest of the nodes. SM4 fails because it is still joining the chorus. SM2 fails because it is in the minority partition of chorus {SM1, SM2, SM3, TE1}. TE1, SM1, and SM3 continue to run because they form a majority group in chorus {SM1, SM2, SM3, SM4, TE1}. In this case, the group {TE1, SM1, SM3} was the majority in the original chorus {TE1, SM1, SM2, SM3}, but is still the majority in the new chorus {TE1, SM1, SM2, SM3, SM4}. This allows TE1, SM1, and SM3 to continue to run even after SM4 is added to their node lists. In general, this behavior occurs whenever the chorus size changes from an even number of nodes to an odd number of nodes (due to a node attempting to join the chorus during a network failure).

[0104] Scenario (I): A network partition causes the new node, the entry node, and some of the peers to be separated from the remaining nodes.

[0105] In Figure 18, SM4 and SM5 request and receive the master catalog from TE1 (chairman of the master catalog), SM1 and SM3 receive MsgObjectAvailable for both SM4 and SM5 from TE1, and a network partition separates SM2 from the rest of the nodes. SM4 and SM5 fail because they are still joining the chorus, and SM2 fails because it is in the minority group of chorus {SM1, SM2, SM3, TE1}. TE1, SM1, and SM3 also fail because they form a minority group in chorus {SM1, SM2, SM3, SM4, SM5, TE1}. Nodes TE1, SM1, and SM3, which continued to operate in scenario (H), now fail because they are in the minority group in the new chorus due to the two nodes that are joining the chorus.

[0106] Conceptually, a chorus with n nodes can tolerate a network partition where up to (n-(n / 2+1)) nodes are isolated from the rest of the nodes in the chorus (or up to (n-(n / 2+1)) simultaneous failures of nodes in the chorus) and still continue to operate. In the case of a single node trying to join the chorus, if n is odd, the chorus can tolerate an isolation of (n-(n / 2+1)-1) nodes and still continue to operate. In the case of a single new node, if n is even, the chorus can tolerate an isolation of (n-(n / 2+1)) and still continue to operate.

[0107] Let the chorus fault tolerance be the maximum number of node failures that the chorus can tolerate without all nodes in the chorus failing. For a chorus with n nodes, if there are no new nodes joining the chorus, the chorus fault tolerance is (n-(n / 2+1)) (column 1 in Table 1). If there is one node trying to join the chorus, the chorus fault tolerance drops to (n-(n / 2+1)) if n is odd, and remains (n-(n / 2+1)) if n is even (column 2 in Table 1). If the number of new nodes trying to join the chorus (simultaneously) is greater than 1, the chorus fault tolerance can drop further. Table 1 shows the maximum number of node failures that the chorus can tolerate for different numbers of nodes in the chorus (n) and different numbers of nodes trying to join the chorus simultaneously (n j ) and the number of failures that the chorus can tolerate. In Table 1 below, we consider the number of simultaneous j There are n nodes, and at least one node in the majority partition is j MsgObjectAvailable has been received from all nodes. TIFF2025031721000002.tif56154 Table 1

[0108] The fault resolution for scenarios B, D, and F (above) is for n=3 and n j =1. Since the Chorus failure tolerance for this configuration is zero, a network partition (or any node failure) while a new node is joining (at least one of the nodes has received MsgObjectAvailable) will cause the entire Chorus to fail. Scenario A is not captured in Table 1 because none of the nodes in the majority group in Scenario A received MsgObjectAvailable. Scenario H is captured in the table entry for n=4 and n j = 1 entry. Scenario H has a Chorus failure tolerance of 1. Chorus has a single node in the minority partition, so Chorus continues to operate. Scenario I has n = 4 and nj =2 entry. The Chorus failure tolerance for this configuration is zero, so a network partition while the node is joined will cause the entire Chorus to fail.

[0109] Dealing with node failures This section discusses how a distributed database system handles one or more node failures (or shutdowns) while resolving a network failure. As discussed above, the process of resolving a network failure event includes nodes exchanging failure detection messages, determining whether or not the nodes should continue to operate based on the exchanged messages, and nodes that decide to continue to operate performing a failover of the suspect node. This process is illustrated in FIG. 19.

[0110] In Figure 19, a chorus contains members {A,B,C,D}. A network partition isolates {A,B,C} from D. Nodes A, B, and C become suspicious of node D, exchange failure detection messages, decide to remain operational, and perform a failover of D. Node D becomes suspicious of nodes A, B, and C, initiates a failure resolution protocol, and takes itself down.

[0111] If a ping (heartbeat) timeout is enabled, a node failure will cause the neighbors of the failed node to initiate (or restart) a failure resolution protocol, agree on the exclusion of the failed node, and remove the failed node from the chorus. If a node failure occurs while the distributed database system is in the middle of resolving a network failure event, the failed node may appear as a new suspect to the neighbors of the failed node, which may cause the neighbors to restart the protocol. Thus, there is no special mechanism to deal with node failures during partition resolution. Instead, the process described herein ensures that nodes that initiate / restart a failure resolution protocol in response to a node failure agree on chorus membership.

[0112] Dealing with node failures during the exchange of failure detection messages If a node fails during the exchange of failure detection messages, it will no longer be on its neighbors' suspect list. As a result, the neighbors will agree on chorus membership / size, similar to the process discussed above. In response to detecting a new suspect resulting from a node failure, the neighbors will restart the failure resolution process with an updated suspect list, which is the union of the suspect nodes resulting from the network failure and the node where the failure occurred. If the neighbors form a majority group based on the updated suspect list, they will continue to operate.

[0113] In Figure 20, node C fails while the nodes are exchanging messages, as a network partition separates {A,B,C} from D. Nodes A and B suspect C and restart the protocol. A and B fail because they do not form a majority group in the chorus {A,B,C,D}.

[0114] Dealing with node failures during failover If a node fails while it is performing a failover (removing the failed node from the chorus membership list), its neighbors may have initiated or completed failover for other suspect nodes. As a result, neighbors may not agree on chorus membership / chorus size when the protocol starts / restarts, since they may have removed one or more suspect nodes from their own node lists.

[0115] In Figure 21, Chorus contains members {A,B,C,D}. A network partition isolates {A,B,C} from D. Nodes A, B, and C start the protocol, exchange failure detection messages, and decide to continue. Nodes A, B, and C initiate a failover of node D. After A completes the failover of D (and removes D from its node list), while B is still performing a failover of D, node C fails. This causes A to suspect C and initiates node failure processing in Chorus {A,B,C} and suspect list {C}. This also causes B to initiate node failure processing in Chorus {A,B,C,D} and suspect list {C,D}. As a result, A and B do not agree on Chorus membership.

[0116] In this case, the nodes become agreed on chorus size as follows:

[0117] Non-failing nodes exchange their complete connectivity information (i.e., their neighbor list and their suspect node list) during rounds 1 and 2 of broadcasting. After receiving round 1 / round 2 messages, nodes compare their suspect node list and neighbor list with the suspect node list and neighbor list of their neighbors. A node may not know about any j If a node notices that its neighbors know about it, it increases its chorus size to n j Then, the fault resolution process is restarted by incrementing the counter by one.

[0118] Thus, if n is the number of nodes in the majority partition, f is the number of nodes that have failed, and e is the number of nodes being evicted for which a failover is being performed, then the nodes in the partition will remain operational if (nf) >= (s / 2+1), where (n <= s <= n+e).

[0119] However, what happens if a node completes a failover while it is still performing a failure resolution process? To increase the chances of keeping chorus members up and running, the following modification can be made to the failure resolution process: After a node completes a failover for the node being evicted, it restarts the failure resolution process (if one is in progress), so that this process runs with a smaller chorus size.

[0120] To ensure that all nodes agree on the chorus size when the nodes restart with a smaller chorus size, this process can be further extended as follows: Nodes exchange their complete connectivity information (i.e., their neighbor list and their suspect node list) during rounds 1 and 2 of broadcasting. Then, nodes compare their suspect node list and neighbor list with the suspect node list and neighbor list of their neighbors. Nodes can then determine which nodes are not aware of the chorus size. j If a node notices that its neighbors know about it, it increases its chorus size to n j Then, the node's neighbors increment r by r and restart the process. j If you remove a node and restart the process, the node will increase its chorus size by r j Then, the counter is decremented by one and the process is restarted.

[0121] This allows the nodes to agree on the chorus size, but they do not need to agree on the chorus membership (or chorus size) as long as the node with the questionable membership and the new node are out of service. In other words, each node can perform a failure resolution process based on the chorus membership determined by that node's master catalog node list. This process is guaranteed to reach the correct result for all nodes as long as the nodes whose membership is not agreed upon are either out of service before the process begins or out of service during the process.

[0122] To see why this is true, consider n+n j Let be the number of nodes in the chorus, n is the number of nodes for which the master catalog is complete, and n j is the sum of the number of failed nodes and the number of nodes that will fail (as in the case of node failure, the master catalogs of these nodes may or may not be complete at the time of the failure) or the number of new nodes that will fail when they start the failure resolution protocol (as in the case of node joining, the master catalogs of these nodes will not be complete at the time of the failure).

[0123] Let s be the size of the master catalog node list of nodes participating in the failure resolution protocol, then n ≤ s ≤ n + n j Note that s may not be the same for all nodes participating in the failure resolution protocol.

[0124] Can a failure resolution protocol guarantee that at most one node in a partition will survive if each node runs the protocol with chorus size set to its own master catalog node list size? Yes, but only if n ≤ s ≤ n + n j Since the majority group size calculated by each node is at least (n / 2+1), each node in the partition is in the majority group (n / 2+1≦majority group size<(n+n j If we can conclude that (n / 2+1) / n+1), then the partition has at least (n / 2+1) nodes. There can be at most one such partition, since there are only n nodes participating in the protocol. Thus, nodes in at most one partition can successfully complete the failure resolution protocol and continue to operate.

[0125] It is not necessary for all nodes in a partition to conclude that they are in the majority all-connected component for that partition to become the winning all-connected component. A subset of the nodes in a partition may conclude that they are not in the majority group depending on their master catalog node list size. These nodes will fail during stage 2 of the process (FIG. 12) while the remaining nodes in the partition restart the process. However, if the remaining nodes can conclude that they are in the majority group upon restart of the process, that may be sufficient to make that all-connected component the winning all-connected component.

[0126] Achieving additional performance goals If a user chooses to shut down more than half of the chorus member nodes, it cannot trigger failure detection. This is achieved by modifying the failure resolution process to not treat manually shut down nodes as suspect nodes.

[0127] Identifying the node that is shutting down Upon receiving a shutdown request from the management layer, the node broadcasts a MsgNodeState message indicating that it is shutting down (e.g., with node state NODE_STATE_SHUTTING_DOWN). The management layer in a distributed database system is the layer of nodes where users can interact with the distributed database. The management layer can keep track of the nodes in the distributed database system and can facilitate interactions between users and the nodes in the distributed database system. For example, if a user wants to shut down a node, the user can issue a shutdown command to the management layer, which then sends a shutdown message to the node specified by the user. This process relies on at least one chorus member receiving this node state message from the node that is shutting down.

[0128] Changes to the incident resolution protocol The following modifications can be made to the problem resolution protocol: (a) During stage 1 of the process (Figure 12), nodes that are known to be shutting down are not considered suspect nodes. (b) Have the chorus members gossip about the node that is shutting down (eg, by exchanging failure detection messages during stages 3 and 4 of the process).

[0129] An example of how these changes can satisfy the desire to identify nodes that have been manually shut down is given below: Consider a chorus with nodes A, B, C, and D. Assume a user shuts down nodes C and D at approximately the same time. Assume that only node A receives node status messages from C, and only node B receives node status messages from D. Node A starts a failure resolution process with chorus {A,B,C,D}, suspect list {D}, and shutting down node list {C}, and sends a round 1 failure detection message to B. Node B starts the protocol with chorus {A,B,C,D}, suspect list {C}, and shutting down node list {D}, and sends a round 1 failure detection message to A. In response to receiving the failure detection message, node A updates its shutting down node list to {C,D}, suspect list to {}, and restarts the protocol. Node B does the same. After round 1, nodes A and B conclude that they are in the majority partition based on chorus size=4 and suspect list size=0, and continue running.

[0130] However, how does the modified protocol arrive at the correct operation if a network partition or link failure occurs while the nodes are shutting down? Consider this scenario. A chorus includes nodes A, B, C, D, and E. A user shuts down node E, and at about the same time a network partition separates {A,B} from {C,D}. Assume all nodes receive a node status message from E. Node A starts the protocol with chorus {A,B,C,D,E}, suspect list {C,D}, and shutting down node list {E}, and sends a round 1 failure detection message to B. Node B also starts the protocol with chorus {A,B,C,D,E}, suspect list {C,D}, and shutting down node list {E}, and sends a round 1 failure detection message to A. Upon receiving the failure detection message, nodes A and B conclude that they are in the majority partition (based on chorus size = 5 and suspect list size = 2) and continue to operate. Nodes C and D also continue to operate using the same logic. The following approach can ensure that the protocol reaches the correct action in this scenario: If a network partition (or link failure) occurs while a node(s) is shutting down, treat the shutting down node(s) as a suspect.

[0131] In summary, if a user shuts down more than half of the chorus member nodes (let's call this node set SD), this process will allow the remaining nodes (let's call this node set NSD) to continue running if the following conditions are true: (A) At least one node in the NSD receives a node state change message from each node in the SD. (B) Each node in the NSD receives a node state message from at least one node in the SD. (The reason for this is that if a node in the NSD does not receive a node state message from any node in the SD, it will become suspicious of all nodes in the SD and will shut itself down in stage 2 before learning about nodes that are shutting down from other nodes.) (C) No network partitions or link failures will occur until the NSD nodes complete the fault resolution protocol and remove the SD nodes from the chorus.

[0132] Conclusion While various embodiments of the invention have been described and illustrated herein, those skilled in the art will readily envision various other means and / or structures for performing the functions and / or obtaining one or more of the results and / or advantages described herein, and each such variation and / or modification is considered to be within the scope of the embodiments of the invention described herein. More generally, all parameters, dimensions, materials, and configurations described herein are intended to be exemplary, and those skilled in the art will readily appreciate that the actual parameters, dimensions, materials, and / or configurations will depend on the particular application or applications for which the teachings of the invention are used. Those skilled in the art will recognize or be able to ascertain, using no more than routine experimentation, many equivalents to the specific embodiments of the invention described herein. Thus, the foregoing embodiments are presented by way of example only, and it will be understood that, within the scope of the appended claims and their equivalents, the embodiments of the invention may be practiced otherwise than as specifically described and claimed. The inventive embodiments of the present disclosure are directed to each individual feature, system, article, material, kit, and / or method described herein. In addition, any combination of two or more such features, systems, articles, materials, kits, and / or methods is within the inventive scope of the present disclosure, if such features, systems, articles, materials, kits, and / or methods are not mutually inconsistent.

[0133] Also, various inventive concepts may be embodied as one or more methods, examples of which are provided. The acts performed as part of a method may be ordered in any suitable manner. As such, embodiments may be constructed in which acts are performed in an order different from that shown, including performing some acts simultaneously even though the example embodiments show such acts as sequential acts.

[0134] All definitions and those used herein should be understood to supersede any dictionary definitions, definitions in documents incorporated by reference, and / or ordinary meaning of the defined term.

[0135] The indefinite articles "a" and "an," as used in the specification and claims, unless clearly dictated otherwise, should be understood to mean "at least one."

[0136] The term "and / or," as used herein and in the claims, should be understood to mean "either or both" of the elements so conjoined, i.e., elements that are conjunctive in some cases and disjunctive in other cases. Multiple elements listed with "and / or" should be construed in the same manner, i.e., as "one or more" of the elements so conjoined. Other elements, other than the elements specifically identified by the "and / or" clause, may optionally be present, whether related or unrelated to those elements specifically identified. Thus, as a non-limiting example, a reference to "A and / or B," when used in conjunction with open-ended language such as "comprising," can, in one embodiment, refer to A only (optionally including elements other than B), in another embodiment to B only (optionally including elements other than A), in yet another embodiment to both A and B (optionally including other elements), and so forth.

[0137] As used herein and in the claims, "or" shall be understood to have the same meaning as "and / or" as defined above. For example, when separating items in a list, "or" or "and / or" shall be construed as inclusive, i.e., including at least one of, but also including more than one of, an element or list of elements, and optionally additional unlisted items. Only terms that clearly indicate otherwise, such as "only one of" or "exactly one of," or "consisting of," when used in the claims, refer to the inclusion of exactly one element of an element or list of elements. In general, the term "or," when used herein, when accompanied by terms of exclusivity, such as "any of," "one of," "only one of," or "exactly one of," shall be construed to indicate only exclusive alternatives (i.e., "one or the other, but not both"). "Consisting essentially of," when used in the claims, shall have its ordinary meaning as used in the field of patent law.

[0138] As used herein and in the claims, the phrase "at least one" in connection with a list of one or more elements should be understood to mean at least one element selected from any one or more of the elements in the list of elements, and does not necessarily include at least one of each and every element specifically listed in the list of elements, nor does it exclude any combination of elements in the list of elements. This definition also recognizes that elements other than the elements specifically identified in the list of elements to which the phrase "at least one" refers may optionally be present, whether related or unrelated to those elements specifically identified. Thus, as one non-limiting example, "at least one of A and B" (or equivalently "at least one of A or B," or equivalently "at least one of A and / or B") can refer to, in one embodiment, at least one A, optionally including more than one A, and no B (and optionally including elements other than B); in other embodiments, at least one B, optionally including more than one B, and no A (and optionally including elements other than A); in yet other embodiments, at least one A, optionally including more than one A, and at least one B, optionally including more than one B (and optionally including other elements); and so forth.

[0139] In the claims, as well as in the above specification, all transitional phrases, such as "comprising," "including," "carrying," "having," "containing," "involving," "holding," "consisting of," and the like, are to be understood to be open-ended, i.e., to mean "including, but not limited to." Only the transitional phrases "consisting of" and "consisting essentially of" shall be closed or semi-closed transitional phrases, respectively, as set forth in the United States Patent Office Manual of Patent Examining Procedures, Section 2111.03.

Claims

1. 1. A method of resolving failures in a distributed database, the distributed database including a plurality of nodes, each node in the plurality of nodes being directly connected to every other node in the plurality of nodes, the method comprising, in response to detecting the failure, At a first node in the plurality of nodes, identifying a suspect node within the plurality of nodes, the suspect node being a node within the plurality of nodes that has become disconnected from the first node as a result of the failure; broadcasting a list of a first suspect node to adjacent nodes within the plurality of nodes, the list of first suspect nodes including the suspect node, the adjacent nodes being nodes within the plurality of nodes that remain directly connected to the first node after the failure; receiving a second list of suspect nodes from at least one of the neighboring nodes; determining connectivity information for the plurality of nodes based at least in part on the first list of suspect nodes and the second list of suspect nodes; determining, based on the connection information, whether the first node is in a winning all-connected component of the distributed database, the winning all-connected component including more than half of the nodes in the plurality of nodes, and each node in the winning all-connected component node being directly connected to every other node in the winning all-connected component node; in response to determining that the first node is within the winning all-connected component of the plurality of nodes, continuing operation of the first node; in response to determining that the first node is not within the winning all-connected components of the plurality of nodes, shutting down the first node to resolve the failure; The method comprising:

2. The method of claim 1 , wherein broadcasting the list of first suspect nodes comprises broadcasting a protocol iteration number indicating an iteration of the method invoked by the first node.

3. comparing the protocol iteration number with a protocol iteration number received with the second list of suspect nodes; The method of claim 2 , further comprising:

4. serializing the protocol iteration number as part of a master catalog, the master catalog including a list of the nodes within the plurality of nodes; The method of claim 2 , further comprising:

5. Determining the connection information includes, at the first node: determining a connectivity graph based at least in part on the connectivity information; identifying the winning all-connected component from the connectivity graph; The method of claim 1 further comprising:

6. Identifying the winning all-connected component comprises: determining the winning total connected component based on a size of the winning total connected component and a size of the plurality of nodes; The method of claim 5 , comprising:

7. The method of claim 1 , wherein determining whether the first node is within the winning all-connected component comprises identifying, at the first node, the winning all-connected component based on the connectivity information.

8. Identifying the winning all-connected component comprises: identifying a first overall connected component of the distributed database based on the connection information, wherein each node in the first overall connected component is directly connected to every other node in the first overall connected component; identifying a second fully connected component of the distributed database based on the connection information, the second fully connected component being different from the first fully connected component, and each node in the second fully connected component being directly connected to every other node in the second fully connected component; determining that the first total connected component includes (i) more nodes than the second total connected component, and (ii) more than half of the nodes in the plurality of nodes; selecting the first fully connected component as the winning fully connected component; The method of claim 7, comprising:

9. Identifying the winning all-connected component comprises: identifying a first overall connected component of the distributed database based on the connection information, wherein each node in the first overall connected component is directly connected to every other node in the first overall connected component; identifying a second fully connected component of the distributed database based on the connection information, the second fully connected component being different from the first fully connected component, and each node in the second fully connected component being directly connected to every other node in the second fully connected component; determining that the first total connected component includes (i) the same number of nodes as the second total connected component, and (ii) more than half of the nodes in the plurality of nodes; selecting the first all-connected component as the winning all-connected component based on unique identifiers of the nodes in the first all-connected component and the second all-connected component; The method of claim 7, comprising:

10. transmitting the second list of suspect nodes from the first node to at least one neighboring node; The method of claim 1 further comprising:

11. updating the first list of suspect nodes based at least in part on the second list of suspect nodes; broadcasting the updated list of first suspect nodes from the first node to the neighboring nodes; The method of claim 1 further comprising:

12. The method of claim 1 , wherein the first node identifies less than half of the plurality of nodes as suspect nodes.

13. identifying, at a third node within the plurality of nodes, more than half of the plurality of nodes as suspect nodes; taking down the third node to resolve the failure; and The method of claim 1 further comprising:

14. and in response to detecting the failure, failing a third node attempting to join the plurality of nodes. The method of claim 1 further comprising:

15. The failure is a first failure, the suspect node is a first suspect node, and at a third node in the plurality of nodes, Detecting a second failure in the distributed database; identifying a second suspect node within the plurality of nodes, the second suspect node being a node within the plurality of nodes that is no longer directly connected to the third node as a result of the second failure; broadcasting a list of third suspect nodes to the first node; restarting the method by the first node; The method of claim 1 further comprising:

16. 1. A method for resolving failures in a distributed database, the distributed database including a plurality of nodes, each node in the plurality of nodes being directly connected to every other node in the plurality of nodes, the method comprising: At a first node in the plurality of nodes, detecting an interruption in communication with a second node in the plurality of nodes; initiating, in response to detecting the interruption, a cooperative broadcast of a list of respective suspect nodes among neighboring nodes in the plurality of nodes, the neighboring nodes being nodes in the plurality of nodes that remain directly connected to the first node, the list of suspect nodes of the first node including the second node; determining connectivity information based on the respective list of suspect nodes; resolving the fault based at least in part on the connection information; and The method comprising:

17. At each adjacent node, performing a comparison of the neighboring node's list of suspect nodes with the list of suspect nodes from at least one other neighboring node; determining at least a portion of the connectivity information based at least in part on the comparison of the neighboring node's list of suspect nodes with the list of suspect nodes from the at least one other neighboring node; 20. The method of claim 16, further comprising:

18. preparing, at each of said adjacent nodes, a connectivity graph based at least in part on said connectivity information; 20. The method of claim 16, further comprising:

19. The method of claim 18 , wherein the first node disables itself based at least in part on the connectivity graph.

20. 20. The method of claim 18, wherein the cooperative broadcasting includes broadcasting by the first node a protocol iteration number indicating the fault resolution protocol that the first node invoked to detect the fault.

21. 1. A method for resolving failures in a distributed database, the distributed database including a plurality of nodes, each node in the plurality of nodes being connected to every other node in the plurality of nodes, the method comprising, in response to detecting the failure, At a first node in the plurality of nodes, determining whether the first node is directly connected to at least half of the nodes in the plurality of nodes; in response to determining that the first node is directly connected to less than half of the nodes in the plurality of nodes, taking down the first node to at least partially resolve the failure; in response to determining that the first node is directly connected to at least half of the nodes in the plurality of nodes, broadcasting a list of first suspect nodes to adjacent nodes in the plurality of nodes, the list of first suspect nodes including nodes that are not directly connected to the first node, the adjacent nodes being nodes that remain directly connected to the first node after the failure; receiving a second list of suspect nodes from at least one of the neighboring nodes; determining whether the first list of suspect nodes matches the second list of suspect nodes; in response to determining that the list of the first suspect node matches the list of the second suspect node, maintaining the first node operational in at least partially resolving the failure; in response to determining that the first list of suspect nodes does not match the second list of suspect nodes, broadcasting a first updated list of suspect nodes to the neighboring nodes based on the first list of suspect nodes and the second list of suspect nodes; receiving at least a second updated list of suspect nodes from at least one of the neighboring nodes; determining connectivity information of the plurality of nodes based at least in part on the first updated list of suspect nodes and the second updated list of suspect nodes; determining a winning all-connected component of the distributed database based on the connection information, the winning all-connected component including more than half of the nodes in the plurality of nodes, and each node in the winning all-connected component being directly connected to every other node in the winning all-connected component; determining whether the first node is within the winning all-connected component; in response to determining that the first node is within the winning all-connected component of the plurality of nodes, continuing operation of the first node to at least partially resolve the failure; in response to determining that the first node is not within the winning all-connected components of the plurality of nodes, shutting down the first node to at least partially resolve the failure; The method comprising:

22. 1. A distributed database system comprising: a plurality of nodes, each node in the plurality of nodes including a corresponding processor and a corresponding memory, and directly connected to every other node in the plurality of nodes, the processor of a first node in the plurality of nodes resolving failures in the distributed database system; identifying a suspect node within the plurality of nodes, the suspect node being a node within the plurality of nodes that has become disconnected from the first node as a result of a failure within the distributed database system; broadcasting a list of a first suspect node to adjacent nodes within the plurality of nodes, the list of first suspect nodes including the suspect node, the adjacent nodes being nodes within the plurality of nodes that remain directly connected to the first node after the failure; receiving a second list of suspect nodes from at least one of the neighboring nodes; determining connectivity information for the plurality of nodes based at least in part on the first list of suspect nodes and the second list of suspect nodes; determining, based on the connection information, whether the first node is in a winning all-connected component of the distributed database, the winning all-connected component including more than half of the nodes in the plurality of nodes, and each node in the winning all-connected component node being directly connected to every other node in the winning all-connected component node; in response to determining that the first node is within the winning all-connected component of the plurality of nodes, continuing operation of the first node; in response to determining that the first node is not within the winning all-connected components of the plurality of nodes, shutting down the first node to resolve the failure; The distributed database system is configured to perform the above-mentioned

23. 23. The distributed database system of claim 22, wherein the processor of the first node is configured to broadcast a fault resolution iteration number along with the list of the first suspect nodes.

24. 24. The distributed database system of claim 23, wherein the processor of the first node is configured to compare the protocol iteration number with a protocol iteration number received with the list of the second suspect node.

25. 24. The distributed database system of claim 23, wherein the processor of the first node is further configured to serialize the protocol iteration number as part of a master catalog, the master catalog including a list of the nodes within the plurality of nodes.

26. The processor of the first node determines the connection information. determining a connectivity graph based at least in part on the connectivity information; identifying the winning all-connected component from the connectivity graph; 23. The distributed database system of claim 22, configured to:

27. Identifying the winning all-connected component comprises: determining the winning total connected component based on a size of the winning total connected component and a size of the plurality of nodes; 27. The distributed database system of claim 26, comprising:

28. 23. The distributed database system of claim 22, wherein determining whether the first node is within the winning all-connected component comprises identifying the winning all-connected component based on the connection information.

29. Identifying the winning all-connected component comprises: identifying a first overall connected component of the distributed database based on the connection information, wherein each node in the first overall connected component is directly connected to every other node in the first overall connected component; identifying a second fully connected component of the distributed database based on the connection information, the second fully connected component being different from the first fully connected component, and each node in the second fully connected component being directly connected to every other node in the second fully connected component; determining that the first total connected component includes (i) more nodes than the second total connected component, and (ii) more than half of the nodes in the plurality of nodes; selecting the first fully connected component as the winning fully connected component; 30. The distributed database system of claim 28, comprising:

30. Identifying the winning all-connected component comprises: identifying a first overall connected component of the distributed database based on the connection information, wherein each node in the first overall connected component is directly connected to every other node in the first overall connected component; identifying a second fully connected component of the distributed database based on the connection information, the second fully connected component being different from the first fully connected component, and each node in the second fully connected component being directly connected to every other node in the second fully connected component; determining that the first total connected component includes (i) the same number of nodes as the second total connected component, and (ii) more than half of the nodes in the plurality of nodes; selecting the first all-connected component as the winning all-connected component based on unique identifiers of the nodes in the first all-connected component and the second all-connected component; 30. The distributed database system of claim 28, comprising:

31. The processor of the first node resolves the failure in the distributed database system.

23. The distributed database system of claim 22, further configured by transmitting the second list of suspect nodes from the first node to at least one adjacent node.

32. The processor of the first node resolves the failure in the distributed database system. updating the first list of suspect nodes based at least in part on the second list of suspect nodes; broadcasting the updated list of first suspect nodes from the first node to the neighboring nodes; 23. The distributed database system of claim 22, further configured to:

Images