DIAGGRAPHED MULTI-CLUSTER-LEVEL STORAGE

Abstract

Example implementations relate to disaggregated storage systems. An example procedure might involve initializing a plurality of Level 1 (L1) clusters in a disaggregated storage system, with each L1 cluster containing multiple compute nodes. The procedure might also involve choosing an L1 leader node in each L1 cluster and forming a Level 2 (L2) cluster with the L1 leader nodes. Alternatively, the procedure might involve choosing an L2 leader node from among the L1 leader nodes in the L2 cluster.

Description

background

[0001] Some computer systems can store and access data in storage networks. A storage network can comprise a group of devices, referred to here as "nodes," that are interconnected via a communication medium (e.g., a network). In some examples, each node can contain hardware and software components.

[0002] Castiglia, Timothy; Goldberg, Colin; Patterson, Stacey: A hierarchical model for fast distributed consensus in dynamic networks, July 27, 2020, refers to a hierarchical model for fast distributed consensus in dynamic networks. Brief description

[0003] It discloses a method according to claims 1 to 6, a non-transitory, machine-readable medium according to claims 7 to 10 and a computing node according to claims 11 to 14. Brief description of the drawings

[0004] Some embodiments are described with reference to the following illustrations. Fig. This is a schematic representation of an example node cluster according to some implementations. Fig. This is an illustration of a sample process in accordance with some implementations. Fig. These are illustrations of an example system according to some implementations. Fig. These are illustrations of example processes in accordance with some implementations. Fig. This is a representation of an exemplary data structure in accordance with some implementations. Fig. These are illustrations of example compute nodes according to some implementations. Fig. This is an illustration of a sample process in accordance with some implementations. Fig. is a diagram of a machine-readable medium on which instructions are stored in accordance with some implementations. Fig. is a schematic diagram of an example compute node according to some implementations.

[0005] In the drawings, identical reference numbers denote similar, but not necessarily identical, elements. The illustrations are not necessarily to scale, and the size of some parts may be exaggerated to make the example shown clearer. Furthermore, the drawings contain examples and / or embodiments that correspond to the description; however, the description is not limited to the examples and / or embodiments shown in the drawings. Detailed description

[0006] In this disclosure, the use of the term "a," "an," or "the" includes the plural forms unless the context clearly indicates otherwise. Similarly, the terms "includes," "including," "comprises," "have," or "have," when used in this disclosure, specify the presence of the elements indicated but do not exclude the presence or addition of other elements.

[0007] In some examples, a disaggregated storage system can include compute nodes and storage devices coupled via network connections. For instance, the disaggregated storage system might include physical storage devices, physical compute nodes, one or more virtual storage devices, one or more virtual compute nodes, or a combination of one or more virtual storage devices and one or more virtual compute nodes. A storage device can contain or manage any number of storage components for the persistent storage of data. Each compute node can be a compute device (e.g., a server, controller, etc.) that can access the data stored in the storage devices. In some examples, the storage devices might collectively hold a variety of volumes (e.g., virtual volumes) or other data objects (e.g.,Provide regions, parts, units, files, or similar entities for storing data. In such examples, only one compute node at a time can have permission to modify the data on a particular storage medium (which can be referred to here as the compute node to which that particular storage medium is "owned" or "assigned"). While the examples described here refer to storage mediums, in other examples, the respective nodes could own or be assigned any other type of data object in a disaggregated storage system. A potential problem with such a disaggregated storage system, however, is to efficiently and reliably determine which compute nodes own which volumes (or other parts, etc.) of the disaggregated storage system at any given time.In some examples, compute nodes can be grouped into a cluster, and cluster management techniques can employ consensus mechanisms to ensure that there is agreement among the compute nodes regarding disk ownership, an agreement that can survive node failures within the cluster. A potential problem with such cluster management techniques is enabling the consensus mechanisms to operate efficiently and survive node failures when managing a relatively large number of compute nodes in the cluster.

[0008] As mentioned above, a disaggregated storage system can, in some examples, implement consensus-based cluster management techniques to maintain the consistency of cluster management data (including, for example, information about which compute nodes own which volumes at a given time) in the event of node failures. In some examples, some or all compute nodes in a cluster can be grouped, and the compute nodes in the cluster can perform an election to select a specific compute node in the cluster to be the cluster leader. The compute node that is the cluster leader can be responsible for updating the cluster management data for the cluster and for managing the replication of the cluster management data to the other compute nodes in the cluster (also referred to here as "followers"). For example, if the leading compute node is to modify the stored cluster management data (e.g.,In response to a request, the leading compute node can record the change (or request) in a log entry and transmit the log entry to all subsequent compute nodes, thereby informing them of the requested changes to the cluster management data. If the log entry is acknowledged by a minimum percentage of the subsequent compute nodes (referred to as "reaching consensus"), the leading compute node can consider the log entry acknowledged. Accordingly, the leading compute node can then initiate the requested change to the cluster management data.

[0009] In some examples, increasing the number of compute nodes in the cluster can increase the probability that at least a minimum number of nodes will remain operational during or after a failure event, thereby improving the cluster's fault tolerance. However, in a single cluster comprising more than a certain number of compute nodes (e.g., more than seven), the consensus process can require a relatively large number of messages to propagate the log entry to subsequent compute nodes and obtain acknowledgments from them, resulting in significant time and bandwidth consumption. Consequently, the performance of a single cluster with a relatively large number of compute nodes can suffer during operations that require consensus among the compute nodes.

[0010] In accordance with some implementations of this disclosure, a disaggregated storage system may include storage devices and a relatively large number of compute nodes (e.g., more than seven) arranged in two or more cluster levels. The compute nodes may be divided into multiple Level 1 (or, here, "L1") clusters, with each L1 cluster containing multiple compute nodes. In each L1 cluster, one node of the L1 cluster may be elected by the compute nodes in that L1 cluster to be the leader of the L1 cluster (referred to here as the "L1 leader" or "L1 leader node"). In some implementations, one compute node from each of the L1 clusters (e.g., the L1 leader nodes) may be grouped into a Level 2 (or, here, "L2") cluster, with each of the compute nodes in the L2 cluster being both a node (or "member") of the L2 cluster and of a corresponding L1 cluster.The compute nodes of the L2 cluster can elect a leader of the L2 cluster (referred to here as the "L2 leader" or "L2 leader node"). The compute nodes in the L2 cluster can be responsible for maintaining and updating multiple copies of cluster management data, as described above, for the disaggregated storage system. For example, the cluster management data maintained and updated by the compute nodes of the L2 cluster can be cluster management data for all compute nodes of the disaggregated storage system or for the compute nodes in the L1 clusters. For example, the cluster management data maintained by the compute nodes of the L2 cluster can specify for all compute nodes of the disaggregated storage system (and / or in one of the L1 clusters) which compute nodes own which volumes (or other parts) of the disaggregated storage system.Updating multiple copies of the cluster management data may involve reaching consensus among the compute nodes (e.g., L1 leaders) in the L2 cluster. In such examples, the remaining compute nodes of the L1 cluster (i.e., the compute nodes not in the L2 cluster) do not participate in reaching this consensus. This allows consensus to be reached without exchanging a relatively large number of messages between all compute nodes in the L1 clusters. In some examples, the compute nodes of the L1 cluster not in the L2 cluster (e.g., L1 cluster follower nodes) may be responsible for choosing the L1 leaders, including replacing L1 leaders that fail during operation. In some examples, each of the L1 leaders of the L1 clusters may be a member of the L2 cluster.In such examples, the follower nodes in the L1 clusters can be used to compensate for the failure of L1 leaders and, consequently, the failure of compute nodes in the L2 cluster. In this way, the hierarchical structure of the L1 and L2 clusters can improve the fault tolerance of the disaggregated storage system by including a relatively large number of compute nodes without suffering the performance loss associated with using a relatively large number of compute nodes in a single cluster (i.e., due to the large number of messages required to reach consensus among all member nodes in the single cluster).

[0011] In some examples, a storage device (e.g., a physical storage device) may contain one or more storage controllers that manage access to stored data. A "data unit" can refer to any portion of data that can be managed separately within the storage system. In some cases, a data unit may refer to a block, a chunk, a collection of chunks, or some other part of data. In some examples, a storage system may store data units in persistent storage. Persistent storage can be achieved using one or more persistent (e.g., non-volatile) storage devices, such as disk-based storage devices (e.g., hard disk drives (HDDs)), solid-state devices (SSDs), such as flash memory devices, or a combination thereof.

[0012] A "control unit" can refer to a hardware processing circuit that may include any or a combination of a microprocessor, a core of a multi-core microprocessor, a microcontroller, a programmable integrated circuit, a programmable gate array, a digital signal processor, or another hardware processing circuit. Alternatively, a "control unit" can refer to a combination of a hardware processing circuit and machine-readable instructions (software and / or firmware) that can be executed on the hardware processing circuit. Fig. 1 - Example of a node cluster

[0013] Fig. Figure 1 shows an example of a Level 1 (L1) cluster 100 in accordance with some implementations. As shown, the L1 cluster 100 can comprise multiple compute nodes 110A-110G (also referred to as "compute nodes 110") interconnected via a network 105. The L1 cluster 100 can be connected to storage devices 120. The L1 cluster 100 and the storage devices 120 can be contained in a disaggregated storage system. In some implementations, the storage devices 120 can contain persistent storage implemented using one or more persistent (e.g., non-volatile) storage devices, such as disk-based storage devices (e.g., hard disk drives (HDDs)), solid-state devices (SSDs) such as flash memory devices, or a combination thereof.In some examples, the storage devices 120 can be connected to the compute nodes 110 via the NVMe-oF (Non-Volatile Memory Express over Fabrics) interface, the iSCSI (Internet Small Computer Systems Interface) interface, or similar.

[0014] In some implementations, each compute node 110 can be assigned a fixed network address (e.g., a fixed Internet Protocol (IP) address). Furthermore, each compute node 110 can be implemented by a compute device (e.g., a server) that includes controllers, memory, storage devices, network devices, etc. (in Fig. (not shown). For example, each compute node 110 can be a physical computing device or a virtual device hosted on a computing device. An example implementation of a compute node 110 is described below with reference to the Fig. described.

[0015] In some implementations, each compute node can have 110 processing resources for running the L1 cluster management software (130). Fig. The L1 Cluster Management Software 100 (L1 Cluster SW 130) is stored on a machine-readable storage medium to provide the L1 Cluster 100. Furthermore, the L1 Cluster Management Software 130 (e.g., the etcd software) can employ consensus-based management with an elected leader (e.g., using the Raft consensus algorithm). For example, suppose that the compute nodes 110 previously conducted an election and chose compute node 110A as the L1 leader (as indicated by the designation "L1 leader" in the L1 Cluster SW 130). Fig. (displayed). In one or more implementations, conducting the election for the L1 leader requires a quorum of compute nodes 110 in L1 cluster 100. As used here, a "quorum" of nodes refers to a minimum number or percentage of nodes that must be operational for an election to take place. In the examples described here, a processing resource can include one or more processors (or other electronic circuits) for executing instructions.

[0016] In some implementations, L1 cluster 100 can be one of several L1 clusters belonging to a disaggregated storage system. Furthermore, in some implementations, a compute node from each L1 cluster can be grouped to form a Level 2 (L2) cluster 150. Several examples are described here where the respective L1 leaders are selected to join the L2 cluster. However, in other examples, any node from a given L1 cluster can be selected to join the L2 cluster. Note that in Fig. The L2 cluster 150 is shown as a dashed line to indicate that the L2 cluster 150 is not included in the L1 cluster 100 (i.e., only the L1 leader 110A is included in both the L1 cluster 100 and the L2 cluster 150).

[0017] In some implementations, each member of the L2 cluster 150 can mount a file system 140 in which cluster management data is stored. For example, Fig. The leading node 110A (also referred to as "L1 leader 110A") can be a member of the L2 cluster 150 and mount the file system 140. In some implementations, the L1 leader 110A can be assigned a representative network address associated with the leadership of the L1 cluster 100. In some implementations, the file system 140 can be contained in the storage devices 120.

[0018] In each L1 cluster, the L2 cluster member (e.g., the L1 leader node) can mount a corresponding file system 140 and write and modify cluster management data of the L2 cluster 150. However, the nodes of the L1 clusters that are not in the L2 cluster (e.g., the L1 follower nodes in each L1 cluster) cannot write or modify the cluster management data of the L2 cluster 150. Furthermore, the follower nodes may be responsible for selecting the L1 leader, including replacing L1 leaders that fail during operation. Note that the L1 leader node and the follower nodes may perform other tasks unrelated to managing the L1 cluster 100 or the L2 cluster 150.

[0019] In some implementations, each L1 leader in the L2 cluster can use the L2 cluster management software (135). Fig. L1 leader (referred to as "L2 Cluster SW 135") can execute instructions from the L2 Cluster Management Software 135, which is stored on a machine-readable storage medium. In some implementations, the L1 leader can execute the L2 Cluster Management Software 135 using the representative network address for its respective L1 cluster. Furthermore, the L1 Cluster Management Software 130 and the L2 Cluster Management Software 135 can be two different software applications.

[0020] In some implementations, each L1 cluster can be statically assigned its own storage partition. Each L1 leader can also access the storage partition assigned to its respective L1 cluster to mount its file system. Each storage partition can hold a separate copy of the cluster management data (also referred to as "L2 cluster management data") used by the L2 cluster. The stored L2 cluster management data might, for example, contain data (such as a set of key-value pairs) that specifies which compute node is responsible for (i.e., owns) accessing each volume of data (or other data object) stored in the disaggregated storage system. In another example, the stored L2 cluster management data might specify which node is responsible for each service available in the disaggregated storage system.

[0021] Alternatively, in some implementations, each L1 leader may not have access to the storage allocated in the disaggregated storage for file system 140 for storing cluster management data. In such implementations, for example, each L1 leader can store their respective copy of the L2 cluster management data in their local storage (e.g., on a storage device in compute node 110A). If an L1 leader fails, a new L1 leader can be elected in the respective L1 cluster. In such examples, the new L1 leader might need to rejoin L2 cluster 150 and then obtain a new copy of the L2 cluster management data from the current L2 leader.

[0022] In some implementations, a disaggregated storage system can include two or more cluster levels (e.g., L1 cluster 100 and L2 cluster 150). This hierarchical, multi-cluster-level structure can allow the use of a relatively large number of nodes (e.g., more than seven) without consuming the time and bandwidth required to reach consensus among all involved nodes (i.e., when they are contained in a single cluster). Accordingly, the hierarchical structure described here can improve the fault tolerance of the disaggregated storage system by including a relatively large number of nodes without suffering the performance loss associated with using a relatively large number of nodes in a single cluster. Some example implementations of disaggregated storage with a hierarchy of multiple cluster levels are described below with reference to the Fig. described.

[0023] In some implementations, each L1 cluster can contain both an L1 leader and an L2 member. In some examples, the L1 leader and the L2 member of an L1 cluster can be two different compute nodes within the L1 cluster. For example, in the L1 cluster example 100 of Fig. After a first compute node (e.g., node 110A) is elected as the L1 leader of L1 cluster 100, a second compute node of L1 cluster 100 (e.g., node 110C) is selected (e.g., via an election or other mechanism) to be the L2 member of L1 cluster 100. Furthermore, in such implementations, the second compute node (e.g., node 110C) would mount file system 140 to write and modify cluster management data of L2 cluster 150. Using two different compute nodes of an L1 cluster for the L1 leader and the L2 member can occur in certain situations, e.g., For example, if the L1 leader is to be assigned other tasks that require a significant processing load, and therefore assigning the tasks of the L2 member to another compute node in the L1 cluster may be more efficient for the system as a whole. Figures 2 and 3A-3J - Example process and system for disaggregated storage

[0024] Fig. This shows an example process 200 for disaggregated memory, in accordance with some implementations. Process 200 can be run using compute node 110 (in Fig. Process 200 can be implemented in hardware or a combination of hardware and programming (e.g., machine-readable instructions that can be executed by one or more processors). The machine-readable instructions can be stored on a non-transitory, computer-readable medium, such as an optical, semiconductor, or magnetic storage device. The machine-readable instructions can be executed by a single processor, multiple processors, or other electronic circuits. For illustration, the details of Process 200 are given below with reference to Fig. The examples shown are consistent with some implementations of a disaggregated storage system. However, other implementations are also possible.

[0025] Block 210 can include the initialization of multiple Level 1 (L1) clusters in a disaggregated storage system. Block 215 can include the election of an L1 leader by the members of each L1 cluster. For example, in a disaggregated storage system 300, multiple L1 clusters 101-105 can be initialized (see Fig. Each L1 cluster can contain multiple compute nodes (e.g., L1 cluster 101 with compute nodes 111A-111G, L1 cluster 102 with compute nodes 112A-112G, etc.). Furthermore, in each L1 cluster, one compute node can be elected as the L1 leader by the other compute nodes in that L1 cluster. As described in Fig. As shown, for example, compute node 111A is elected as the L1 leader of L1 cluster 101 (as indicated by the label "L1 leader"), node 112A is elected as the L1 leader of L1 cluster 102, and so on. In some implementations, performing the election of an L1 leader requires a quorum of compute nodes in that L1 cluster. Furthermore, performing the election of the L1 leader involves each compute node separately performing actions to participate in the election. For example, in each L1 cluster, a controller on each compute node can execute instructions from the L1 cluster software (e.g., those shown in the diagram). Fig. Run the L1 cluster management software shown (130) to cast a vote in the election of the L1 leader.

[0026] In some implementations, each compute node in the disaggregated storage system 300 can be assigned a fixed network address. Furthermore, the current L1 leader of each L1 cluster can be assigned a different representative network address connected to the line of that particular L1 cluster. For example, suppose that the representative network address IP1 is assigned to the compute node currently acting as the L1 leader of L1 cluster 101, the representative network address IP2 is assigned to the compute node currently acting as the L1 leader of L1 cluster 102, and so on.

[0027] Referring again to Fig. Block 220 can involve the formation of a Level 2 (L2) cluster, which includes the L1 leaders of the L1 clusters. Block 225 can involve the election of an L2 leader by the members of the L2 cluster. For example, L2 cluster 310 can be formed from the L1 leaders of each L1 cluster (e.g., L1 leader 111A from L1 cluster 101, L1 leader 112A from L1 cluster 102, etc.) (see Fig. Each L1 leader can be designated as the "representative" of its respective L1 cluster in L2 cluster 310. Furthermore, the members of L2 cluster 310 can conduct an election, thereby electing compute node 111A as the L2 leader of L2 cluster 310 (as indicated by the designation "L2 leader"). In one or more implementations, conducting the L2 leader election requires a quorum of L1 leaders in L2 cluster 310 (i.e., a minimum number or percentage of L1 leaders willing to participate in the election). Additionally, conducting the L2 leader election requires each L1 leader to take separate actions to participate in the election. For example, a controller in each L1 leader can issue instructions to the L2 cluster software (e.g., the one in Fig. Run the L2 cluster management software shown (135) to cast a vote in the election of the L2 leader. In some implementations, the current L2 leader can be assigned a specific network address associated with leading L2 cluster 150. For example, suppose that the network address IP50 is assigned to the compute node that is currently acting as the L2 leader of L2 cluster 310.

[0028] Referring again to Fig. Block 230 can involve the processing of data requests in the L2 cluster. For example, the L2 leader 111A might receive a request to change the cluster management data of L2 cluster 310 (e.g., to change the compute node responsible for accessing a specific volume of stored data) (see Fig. The request can be received directly from the client or indirectly from another node of the disaggregated storage system 300. The L2 leader 111A can then record the request in a log entry and forward the log entry to the subsequent nodes of the L2 cluster 310 (i.e., the L1 leaders 112A, 113A, 114A, and 115A). Each L1 leader can update its respective copy of the L2 cluster management data (i.e., store it in the storage partition allocated to the respective L1 cluster) to reflect the log entry and can acknowledge the log entry to the L2 leader when the update of its respective copy of the L2 cluster management data is complete. The L2 leader can determine when enough of the subsequent nodes of the L2 cluster 310 have acknowledged the log entry to reach consensus.In some implementations, consensus can be reached when the number or percentage of received acknowledgments exceeds a defined threshold (e.g., when at least 50% of the other nodes have acknowledged). Once the L2 leader node 111A determines that consensus has been reached, the request can be executed and / or acknowledged (e.g., by the L2 leader making the requested change to the L2 cluster management data).

[0029] Referring again to Fig. Decision block 240 can include determining whether the L2 leader has failed. If it is determined that the L2 leader has not failed ("NO"), then process 200 can continue in decision block 280, including determining whether an L1 leader has failed. If it is determined that no L1 leaders have failed ("NO"), process 200 can return to block 230 (i.e., continue processing requests in the L2 cluster). However, if decision block 280 determines that at least one L1 leader has failed ("YES"), process 200 can continue in block 290, including replacing a failed L1 leader by election and having the newly elected L1 leader assume the failed L1 leader's position in the L2 cluster. After block 290, process 200 can return to block 230.However, if decision block 240 determines that the L2 leader has failed (“YES”), process 200 can continue in decision block 250, including determining whether the L2 cluster is still quorate. If decision block 250 determines that the L2 cluster is still quorate (“YES”), process 200 can continue in block 270, including the election of a new L2 leader by the available L1 leaders (i.e., L1 leaders who have not failed) from among the available L1 leaders. After block 270, process 200 can continue with block 290 (as described above). Fig. For example, it shows that the L2 leader 111A in the disaggregated storage system 300 has failed. Subsequently, the remaining nodes of the L2 cluster 310 (i.e., the L1 leaders 112A, 113A, 114A, 115A) perform an election in which compute node 112A (i.e., the L1 leader in the L1 cluster 102) is selected as the new L2 leader (see Fig. ). As in Fig. As shown, L1 cluster 101 has lost its L1 leader (i.e., the failed compute node 111A) and therefore no longer has a representative in L2 cluster 310. Accordingly, the remaining nodes of L1 cluster 101 (i.e., compute nodes 111B-111G), as shown in Fig. This is illustrated by an election in which compute node 111B is selected as the new L1 leader for L1 cluster 101. The new L1 leader 111B can then be incorporated into L2 cluster 310 as a representative for L1 cluster 101 (e.g., by assigning it the representative network address for L1 cluster 101 contained within L2 cluster 310). L2 cluster 310 can then resume processing client requests. In this way, the disaggregated storage system 300 can recover from the loss of an L2 leader or an L1 leader and resume processing client requests.

[0030] If decision block 250 determines that the L2 cluster is no longer able to make decisions (“NO”), process 200 can continue in block 260, including restoring the quorum in the L2 cluster and electing a new L2 leader. After block 260, process 200 can return to block 230 (as described above). Fig. This shows, for example, that all L1 leaders in the disaggregated storage system 300 have failed, and therefore the L2 cluster 310 lacks the capacity to make decisions and conduct an election. In response, as shown in Fig. As shown, each L1 cluster selects a new L1 leader, and these new L1 leaders (i.e., L1 leaders 111C, 112C, 113C, 114C, 115C) can be admitted to (or otherwise join) L2 cluster 310. As shown in Fig. As shown, the member nodes of L2 cluster 310 can then perform an election that selects compute node 112C (i.e., the L1 leader in L1 cluster 102) as the new L2 leader. In this way, the disaggregated storage system 300 can recover from the loss of all L1 leaders and resume processing client requests. Note that in Fig. Although it is shown that all failed L1 leaders are replaced before the election of the L2 leader takes place, implementations are not restricted in this respect. For example, if the required quorum for L2 cluster 310 is three members, the L2 leader can be elected after only three L1 leaders have been replaced and are included in L2 cluster 310.

[0031] In some implementations, the disaggregated storage system 300 can recover from the loss of a maximum number or percentage of nodes (e.g., 70% of the total number of nodes). Fig. For example, it is shown that the disaggregated storage system 300 has lost a maximum number of nodes. In response, it selects, as in Fig. As shown, each L1 cluster that still contains functioning nodes (i.e., L1 clusters 103, 104, 105) gets a new L1 leader, and these new L1 leaders (i.e., L1 leaders 113D, 114D, 115D) can be incorporated into L2 cluster 310. As shown in Fig. As shown, the member nodes of L2 cluster 310 can then perform an election that selects compute node 113D (i.e., the L1 leader in L1 cluster 103) as the new L2 leader. In this way, the disaggregated storage system 300 can recover from the loss of a maximum number of compute nodes and resume processing client requests.

[0032] The examples described here can enable efficient consensus mechanisms by using multiple cluster levels, even when a relatively large number of compute nodes are present in the cluster being managed, while simultaneously being able to withstand the failure of multiple compute nodes. In the example of Fig. For example, the disaggregated storage system 300 comprises 35 nodes. Fig. The example configuration shown, with five L1 clusters of seven members each and one L2 cluster comprising one node from each of the L1 clusters, can significantly reduce network overhead compared to a flat cluster with 35 members (e.g., by 1 / 7 or 86%), while decreasing fault tolerance by about 35% in the worst case and increasing fault tolerance by about 35% in the best case.

[0033] For example, if all 35 nodes were contained in a single, flat (i.e., non-tiered) cluster with all 35 nodes as members of the cluster, then any transaction initiated with the leader of that cluster could be replicated to the other 34 members of the cluster. In contrast, in the example of Fig. The L2 leader replicates a transaction via the cluster management data to the four other L2 members, which is far less. Specifically, in the example of Fig. The original request comprises five requests (i.e., the original request and four replication requests), whereas the flat cluster can comprise 35 requests for a transaction (i.e., the original request and 34 replication requests), which can significantly reduce network overhead (5 / 35 = 1 / 7). Furthermore, worst-case resilience would be the minimum number of node failures that could lead to a loss of quorum. In the example of Fig. Twelve nodes are the minimum number whose failure could lead to a loss of quorum (i.e., specifically the loss of four members from each of the three different L1 clusters [4*3 = 12]), whereas in a flat cluster with 35 nodes, the loss of 18 of the 35 nodes would lead to a loss of quorum. Thus, the flat cluster can survive 17 failures, whereas the example in Fig. can survive 11 node failures (this corresponds to a reduction in fault tolerance of approximately 35% in the worst case [from 17 to 11] for the example in Fig. The best-case resilience would be the maximum number of nodes that can fail before the quorum is lost. As in Fig. As illustrated, the example of Fig. 7 out of 7 nodes in two L1 clusters (i.e., 14 nodes) and 3 out of 7 nodes in the other three L1 clusters (i.e., 9 more nodes for a total of 23 nodes) can fail and still remain quorate (whereas the loss of the 24th node would cause a failure). For the shallow cluster, there is no difference between minimum and maximum fault tolerance, so it could lose 17 out of 35 nodes before a failure occurs. Thus, the example in Fig. In the best case scenario, the reliability will increase by 35%, from 17 node failures to 23 node failures.

[0034] The in Fig. The example configuration shown can also be compared to another alternative without multiple cluster levels, where, for example, a fixed set of 5 of the 35 nodes are selected as cluster members, while the other 30 nodes are not members of the cluster (i.e., they do not participate in quorum determination). Such an example can exhibit a similar amount of steady-state network traffic, since the fixed set of 5 members is the same size as the L2 cluster in the example. Fig. With a fixed set of 5 member nodes, however, there is no automatic recovery if a member node fails, and it is the responsibility of an administrator to repair the node to restore the cluster to its full membership. In the fixed cluster, the worst-case scenario is that all failures occur among the fixed member nodes, and in such a case, the failure of the third node would result in a loss of quorum, meaning the alternative would only guarantee survival for two node failures. In contrast, the example of Fig. at least 11 failures (as described above) represent a 550% improvement over the fixed-set alternative (i.e., from 2 nodes to 11 nodes). 11 / 2 = 550%. The best fault tolerance of the fixed-set cluster alternative would be the loss of all non-member nodes and 2 member nodes, allowing it to survive up to 32 node failures. The example in Fig. In the best case, it is therefore 28% less resilient (i.e., failures from 32 to 23 knots are survivable).

[0035] Note that, although Fig. As the disaggregated storage system 300 with two cluster levels (i.e., L1 and L2 clusters) demonstrates, implementations in this respect are not limited. In particular, it is conceivable that the disaggregated storage system 300 could include more than two cluster levels. An example of a storage system with more than two cluster levels is given below with reference to Fig. described.

[0036] Note also that while the use of multiple cluster levels is described above as an implementation in a disaggregated storage system, implementations in this regard are not limited. In particular, it is conceivable that multiple cluster levels (e.g., those described above with reference to...) could be used in a disaggregated storage system. Fig. The described L1 and L2 clusters can be implemented in other types of storage systems, in other types of computer systems, etc. Fig. 4A-4D - Exemplary procedures for setting up a disaggregated storage system

[0037] Fig. They demonstrate example procedures for setting up a disaggregated storage system in accordance with several implementations. For example, the ones in the Fig. The processes shown can be performed, for example, to disaggregate the storage system 300 (shown in Fig. to set up by adding compute nodes in specific growth directions.

[0038] In Fig. The disaggregated storage system 300 is depicted at the time of initialization. As shown, the disaggregated storage system 300 can be initialized with a single compute node 111A in the L1 cluster 101. The single compute node 111A can be referred to as the L1 leader. Starting from the in Fig. In the depicted state, the disaggregated storage system 300 can be expanded by successively adding a compute node to each of the L1 clusters 102-105 (represented by the arrow labeled "Growth direction A"). Furthermore, the individual compute node in each L1 cluster can be designated as the respective L1 leader.

[0039] As in Fig. As shown, the L1 leaders (i.e., L1 leaders 111A, 112A, 113A, 114A, 115A) can be grouped after an L1 leader has been designated in each L1 cluster to form the L2 cluster 310. Furthermore, the L1 leaders can hold an election and choose compute node 111A to be the L2 leader of L2 cluster 310. The disaggregated storage system 300 can then be extended to form L1 cluster 101 by adding a set of two compute nodes (or any other number of compute nodes, e.g., one, three, etc.) (represented by the arrow labeled "Growth Direction B").

[0040] Once the set of two compute nodes (i.e., compute nodes 111B and 111C) has been added to L1 cluster 101, the disaggregated storage system 300 can be expanded by adding a set of two compute nodes first to L1 cluster 102 and then successively to L1 clusters 103, 104, and 105 (e.g., as indicated by the arrow labeled "Growth Direction C"). Subsequently, another set of two compute nodes can be added successively to each of L1 clusters 101-105, and finally, a single compute node can be added successively to each of L1 clusters 101-105. Once all L1 clusters 101-105 are full, the disaggregated storage system 300 can be set up with its full capacity of compute nodes (see Fig. ). Fig. 5 - Example of a data structure

[0041] Fig. This shows an example Data Structure 500 in accordance with several implementations. Data Structure 500 can represent a hierarchy of multiple cluster levels contained in a disaggregated storage system (e.g., the one in Fig. (disaggregated storage system 300 shown). In some implementations, the data structure can comprise 500 N levels of cluster levels, where N is an integer greater than one. As in Fig. As depicted, the lowest level of the 500 data structure can, for example, contain multiple Level 1 (L1) clusters. Furthermore, the next higher level can contain multiple Level 2 (L2) clusters, with each L2 cluster comprising a group of multiple L1 clusters. This grouping can continue for each higher level of the 500 hierarchy, and the highest level, N, can contain a single Level N cluster (e.g., if N is an integer greater than two).

[0042] In some implementations, for each pair of adjacent levels, the cluster leaders in the lower level can form a cluster in the higher level. For example, L2 cluster 310 can be formed from the L1 leaders of L1 clusters 101, 102, 103, 104, and 105 (see Fig. Furthermore, for each level of data structure 500, the failure of one or more compute nodes in the above can be considered with reference to the Fig. be treated in the manner described. Fig. 6A-6B - Example computation node

[0043] Fig. This shows an example compute node 610 in accordance with some implementations. Compute node 610 can generally correspond to an example implementation of compute node 110 (as above, with reference to...). Fig. (described). As shown, the compute node 610 can include a controller 620, a memory 630, a memory 640, and a baseboard management controller (BMC) 650. The memory 640 can include one or more non-transferable storage media such as hard disk drives (HDDs), solid-state drives (SSDs), optical disks, etc., or a combination thereof. The memory 630 can be implemented by one or more storage devices, including volatile storage devices (e.g., random-access memory (RAM)), non-volatile storage devices (including persistent memory), or a combination thereof.

[0044] In some implementations, the BMC 650 can be a specialized controller embedded on an expansion card or on the mainboard of the host device 110. For example, the BMC 650 can support the Intelligent Platform Management Interface (IPMI) architecture, which defines a set of common interfaces to computer hardware and firmware that system administrators can use to monitor the health and manage a computer device. Furthermore, the BMC 650 can provide remote management access to the compute node 610 and enable such remote management access via an out-of-band communication channel that isolates management communication from the communication of the compute node 610's operating system. In some implementations, the BMC 650 can enable lights-out management of the compute node 610, which allows remote management access (e.g.,Access to the system console) on compute node 610 is enabled, regardless of whether compute node 610 is powered on, whether primary network subsystem hardware is functioning, or whether the operating system of compute node 610 is running.

[0045] In some implementations, BMC 650 can be used for recovery if compute node 600 becomes unresponsive while acting as the L1 leader of an L1 cluster. For example, suppose compute node 610 represents L1 leader 111A of L1 cluster 101 (shown in Fig. and therefore the representative network address for L1 cluster 101 can be assigned to it. Let us further assume that compute node 610 becomes unresponsive or "frozen" (e.g., due to an operating system crash) while acting as the L1 leader. Accordingly, the remaining compute nodes of L1 cluster 101 can perform an election to select a new L1 leader. However, it is assumed that the unresponsive compute node 610 retains ownership of the representative network address, and therefore the elected compute node cannot act as the new L1 leader. In some implementations, the elected compute node (or another compute node or entity) can instruct the BMC 650 to shut down or restart the unresponsive compute node 610, thereby releasing the representative network address from the unresponsive compute node 610.In this way, the BMC 650 of the non-responsive compute node 610 can be used to enable the selected compute node to assume the role of the L1 leader.

[0046] In Fig. A compute node 615 is represented according to some implementations. As in Fig. As shown, compute node 615 can use the same components as compute node 610 (in Fig. (shown) except that the BMC 650 is replaced by the Watchdog Timer 660. In some implementations, the Watchdog Timer 660 may be a circuit or software that generates or receives a periodic signal (e.g., every ten seconds) during the normal operation of the Compute Node 615. Furthermore, the Watchdog Timer 660 may measure the time between the periodic signals and determine whether the measured time exceeds a timeout threshold. In some implementations, the measured time may exceed the timeout threshold if the Compute Node 610 becomes unresponsive. Accordingly, if the Watchdog Timer 660 detects that the timeout threshold has been exceeded, it may cause the unresponsive Compute Node 610 to restart or reboot, thereby releasing the representative network address from the unresponsive Compute Node 610.In this way, the watchdog timer 660 of the unresponsive compute node 610 can be used to allow a selected compute node to take over L1 leadership. Fig. 7 - Example process for disaggregated storage

[0047] In Fig. This is an example Process 700 for disaggregated storage according to some implementations. Process 700 can be implemented in hardware or a combination of hardware and programming (e.g., machine-readable instructions that can be executed by one or more processors). The machine-readable instructions can be stored on a non-transitory, computer-readable medium, such as an optical, semiconductor, or magnetic storage device. The machine-readable instructions can be executed by a single processor, multiple processors, a single processing machine, multiple processing machines, etc. For illustration, details of Process 700 are given below with reference to the Fig. The examples described are consistent with some implementations. However, other implementations are also possible.

[0048] Block 710 can involve the initialization of a plurality of Level 1 (L1) clusters in a disaggregated storage system, each L1 cluster comprising a plurality of compute nodes. Block 720 can involve, for each L1 cluster of the plurality of L1 clusters, the selection of an L1 leader node from the plurality of nodes in the L1 cluster by the plurality of nodes in the L1 cluster. Block 730 can involve the formation of a Level 2 (L2) cluster containing the L1 leader nodes of the plurality of L1 clusters. Block 740 can involve the L1 leader nodes in the L2 cluster selecting an L2 leader node from among the L1 leader nodes in the L2 cluster. Fig. 8 - Example of a machine-readable data carrier

[0049] Fig. Figure 800 shows a machine-readable medium on which instructions 810-840 are stored in accordance with some implementations. Instructions 810-840 can be executed by a single processor, multiple processors, a single processing machine, multiple processing machines, etc. In some implementations, for example, instructions 810-840 can be executed by the controller 620 of compute node 610 (shown in Figure 800). Fig. or from the controller 620 of the compute node 615 (shown in Fig. The machine-readable medium 800 can be a non-transient storage medium, for example an optical, semiconductor or magnetic storage medium.

[0050] Instruction 810 can be executed to join a specific Level 1 (L1) cluster in a disaggregated storage system, where the disaggregated storage system comprises a plurality of L1 clusters. For example, compute node 111A, as in Fig. As shown, join L1 cluster 101 of disaggregated storage system 300. Disaggregated storage system 300 also includes other L1 clusters 102-105.

[0051] Instruction 820 can be executed to participate in an election for an L1 leader node of the respective L1 cluster. For example, compute nodes 111A-111G in L1 cluster 101 conduct an election, and compute node 111A is elected as the L1 leader node of L1 cluster 101 (see Fig. .

[0052] Instruction 830 can be executed to join a Level 2 (L2) cluster containing L1 leader nodes from the multitude of L1 clusters in response to being elected as the L1 leader node. For example, L1 leader node 111A is grouped with the L1 leader nodes of the other L1 clusters 102-105 to form L2 cluster 310 (see Fig. .

[0053] Instruction 840 can be executed to participate in an election for an L2 leader node of the L2 cluster. For example, the L1 leader nodes in L2 cluster 310 conduct an election, and L1 leader node 111A is elected as the L2 leader node of L2 cluster 310 (see Fig. . Fig. 9 - Example of a compute node

[0054] Fig. Figure 1 shows a schematic diagram of an example compute node 900. In some examples, compute node 900 may be a computing device that generally corresponds to one or more of the following elements: compute node 110 (shown in Figure 1). Fig. ), the compute node 610 (represented in Fig. and / or the compute node 615 (shown in Fig. As shown, the compute node 900 can contain a hardware processor 902 and a machine-readable memory 905 with instructions 910-940. The machine-readable memory 905 can be a non-transferable medium. The instructions 910-940 can be executed by the hardware processor 902 or by a processing machine contained within the hardware processor 902.

[0055] Instruction 910 can be executed to join a specific Level 1 (L1) cluster in a disaggregated storage system, where the disaggregated storage system comprises a plurality of L1 clusters. For example, compute node 111A, as in Fig. As shown, join L1 cluster 101 of disaggregated storage system 300. Disaggregated storage system 300 also includes other L1 clusters 102-105.

[0056] Instruction 920 can be executed to participate in the election of an L1 leader node of the respective L1 cluster. For example, compute nodes 111A-111G in L1 cluster 101 conduct an election, and compute node 111A is elected as the L1 leader node of L1 cluster 101 (see Fig. .

[0057] Instruction 930 can be executed to join a Level 2 (L2) cluster containing L1 leader nodes from the multitude of L1 clusters in response to being elected as the L1 leader node. For example, L1 leader node 111A is grouped with the L1 leader nodes of the other L1 clusters 102-105 to form L2 cluster 310 (see Fig. .

[0058] Instruction 940 can be executed to participate in an election for an L2 leader node of the L2 cluster. For example, the L1 leader nodes in L2 cluster 310 conduct an election, and L1 leader node 111A is elected as the L2 leader node of L2 cluster 310 (see Fig. .

[0059] In accordance with the implementations described here, a disaggregated storage system can include storage devices and a relatively large number of compute nodes (e.g., more than seven) arranged in two or more cluster levels. The compute nodes can be divided into multiple Level 1 (L1) clusters, with each L1 cluster containing multiple compute nodes. Within each L1 cluster, a leader can be elected by the compute nodes in that L1 cluster. Furthermore, the L1 leader nodes can be grouped into a Level 2 (L2) cluster, and then an L2 cluster leader can be elected (referred to here as the "L2 leader" or "L2 leader node"). The L1 leaders in the L2 cluster can be responsible for maintaining and updating multiple copies of the cluster management data. Furthermore, updating multiple copies of the cluster management data may involve reaching a consensus between the L1 leaders in the L2 cluster.In contrast, the follower nodes in the L1 clusters do not participate in consensus. Thus, consensus can be reached without exchanging a relatively large number of messages between all compute nodes in the L1 clusters. Furthermore, the follower nodes in the L1 clusters can be responsible for selecting the L1 leaders, including replacing L1 leaders that fail during operation. Similarly, the follower nodes in the L1 clusters can be used to compensate for the failure of L1 leaders in the L2 clusters. In this way, the hierarchical structure of the L1 and L2 clusters can improve the fault tolerance of the disaggregated storage system by incorporating a relatively large number of compute nodes without incurring the performance degradation associated with using a large number of compute nodes in a single cluster.

[0060] Note that the Fig. Various examples show that the implementations are not limited in this respect. Regarding Fig. For example, it is considered that the L1 cluster 100 and the compute nodes 110 could include additional devices and / or components, fewer components, different components, different arrangements, etc. Furthermore, the compute nodes 110 could be implemented as virtual devices and / or the storage devices 120 as virtual storage nodes. In another example, it is considered that the functionality of the cluster management software 130 described above could be integrated into any other software of the L1 cluster 100, into any controller or circuit of the L1 cluster 100, etc. Other combinations and / or variations are also possible.

[0061] Data and instructions are stored in appropriate storage devices, which are implemented as one or more computer-readable or machine-readable storage media. Storage media include various forms of non-transient memory, including semiconductor memory such as dynamic or static random-access memory (DRAM or SRAM), erasable and programmable read-only memory (EPROM), electrically erasable and programmable read-only memory (EEPROM), and flash memory; magnetic disks such as hard disks, floppy disks, and removable disks; other magnetic media including tapes; optical media such as compact discs (CDs) or digital video discs (DVDs); or other types of storage devices.

[0062] It should be noted that the instructions discussed above can be provided on a single computer-readable or machine-readable storage medium, or alternatively, on multiple computer-readable or machine-readable storage media distributed throughout a large system, possibly with multiple nodes. Such computer-readable or machine-readable storage medium or media are considered part of an article (or article of manufacture). An article or article of manufacture can refer to any single component manufactured or to multiple components. The storage medium or media can be located either in the machine on which the machine-readable instructions are executed or at a remote location from which machine-readable instructions can be downloaded for execution over a network.

[0063] The foregoing description includes numerous details to provide an understanding of the subject matter disclosed herein. However, implementations may be practiced without some of these details. Other implementations may include modifications and deviations from the details described above. It is intended that the accompanying claims cover such modifications and variations.

Claims

[1] Method (700) comprising the following: Initializing (710) a plurality of Level 1 (L1) clusters (101-105) in a disaggregated storage system (300), wherein each L1 cluster comprises a plurality of compute nodes (111A-111G, 112A-112G, 113A-113G, 114A-114G, 115A-155G); for each L1 cluster of the multitude of L1 clusters (101-105), selecting (720) an L1 leader node from the multitude of nodes in the L1 cluster by the multitude of nodes in the L1 cluster; Forming (730) a Level 2 (L2) cluster (150; 310) containing the L1 leader nodes of the multitude of L1 clusters (101-105); Selecting (740) an L2 leader node from the L1 leader nodes contained in the L2 cluster (150; 310) by the L1 leader nodes contained in the L2 cluster (150; 310); Writing data by the L1 leader nodes in the L2 cluster (150; 310) to at least one storage device of the disaggregated storage system (300); and Mounting a file system (140) by the L1 leader nodes in the L2 cluster (150; 310) for writing the data to the at least one storage device, where follower nodes in the L1 clusters lack the file system (140) for writing the data. [2] Method (700) according to claim 1, comprising: Receiving a request to write data to the at least one storage device by the L2 leader node; The L2 leader node captures the request in a log entry; Transmission of the protocol entry by the L2 leader node to other L1 leader nodes in the L2 cluster (150; 310); Determine, via the L2 leader node, whether at least a minimum number of the other L1 leader nodes of the L2 cluster (150; 310) have confirmed the log entry; and In response to a determination that at least the minimum number of other L1 leader nodes of the L2 cluster (150; 310) has acknowledged the log entry, execute the request to write data to the at least one storage device. [3] Method (700) according to claim 1, wherein the disaggregated storage system (300) comprises a hierarchy of N cluster levels, and wherein N is an integer greater than two. [4] Method (700) according to claim 1, comprising: Determine that the L2 leader node has failed, wherein the L2 leader node is contained in a first L1 cluster and wherein the L2 cluster (150; 310) contains one or more L1 leader nodes that remain operational; and in response to the determination that the L2 leader node has failed: Selecting a new L2 leader node by the one or more L1 leader nodes that remain operational; After selecting the new L2 leader node, select a new L1 leader node for the first L1 cluster, with the new L1 leader node being included in the L2 cluster (150; 310). [5] Method (700) according to claim 1, comprising: Determine whether the L2 cluster (150; 310) has a quorum of L1 leader nodes; and in response to the finding that the L2 cluster (150; 310) does not have the quorum of L1 leader nodes: restoring the quorum of L1 leaders in the L2 cluster (150; 310); Selection of a new L2 leader node by the quorum of L1 leaders in the L2 cluster (150; 310). [6] Method (700) according to claim 1, comprising: Determine that a specific L1 leader node for a specific L1 cluster has failed, while the L2 leader node remains operational; and in response to the determination that a specific L1 leader node has failed: Selecting a new L1 leader node for the specified L1 cluster from the available compute nodes of the specified L1 cluster, with the new L1 leader node being added to the L2 cluster (150; 310). [7] Non-transitory, machine-readable medium (800) that stores instructions which, when executed, cause a processor to: to join a specific Level 1 (L1) cluster in a disaggregated storage system (300) (810), wherein the disaggregated storage system (300) comprises a plurality of L1 clusters (101-105); to participate in the selection of an L1 leader node of the specified L1 cluster (820); in response to being chosen as an L1 leader node, to join a Level 2 (L2) cluster (150; 310) (830) which includes L1 leader nodes from the multitude of L1 clusters (101-105); to participate in the selection of an L2 leader node of the L2 cluster (150; 310) (840); and after joining the L2 cluster (150; 310): to write data to at least one storage device of the disaggregated storage system (300); and to mount a file system (140) to write the data to the at least one storage device, where follower nodes in the L1 clusters lack the file system (140). [8] Non-transitory machine-readable medium according to claim 7, comprising instructions which, when executed, cause the processor to: After being selected as an L2 leader node, a request to write data to at least one storage device must be received; to record the request in a log entry; to transmit the protocol entry to other L1 leader nodes in the L2 cluster (150; 310); in response to a determination that at least a minimum number of the other L1 leader nodes of the L2 cluster (150; 310) have acknowledged the log entry, to initiate the execution of the request to write the data to the at least one storage device. [9] Non-transitory machine-readable medium according to claim 7, comprising instructions which, when executed, cause the processor to: in response to a determination that the L2 leader node has failed, to participate in a selection process for a new L2 leader node. [10] Non-transitory machine-readable medium according to claim 7, wherein the disaggregated storage system (300) comprises a hierarchy of N cluster levels, and wherein N is an integer greater than two. [11] Computing nodes (110, 111A; 610; 615; 900) comprising: a controller (620); and a machine-readable memory (905) that stores instructions, the instructions of which can be executed by the controller (620) to: to join a specific Level 1 (L1) cluster from a variety of L1 clusters (101-105) (910); to participate in the selection of an L1 leader node of the specified L1 cluster (920); in response to being selected as an L1 leader node, to join a Level 2 (L2) cluster (150; 310) (930) which includes L1 leader nodes from the multitude of L1 clusters (101-105); to participate in the selection of an L2 leader node of the L2 cluster (150; 310) (940); and after joining the L2 cluster (150; 310): to write data to at least one storage device assigned to the L2 cluster (150; 310); and to mount a file system (140) to write the data to the at least one storage device, where follower nodes in the L1 clusters lack the file system (140). [12] Computing node (110, 111A; 610; 615; 900) according to claim 11, comprising instructions that can be executed by the controller (620) to: After being selected as an L2 leader node, a request to write data to at least one storage device must be received; to record the request in a log entry; to transmit the protocol entry to other L1 leader nodes in the L2 cluster (150; 310); in response to a determination that at least a minimum number of the other L1 leader nodes of the L2 cluster (150; 310) have acknowledged the log entry, to initiate the execution of the request to write the data to the at least one storage device. [13] Computing node (110, 111A; 610; 615; 900) according to claim 11, comprising instructions that can be executed by the controller (620) to: in response to a determination that the L2 leader node has failed, to participate in a selection process for a new L2 leader node. [14] Computing node (110, 111A; 610; 615; 900) according to claim 11, wherein the disaggregated storage system (300) comprises a hierarchy of N cluster levels, and wherein N is an integer greater than two.

Images