Computer-implemented method for preselecting writers in a distributed file system and network of servers

Abstract

A computer-implemented method may include receiving proposals to change data stored in a distributed replicated file system coupled with a network, the distributed replicated data system including a plurality of nodes, each including a server. A metadata service maintains and updates a replica of a namespace in the distributed replicated data system and coordinates updates to the data by generating an ordered set of agreements corresponding to the received proposals, the ordered set of agreements specifying an order in which nodes will change the data stored in the data nodes and cause corresponding changes to the state of the namespace. For each agreement in the generated ordered set of agreements, a corresponding writer list is provided, the writer list including an ordered list of nodes to execute the agreement and make corresponding changes to the namespace. The ordered set of agreements may then be transmitted, for each agreement in the ordered set of agreements, along with a corresponding writer list or a pre-generated index thereto, to a plurality of nodes, each of which may be configured to execute only those agreements for which the node is the first listed node on the received writer list.

Description

Technology background

[0001] The field of embodiments disclosed herein includes distributed replicated data systems. Some distributed data systems may define logical constructs known as zones. Each such zone may include a server tasked with executing commands received by writing to and reading from that zone's metadata service (hereinafter, MDS). Many systems provide such a metadata service through a single server. Read commands may optionally be configured to bypass that server for applications where read consistency may be sacrificed to improve performance. When multiple servers exist in each zone for failover purposes, a choice must be made to ensure that only one of the multiple servers in each zone is designated to execute a given command at any given time, although the designated server may vary depending on the command. In such cases, logic is provided to select a new server to which writes are permitted when a currently writable server is deemed to have crashed. However, accurately distinguishing between a crashed server and a slow server may be difficult. In fact, when periodic heartbeat signals indicating the continued normal operation of a server are not received within the expected time frame, the server may actually have crashed, may simply have slowed down, or communication with the server may be delayed or impossible due to temporary network latency issues or other reasons. In such a case, if a new server is programmatically made writable and the old server is revived for some reason, there may now be two servers that are enabled to execute the same command by writing to the back-end storage. As a result, data integrity cannot be guaranteed as the two servers will now independently execute the same command, potentially resulting in data corruption.

[0002] Thus, some distributed systems require a human to manually verify the death or non-operation of a first server before allowing a second server to write to the backend data store. This ensures that a server thought to be dead is in fact dead, thereby preventing two active writers in the same zone at the same time. Such human verification is suboptimal because no data can be written until a human manually verifies (and thereby deactivates) the currently writable server. [Brief explanation of the drawings]

[0003] [Figure 1] 1 is a diagram of a distributed replicated data system in which embodiments may be implemented.

[0004] [Figure 2] 1 is a diagram illustrating a contributor list and aspects of the computer-implemented method, according to an embodiment.

[0005] [Figure 3] 1 is a diagram illustrating a contributor list and aspects of the computer-implemented method in the presence of a failed (or suspected failed) node, according to an embodiment.

[0006] [Figure 4] 1 is a flowchart illustrating aspects of a computer-implemented method according to one embodiment.

[0007] [Figure 5] 1 is a flowchart illustrating aspects of a computer-implemented method according to one embodiment.

[0008] [Figure 6] 1 is a diagram illustrating a distributed state machine spanning a single proposer and multiple learners, according to one embodiment.

[0009] [Figure 7]1 is a diagram illustrating a pre-generated, indexed author list according to one embodiment.

[0010] [Figure 8] 1 is a diagram illustrating a pre-generated, indexed author list according to one embodiment.

[0011] [Figure 9] 1 is a flowchart of a computer-implemented method according to one embodiment.

[0012] [Figure 10] FIG. 1 is a block diagram of a computing device in which the embodiments shown and described herein may be implemented. DETAILED DESCRIPTION OF THE INVENTION

[0013] Distributed System: A distributed system comprises a collection of distinct computing and / or storage processes and / or devices that may be spatially separated and that may communicate with each other through the exchange of messages or events.

[0014] Replicated State Machines: The replicated state machine approach is a way of implementing fault-tolerant services by replicating servers and coordinating client interactions with the server replicas. These state machines are "replicated" because the state of the state machine progresses in exactly the same ordered fashion on all learners. Replicas of a single server run on separate processors in a distributed system, and a protocol is used to coordinate client interactions with these replicas. One example and implementation of a replicated state machine is a deterministic state machine (DSM), which advances its state deterministically.

[0015] Suggestor: According to one embodiment, a proposer is a process configured and enabled to propose suggestions, some of which may be configured to change data.

[0016] Acceptor: According to one embodiment, an acceptor is a process configured to participate in determining the order of proposals made by proposers. According to one embodiment, an agreement (e.g., an agreed-upon proposal) occurs only when a majority of acceptors determine that the proposals are made in, among other things, the global sequence of agreement (described further below). According to one embodiment, an acceptor may be configured to only participate in determining the order of agreement and not reason / be interested in the underlying content of the agreement (as described herein, the value of the agreement is opaque to the Distributed Coordination Engine, also described below). An acceptor may be configured as an application-independent entity.

[0017] Learner: According to one embodiment, the learner learns the agreements made between proposers and acceptors and applies the agreements to applications through their output proposal sequences in a deterministic order. In one embodiment, an agreement identity is provided for each replicated state machine, as is a persistence store that allows the sequence of agreements to be durably recorded. Each proposal is guaranteed to be delivered at least once to each learner in a particular membership.

[0018] The Hadoop Compatible File System (HCFS) namespace is a hierarchical structure of files and directories. Hadoop is an open-source, Java-based programming framework that supports the processing and storage of extremely large datasets in a distributed computing environment. It is part of the Apache project, provided by the Apache Software Foundation. Files and directories are represented on the NameNode by an inode. The inode records attributes or metadata such as permissions, modification and access times, namespace, and disk space allocation. File contents are divided into large data blocks (typically 128 MB), and each data block of a file is independently replicated on multiple DataNodes (typically three). One implementation of HCFS is the Hadoop Distributed File System (HDFS). The NameNode is the HDFS metadata service responsible for tracking changes in the namespace. The NameNode maintains the namespace tree and the mapping of blocks to DataNodes. That is, the NameNode tracks the location of data within a Hadoop cluster and coordinates client access to that location. Traditionally, each cluster has a single NameNode. A cluster can have thousands of DataNodes and tens of thousands of HDFS clients per cluster, with each DataNode running multiple application tasks simultaneously. The list of inodes and data blocks that define the metadata for the name system is called an image. The NameNode maintains the entire namespace image in random access memory (RAM).

[0019] The roles of proposer (the process that makes a proposal to change the state of a namespace to the membership), acceptor (the process that votes on whether a proposal to change the state of a namespace should be agreed to by the membership), and learner (the process that learns the agreement made by the membership) are defined, for example, in the implementation of the Paxos algorithm described in Lamport, L.: The Part-Time Parliament, ACM Transactions on Computer Systems 16, 2 (May 1998), 133-169, which is incorporated herein in its entirety. According to one embodiment, multiple nodes may be configured to perform each of these roles. A distributed coordination engine (also referred to as DConE) may enable multiple learners to agree on the order of events sent to the engine by multiple proposers, leveraging multiple acceptors to achieve high availability. To achieve reliability, availability, and scalability, multiple simultaneously active NameNodes (sometimes referred to herein collectively as MDSs) may be provided by replicating the state of the namespace on multiple nodes, with the requirement that the state of the nodes on which the namespace is replicated be consistent across such nodes.

[0020] This consistency between NameNodes in different zones may be guaranteed by DConE, a reconciliation engine that may be configured to accept proposals to update the namespace, rationalize the proposals into a global sequence of ordered updates, and only then allow MDSs to learn and apply updates to their respective states in a particular, agreed-upon order. As used herein, "consistency" means one-copy equivalence as detailed in "Concurrency Control & Recovery in Database Systems" by Bernstein et al., published by Addison-Wesley, 1987, Chapters 6, 7, and 8, which is incorporated herein in its entirety. Because NameNodes start from the same state and apply the same deterministic updates in the same deterministic order, their respective states evolve in the same way over time, maintaining consistency.

[0021] Thus, according to one embodiment, the namespace is: a) each MDS is permitted to modify its namespace replica, and b) Updates to one namespace replica must be propagated to namespace replicas in other MDSs in other zones so that namespace replicas remain consistent with each other, across MDSs, and across zones. It may be replicated across multiple NameNodes (or, more generally, Metadata Servers or MDSs), subject to:

[0022] FIG. 1 illustrates a cluster operating a single distributed file system 102 across different geographically (or otherwise) separated zones. The distributed file system may incorporate aspects of HDFS, for example. Each of the DataNodes (denoted as “DNs” in FIG. 1 ) may be configured to communicate only within their own zone (through DataNode-to-server remote procedure call (RPC) protocol). That is, DNs in Zone 1 may communicate only with nodes (servers) 110, 112, 114, ... (or neighboring nodes in Zone 1), and DNs 132, 134, 136, 138, ... (or neighboring servers in Zone 2) may communicate only with nodes (servers) 116, 118, 120, ... (or neighboring servers in Zone 2). In one embodiment, only one Metadata Service (MDS) storing a replica of the namespace may exist in each zone, as shown by MDS 103 for Zone 1 and MDS 105 for Zone 2. Nodes (servers) in each zone, in turn, communicate only with their respective zone's MDS. Thus, nodes 110, 112, 114, etc., communicate with MDS 103, and nodes 116, 118, 120, etc., in zone 2, communicate only with MDS 105. The MDSs in both zones 1 and 2 may coordinate with each other to maintain consistency of the namespace state throughout the different zones of the distributed file system 102 by using one or more (e.g., an odd number, such as three for high availability (HA)) inter-zone servers 140, 142 to stream changes to the namespace across the WAN 108 between the zones. Such changes may be received by servers (nodes) in other zones, which then write the changes locally to their zone's back-end storage, thereby enabling all reads and writes to be performed as local operations, thereby eliminating the need for cross-zone security.

[0023] DConE process 122 may be configured to ensure that the same deterministic updates to the namespace state are applied in the same deterministic order on all MDSs across all zones. In one embodiment, DConE process 122 may be embedded in the MDS of each zone. That deterministic order may be defined by a global sequence number (GSN). Thus, the primary role of DConE process 122 is to process agreed-upon proposals to modify or otherwise update the state of namespace replicas according to commands received by the servers in each zone from HDFS clients, and to convert them into a globally ordered sequence of agreements indexed by the GSN. Servers (or nodes, these terms may be used interchangeably) may then sequentially apply agreements from that ordered sequence, generating updates to the state of namespace replicas in their zones. The GSN may be configured as a unique, monotonically increasing number. However, the GSN may be configured otherwise as would be recognized by one skilled in the art. In this way, through sequential execution of the ordered set of agreements (through the GSN mechanism) generated by DConE process 122, and through the flow of changes in each zone to all other zones, the state of the namespace replicas stored in each zone is made or maintained in a consistent state. Because the MDSs start from the same state, the application of these ordered updates ensures consistency of replicas across zones in that the snapshots on MDSs that processed agreements with the same GSN will be the same both within and across zones.

[0024] Once DConE process 122 delivers the agreement, and changes are flushed between zones, metadata in the namespace replicas maintained by the MDS may be adjusted instantly (or nearly instantly, considering the bandwidth and latency inherent in the network). Similarly, all file system data is also automatically replicated across the distributed file system. In this manner, consistent, continuous data replication occurs between file systems in a cluster (e.g., without limitation, Hadoop). Client applications may be configured to interact with a virtual file system that aggregates the underlying storage across multiple zones. When changes are made to files in one zone, those changes are consistently replicated to other zones. One embodiment may include a software application that enables Hadoop deployments to replicate HCFS data between (e.g., Hadoop) clusters running different, even incompatible, versions of Hadoop, such as CDH, HDP, EMC Isilon, Amazon S3 / EMRFS, and MapR. According to one implementation, replication between different vendor distributions and versions of Hadoop is also possible.

[0025] Advantageously, embodiments may provide a virtual file system for Hadoop that is compatible with all Hadoop applications, a single virtual namespace that consolidates storage from various types of Hadoop, a globally distributed storage mechanism, and WAN replication that uses active / active replication techniques to deliver replicated, single-copy consistent HDFS data across data centers across a wide area.

[0026] According to one embodiment, some or all of the functionality described herein may be implemented higher in the distributed file system stack, within one or more servers adjacent to the MDS. In this way, rather than running deep at the NameNode level, one embodiment may be configured to act as a proxy application for the distributed file system.

[0027] In an ordered global sequence of agreed-upon proposals (issued from a deterministic state machine (DSM) or from some other source), some commands in the sequence of commands may be dependent on other commands. Dependent commands must be executed in the correct order. For example, consider commands A, B, and C. Command B depends on command A, and command C depends on commands B and A. It may be desirable to distribute the execution load of such commands across multiple servers or nodes within a zone for load balancing or other purposes. For example, node 1 may be assigned to execute command A, node 2 may be assigned to execute command B, and node 1 may be assigned to execute command C. In order for node 2 to execute command B, it must be made aware that node 1 has finished executing command A. Similarly, the same is true for node 1 when it executes command C, since it needs confirmation that node 2 has finished executing command B. This approach can result in significant delays in processing client change requests (i.e., commands) when delays occur in inter-node communication.

[0028] As noted above, execution of a dependent command must be delayed until the command or commands it depends on have been executed. Non-dependent commands may be executed in parallel. In the event of a failure, a command may be executed again, but under no circumstances may a sequence of commands be executed again. That is, while each command is idempotent (executed once or multiple times will produce the same result), execution of a sequence of two or more commands is not idempotent in that re-execution of the same sequence will not produce the same result. Thus, while each command may be executed individually more than once, such a sequence of commands cannot.

[0029] For scalability and high availability, there may be multiple servers executing commands, and each server in a zone may execute multiple independent commands in parallel. Although each node receives the entire ordered global sequence of commands, each command in the global sequence should only be executed by a single node. Nodes or servers are considered herein to be unreliable (in that they are prone to failure and may not necessarily be recoverable) and are configured to communicate with each other as described above.

[0030] One embodiment is a computer-implemented method for pre-selecting writers (i.e., pre-selecting nodes (in one embodiment, preferred) on which to execute commands configured to update the state of a namespace), which achieves scalability, high availability, and ensures fast and reliable execution of an ordered global sequence of configuration commands, while maintaining safety with respect to preventing re-execution of a sequence of commands. List of contributors

[0031] In one embodiment, before insertion into the ordered global sequence generated by DConE 122, as shown in FIG. 2, each command or command may be associated with a list of nodes (servers) to execute the ordered command (or, in the case of consensus, agreement) according to execution priority. In one embodiment, this list of nodes may be ordered so that the nodes appear on the list in order of execution priority. This ordered list of nodes is also referred to herein as writer list 202. In one embodiment, each node may be configured to execute only commands for which it is the first node in the ordered writer list that it receives. In one embodiment, if the first node in the writer list becomes inactive (e.g., an expected heartbeat signal is not received in a timely manner), the next operational node or server in the ordered writer list 202 may be designated as the node to execute the command. Then, in real time or near real time, the node broadcasts information to all or a predetermined number of nodes that allows all nodes (including itself) to be informed of all commands that have already been executed, including which commands it has executed itself, in time.

[0032] As shown in FIG. 2 , command 1 is an independent command and is associated with a writer list 202 in which server 112 is the first listed node. Therefore, node 112 is the preferred server for executing command 1. If node 112 fails for any reason before executing command 1, or after executing command 1 but before it can signal that it has executed command 1, execution of command 1 falls back to node 110 (the next listed node in writer list 202), and then, in the unlikely event that server 110 fails, to the next listed node 114. Any known failed node is demoted to the bottom of the writer list created (or accessed) for subsequent commands. Because command 2 depends on the execution of command 1, a writer list 204 is associated with command 1, and this writer list is the same as writer list 202. Command 3 is an independent command and may be associated with a different writer list 206. Note that while the same writer list as 202 and 204 may have been associated with command 3, load balancing and / or other considerations may have recommended a different writer list for that command.

[0033] More specifically, according to one embodiment, the process of placing commands in an agreed-upon ordered global sequence may learn about suspected and failed nodes and may track the operational state of all nodes. For each new execution of a client command, the writer list, according to one embodiment, may include an ordered sequence of preferred operational nodes on which to execute the command, followed in one embodiment by nodes that have failed, are failing, or are suspected of failing. Thus, when a node has failed or is suspected of failing, such a node will be relegated to or toward the bottom of the writer list.

[0034] In one embodiment, operational nodes in the writer list may be ordered so that dependent commands preferably receive the same writer list (as shown at 202 and 204 in FIG. 2 ) to reduce inter-node execution dependencies, and so that all nodes have approximately the same rank distribution. In fact, dependent commands may preferably receive the same writer list to reduce inter-node execution dependencies and, therefore, latency. This is because the node selected to execute a non-dependent command is the node best positioned to execute the command(s) that depend on it, thereby avoiding the latency inherent in waiting for updates to be transmitted to other nodes that execute the dependent command(s). According to one embodiment, all nodes may have approximately the same rank distribution in the generated writer list to evenly distribute the computational load across all available (i.e., operational) servers or nodes. Other considerations may, in some circumstances, dictate a different ordering of nodes in the writer list that modifies the listed priorities. Node failure

[0035] Servers are considered prone to failure. Node failure means that the node will stop executing commands, stop reporting its execution status to other nodes, or both. Other nodes have no way to distinguish between these two events (the node is delayed, not reporting, or has failed). In one embodiment, when a node fails, the node's ranking in the writer list changes, at least for all new or subsequent commands. Indeed, in one embodiment, a recognized node failure results in the node being pushed back in the writer list (i.e., away from the top and toward the bottom) for new or subsequent commands, and eventually the system will stop generating computational load for the recognized failed node until the server recovers and signals that it has returned to normal operation. In FIG. 3, node 112 has failed or is suspected of having failed. When a new command is issued, node 112 will be demoted to the bottom or near the bottom of writer list 302 associated with the new command.

[0036] Commands that were previously assigned to be executed by the failed node as the preferred writer, but have not yet been executed, will become stuck (along with all commands that depend on them) from the perspective of other nodes. According to one embodiment, once an external entity has reliably verified that the suspected failed node is inoperable and will not suddenly resume operation, these stuck commands may be handled in one of two ways, according to one embodiment: 1) Declaration of dead node

[0037] Upon failure of node X, as shown at 402 in Figure 4, all other nodes may be notified (in one embodiment, by the DConE process embedded in the MDS) of the failure of node X, as shown at 404. Upon receiving such notification, the node, according to one embodiment, may remove node X from its writer list, as shown at 406, from all pending commands that had node X as the first preferred node for execution purposes, as shown in Figure 4, at which point, because node X has been removed from the top of the writer list, the commands will be executed by another (in one embodiment, the next) node on the writer list, as shown at 408. 2) Exchange node

[0038] Similarly, as shown in FIG. 5, upon failure of node X as shown at 402, all other nodes may be notified of node X's failure (in one embodiment, by a DConE process embedded in the MDS of the zone containing the failed node X), as shown at 502. Upon receiving such notification, the nodes, according to one embodiment, may remove node X from their writer list from all pending commands that had node X as their first-priority node for execution, as shown at 406. According to one embodiment, a single other replacement node may be designated to take over commands previously assigned to the failed node X for execution. This newly designated replacement node may be a new node (server), in which case it may query other nodes, as shown at 504, to determine which commands have been assigned to node X for execution, so that the new node may execute those commands in place of the failed node X, as shown at 506. Execution status transmission

[0039] Each node broadcasts information about the commands it has executed for two reasons: (1) to unblock dependent commands assigned to other MDSs, and (2) To prevent re-execution of non-idempotent commands in the event of a node failure. Note that in one embodiment, if a command is idempotent, there is no need to prevent the command from being immediately re-executed unless there are intervening dependent commands. In particular, embodiments do not need to protect against server failure after executing a command, except before disseminating its information. Instead, according to one embodiment, non-idempotent commands are corrected to make their results equivalent, rather than being made idempotent.

[0040] The precise manner in which such transmission occurs need not be enumerated. However such transmission occurs, it should meet the following constraints or characteristics: To satisfy reason (1), the transmission should occur quickly, i.e., in near real time. The act of broadcasting information about executed commands should be persistent and guaranteed to be delivered. This way, nodes that fail and eventually return to a working state can and will eventually learn the broadcast information. The act of broadcasting information about an executed command should be operational as long as the broadcasting node (the node that executed the command about which information is being broadcast) is operational. Each node should be able to acknowledge the delivery of such outgoing information to other nodes to ensure continuity.

[0041] One embodiment involves a deterministic finite state machine with a single proposer, a single acceptor (the writing node itself), and learners (other nodes) that learn the results of the command. In such an embodiment, each executer server (node) has its own deterministic finite state machine. As a result, such a finite state machine may serve as the mechanism by which inter-node communication can occur, as described below. Continuity Guarantee

[0042] If a node fails, it must not lose any command execution state, except as permitted by idempotence. Suppose command C is executed. Before this information has been fully broadcast, command D, which depends on C, is executed. At this point, even if C is idempotent, re-executing C would violate safety, but this could occur due to insufficient broadcasting. For example, a node may fail after executing a sequence of commands but before successfully broadcasting its "command completed" updated state. Another node may then mistakenly re-execute the sequence of commands, which is a common violation of safety. Thus, in one embodiment, to tolerate the failure of N nodes, a node must, according to one embodiment, verify successful broadcasts to at least N+1 nodes (including itself) before executing any commands that are later in the sequence and depend on the results of that command. Different policies and risk tolerances will dictate the number of nodes whose broadcasts must be verified.

[0043] Participant roles in the Paxos consensus algorithm include proposer (the process that proposes a command that changes the state of a namespace to the membership), acceptor (the process that votes on whether a proposed command that changes the state of a namespace should be agreed upon by the membership), and learner (the process that learns the agreement reached by the membership). According to one embodiment, a node may be configured to perform any one of these roles at any given time. As described above, the DConE process may enable multiple learners to agree on the ordering of events sent to the engine by multiple proposers, leveraging multiple acceptors to achieve high availability.

[0044] Thus, under Paxos, when there are multiple acceptors, a command proposed by a proposer will proceed unless most (i.e., the majority) of the multiple acceptors fail. However, according to one embodiment, in the special case where there is only one acceptor node, the proposed command may proceed (i.e., take its place in the ordered sequence of executed agreements) without consulting that single acceptor node. The proposed command may thus proceed directly from the proposer of the deterministic state machine to agreement with the learner, bypassing the (single) acceptor. Paxos-style consensus is therefore not required in this case, since the proposer "knows" that it is not contradicting itself.

[0045] Information about every executed command must be disseminated to all learners. If there is only one proposer node, that node executes the command and necessarily learns information about the command's changed (pending to executed) state. Thus, this single proposer effectively serves as a persistent conduit (to itself) for disseminating information about executed commands. In one embodiment, a persistent conduit for disseminating information about executed commands may also be established for all learner nodes. According to one embodiment, the proposer may have a deterministic state machine associated with it (for which it is the only proposer) that enables all learners to learn information about executed commands. Each other node may also have a deterministic state machine (active when it is the sole proposer) that provides such information to all other nodes when they act as learners and enables them to change their state accordingly.

[0046] In one embodiment, as shown in FIG. 6 , each node may therefore have a deterministic state machine (DSM) for which it is the sole proposer and all other nodes are learners. As shown, node 602 has DSM 601, in which node 602 is the sole proposer and all other nodes 604, 606, 608, 610, etc. are learners. Similarly, node 604 has DSM 603, in which node 604 is the sole proposer and all other nodes 602, 606, 608, 610, etc. are learners. Similarly, node 606 has DSM 605, in which node 606 is the sole proposer and all other nodes 602, 604, 608, 610, etc. are learners, and so on. In this case, DSMs 601, 603, 605, etc. act as pre-established, persistent message queues, thereby significantly reducing inter-node traffic and reducing the latency inherent in point-to-point communication between nodes. Optimizing an arbitrary author list

[0047] As described above, for each new command in the client commands, the writer list, according to one embodiment, may include a prioritized sequence of operational nodes that should execute the command, followed, in one embodiment, by nodes that have failed, are failing, or are suspected of failing. In this manner, the preferred operational node would be at the top of the writer list, followed by an ordered list of fallback operational nodes that should execute the command if the node at the top of the list is found to be inoperative, and finally, known currently inoperative nodes at the end of the list. Thus, when a node fails or is suspected of failing, such a node would be relegated to the bottom of the writer list. According to one embodiment, if the list of available nodes is known in advance to all participants, it becomes possible to generate separate and unique writer lists for each possible permutation of nodes. For example, if there are n nodes, n! different writer lists can be pre-generated and indexed. These pre-generated and indexed writer lists may be pre-distributed to each node (i.e., before processing the command) as a list of writer lists, as shown at 702 in Figure 7. In the example shown in Figure 7, there are five available, or possibly available, nodes to which the command may be assigned for priority execution. As shown in Figure 7, there are five available nodes, and there are 120 different orderings of these five nodes, and each of these unique orderings of nodes may construct a writer list, with each writer list identified by an index k=1 to k=120. Even with a large number of nodes, such a simple writer list would not take up much storage space at each node, and transmission of such a writer list would not be prohibitive in terms of bandwidth—especially since such transmission would only need to be performed once.

[0048] Then, instead of sending a writer list with each command to be executed detailing the preferred node (the node listed first) for executing that command, followed by fallback nodes, and finally non-operational, failed, or suspected failed nodes, a pre-generated and pre-distributed list 702 of simple k-based writer lists may accompany the command to be executed. Sending the k-based list 702 rather than the full k-based writer list significantly reduces traffic overhead and provides a ready mechanism for designating a preferred node or server for executing any command. Upon receiving index k, only the node at the top of the k-based writer list corresponding to the received index will execute the command. The selection of the k-based list of writer lists, according to one embodiment, may take into account many factors, including, for example, load balancing, command dependencies, geographic proximity, network conditions and network latency, as well as knowledge of which nodes are currently non-operational, failed, or suspected to be failed. Other factors may also be considered when selecting index k. When the index k for the command to be executed is sent, the selected index k will correspond to a list in which at least non-operational, failed or suspected failed nodes appear lowest.

[0049] As shown in FIG. 8, therefore, the DConE process 122 (which may be embedded in the MDS of each zone) may broadcast to all nodes, as shown at 802, the GSN and the command to be executed, along with the index k of the writer list, where the index k specifies a unique writer list, which in turn specifies the preferred node (and fallback, lower list node) on which to execute the command.

[0050] 9 is a flowchart of a computer-implemented method according to one embodiment. As shown therein, block B902 may request to receive proposals to change data stored in a distributed replicated file system coupled to a network, the distributed replicated data system including a plurality of servers (also referred to herein as nodes) and a metadata service configured to maintain and update replicas of a namespace in the distributed replicated file system. Block B904 may then be executed, in which updates to the data may be coordinated by generating an ordered set of agreements corresponding to the received proposals, the ordered set of agreements specifying the order in which nodes may change data stored in the data nodes and cause corresponding changes to the state of the namespace. Each node may be configured to delay making changes to the data and causing changes to the state of the namespace until the ordered set of agreements is received. As shown in B906, for each agreement in the generated ordered set of agreements, a corresponding writer list including an ordered list of nodes may be provided or identified to implement the agreement and cause corresponding changes to the namespace. As shown in B908, the set of ordered agreements may then be sent to a plurality of nodes, along with, for each agreement in the set of ordered agreements, a corresponding author list or a pre-generated index thereto, each of which may, according to one embodiment, be configured to execute only those agreements for which it is the first listed node on the author list.

[0051] According to further embodiments, providing may include generating contributor lists for at least some of the generated ordered set of agreements. Providing may also include selecting from among multiple pre-generated contributor lists. The contributor lists may include an ordered list of preferred operating nodes toward the top of the contributor list and a list of failed or suspected failed nodes toward the bottom of the contributor list. The act of providing may further include providing or identifying the same contributor list for a second proposal that depends on execution of the previous first proposal, in addition to providing it for the first proposal. The computer-implemented method may further include enabling a next-listed node in the contributor list to execute the agreement when a first-listed node in the contributor list fails or is suspected to have failed. A given replacement node may also be enabled to execute the agreement when a first-listed node in the contributor list fails. Each node or server that executed the agreement may be further configured to broadcast information regarding the executed agreement to each of the multiple nodes. Broadcasting may further include ensuring delivery of the broadcasted information. As a node executes an agreement, a deterministic state machine may be updated with information about the executed agreement, and the deterministic state machine may be coupled to each of the other nodes and serve as a persistent messaging service among the multiple nodes. The computer-implemented method may further include learning of a failing or failed node and placing the failed or failing node at the bottom of all generated writer lists. In one embodiment, an indexed writer list may be pre-generated for each of all possible combinations of orderings of the multiple nodes and distributed to each of the multiple nodes. The method may then include selecting one of the indexed writer lists and sending an ordered set of agreements to the multiple nodes, along with, for each agreement in the ordered set of agreements, an index to the selected one of the pre-generated indexed writer lists.

[0052] Another embodiment is a network of nodes configured to implement a distributed file system. The cluster may include a plurality of data nodes, each configured to store data blocks of client files, a plurality of servers or nodes, each configured to read and / or mutate data stored in the data nodes and, in response to changes to the data blocks of the client files, to cause corresponding updates to the cluster's namespace state, and a distributed coordination engine embedded in a metadata service, the distributed coordination engine configured to reconcile received proposals to mutate data blocks by generating an ordered set of agreements corresponding to the received proposals, the ordered set of agreements specifying the order in which nodes will make changes to the data stored in the data nodes and cause corresponding changes to the namespace state. The metadata service may be further configured to: provide, for each agreement in the generated ordered set of agreements, a corresponding writer list including an ordered list of nodes to execute the agreement and cause corresponding changes to the namespace, and to transmit the ordered set of agreements to the plurality of nodes, along with the corresponding writer list or pre-generated index for each agreement in the ordered set of agreements. In this way, each of a plurality of nodes may only allow agreements for which it is the first listed node in the contributor list to be enforceable. Physical Hardware

[0053] FIG. 10 shows a block diagram of a computing device in which embodiments may be implemented. The computing device of FIG. 10 may include a bus 1001 or other communication mechanism for communicating information and one or more processors 1002 coupled with the bus 1001 for processing information. The computing device may further include a random access memory (RAM) or other dynamic storage device 1004 (referred to as main memory) coupled to the bus 1001 for storing information and instructions to be executed by the processor(s) 1002. The main memory (tangible and non-transitory; this term excludes signals per se or waveforms herein) 1004 may also be used to store temporary variables or other intermediate information during execution of instructions by the processor(s) 1002. The computing device of FIG. 10 may also include a read-only memory (ROM) and / or other static storage device 1006 coupled to the bus 1001 for storing static information and instructions for the processor(s) 1002. A data storage device 1007, such as a magnetic disk and / or solid-state data storage device, may be coupled to bus 1001 for storing information and instructions, such as may be required to perform the functions shown and disclosed in connection with FIGS. 1-9 . The computing device may also be coupled to a display device 1021 via bus 1001 for displaying information to a computer user. An alphanumeric input device 1022, including alphanumeric and other keys, may be coupled to bus 1001 for communicating information and command selections to processor(s) 1002. Another type of user input device is a cursor control 1023, such as a mouse, trackball, or cursor direction keys, for communicating directional information and command selections to processor(s) 1002 and for controlling cursor movement on display 1021. The computing device of FIG. 10 may be coupled to a network 1026 via a communication interface (e.g., a modem, network interface card, or NIC) 1008.

[0054] As shown, storage 1007 may include a direct-access data storage device, such as a magnetic disk 1030, non-volatile semiconductor memory (EEPROM, Flash, etc.) 1032, or a hybrid data storage device comprising both magnetic disk and non-volatile semiconductor memory, as indicated at 1031. Reference numerals 1004, 1006, and 1007 are examples of tangible, non-transitory computer-readable media having data stored thereon that represent sequences of instructions that, when executed by one or more computing devices, implement aspects of the distributed systems and computer-implemented methods described and illustrated herein. Some of these instructions may be stored locally at a client computing device, while other of these instructions may be stored (and / or executed) remotely and communicated to a client computing device over network 1026. In other embodiments, all of these instructions may be stored locally at a client or other standalone computing device, while in still other embodiments, all of these instructions are stored and executed remotely (e.g., at one or more remote servers), with the results communicated to the client computing device. In yet another embodiment, the instructions (processing logic) may be stored on another form of tangible, non-transitory computer-readable medium, such as that shown at 1028. For example, reference numeral 1028 may be implemented as an optical (or some other storage technology) disk, which may constitute a suitable data carrier such that the instructions stored thereon can be loaded into one or more computing devices, thereby reconfiguring the computing device(s) into one or more embodiments described and illustrated herein. In another implementation, reference numeral 1028 may be embodied as an encrypted solid-state drive. Other implementations are possible.

[0055] Embodiments of the present invention relate to the use of computing devices to perform the functions disclosed herein. According to one embodiment, the methods, devices, and systems described herein may be provided by one or more computing devices in response to processor(s) 1002 executing sequences of instructions, contained in memory 1004, embodying aspects of the computer-implemented methods illustrated and described herein. Such instructions may be read into memory 1004 from data storage device 1007 or another computer-readable medium, such as another data carrier (optical, magnetic, etc.), as illustrated at 1028. Execution of the sequences of instructions contained in memory 1004 causes processor(s) 1002 to perform the steps and functions described herein. In alternative embodiments, hardwired circuitry may be used in place of or in combination with software instructions to implement the described embodiments. Thus, the embodiments are not limited to any specific combination of hardware circuitry and software. Indeed, those skilled in the art should appreciate that any suitable computer system may implement the functions described herein. A computing device may include one or more microprocessors operating to perform desired functions. In one embodiment, instructions executed by a microprocessor or microprocessors are operable to cause the microprocessor(s) to perform the steps described herein. The instructions may be stored on any computer-readable medium. In one embodiment, the instructions may be stored on non-volatile semiconductor memory external to or integrated with the microprocessor. In another embodiment, the instructions may be stored on a disk and loaded into volatile semiconductor memory prior to execution by the microprocessor.

[0056] Some portions of the above detailed description describe symbolic representations of processes and operations by a computing device, which may include computer components including a local processing unit, memory storage for the local processing unit, a display device, and input devices. A command, as the term is used in this disclosure, may correspond to a high-level directive from a client process, which may result in the execution of multiple operations on one or more computers. Operations may include single-machine instructions. Furthermore, such processes and operations may utilize computer components in a heterogeneous distributed computing environment, including, for example, remote file servers, computer servers, and memory storage devices. These distributed computing components may be accessible to the local processing unit over a communications network.

[0057] The computer-executed processes and operations involve the manipulation of data bits by a local processing unit and / or a remote server, and maintaining those bits in data structures in one or more local or remote memory storage devices that provide a physical organization for the collections of data bits stored in the memory storage devices and represent elements of the electromagnetic spectrum.

[0058] Processes, such as computer-implemented methods described and illustrated herein, may be generally defined as a sequence of computer-executed steps leading to a desired result. These steps generally require physical manipulations of physical quantities. Usually, though not necessarily, these quantities may take the form of electrical, magnetic, or optical signals capable of being stored, transferred, combined, compared, and otherwise manipulated. Conventionally, those skilled in the art refer to these signals as bits or bytes (when the signals have binary logic levels), pixel values, work, values, elements, symbols, characters, terms, numbers, points, records, objects, images, files, directories, subdirectories, or the like. However, it should be borne in mind that these and similar terms are intended to be associated with the appropriate physical quantities for computer commands and that these terms are merely conventional labels applied to physical quantities within the operating range and during the operation of a computer.

[0059] It should also be understood that operations within a computer are often referred to in terms such as adding, comparing, moving, positioning, arranging, illuminating, removing, modifying, etc. The commands described herein are machine operations that are performed in conjunction with various inputs provided by a human or artificial intelligence agent operator or user interacting with the computer. Machines used to perform the operations described herein include local or remote general purpose digital computers or other similar computing devices.

[0060] It should also be noted that the programs, processes, methods, etc. described herein are not related to or limited to any particular computer or apparatus, or to any particular communications network architecture. Rather, various types of general-purpose hardware machines may be used with program modules constructed in accordance with the teachings described herein. Similarly, it may prove advantageous to construct specialized apparatus to perform the method steps described herein through a special-purpose computer system of a particular network architecture having hardwired logic or programs stored in non-volatile memory, such as read-only memory.

[0061] While certain exemplary embodiments have been described, these embodiments are presented by way of example only and are not intended to limit the scope of the embodiments disclosed herein. Thus, nothing in the foregoing description is intended to imply that any particular feature, characteristic, step, module, or block is necessary or essential. Indeed, the novel methods and systems described herein may be embodied in a variety of other forms; furthermore, various omissions, substitutions, and changes may be made in the form of the methods and systems described herein without departing from the spirit of the embodiments disclosed herein.

Claims

1. receiving a proposal to change data stored in a distributed replicated file system coupled to a network, the distributed replicated file system including a plurality of nodes, each node including a server; a metadata service configured to maintain a replica of a namespace of the distributed replicated file system; coordinating updates to the data by generating an ordered set of agreements corresponding to the received proposals, the ordered set of agreements specifying an order in which the nodes will modify data stored in data nodes to cause changes to the state of the namespace, and each of the nodes is configured to delay making modifications to the data stored in the data nodes to cause changes to the state of the namespace until the ordered set of agreements is received; for each agreement in the generated ordered set of agreements, providing an index to or identifying a corresponding writer list including an ordered list of nodes to implement the agreement and make corresponding changes to the namespace; sending the ordered set of agreements to the plurality of nodes, together with, for each agreement in the ordered set of agreements, the corresponding contributor list or a specified index for the corresponding contributor list, wherein each of the plurality of nodes is configured to execute only those agreements for which it is the first listed node on the provided or specified index for the corresponding contributor list; causing the metadata service to update the maintained replica of the namespace; 10. A computer-implemented method comprising:

2. The computer-implemented method of claim 1 , wherein providing comprises generating the contributor lists for at least some of the generated ordered sets of agreements.

3. The computer-implemented method of claim 1 , wherein providing comprises selecting from among a plurality of pre-generated contributor lists.

4. 2. The computer-implemented method of claim 1, wherein the writer list includes an ordered list of preferred working nodes toward the top of the writer list and a list of failed or suspected failed nodes toward the bottom of the writer list.

5. 2. The computer-implemented method of claim 1, wherein providing further comprises providing the same list of contributors for a first proposal as well as for a second proposal that depends on the execution of a previous first proposal.

6. 2. The computer-implemented method of claim 1, further comprising making the agreement enforceable to a next listed node in the contributor list when a first listed node in the contributor list fails or is suspected of failing.

7. The computer-implemented method of claim 1 , further comprising enabling a predetermined replacement node to enforce the agreement when an initially listed node in the writer list fails.

8. The computer-implemented method of claim 1 , further comprising: transmitting, by each node that executed an agreement, information regarding the executed agreement to each of the plurality of nodes.

9. The computer-implemented method of claim 8 , wherein transmitting further comprises ensuring delivery of the transmitted information.

10. 10. The computer-implemented method of claim 1, further comprising: when a node executes an agreement, updating a deterministic state machine with information about the executed agreement, the deterministic state machine being coupled to each other node and serving as a persistent messaging service between the plurality of nodes.

11. 10. The computer-implemented method of claim 1, further comprising learning of a failing or failed node and placing the failing or failed node at the bottom of any generated writer list.

12. pre-generating the indexed writer list for each of all possible combinations of orderings of the plurality of nodes; distributing the pre-generated indexed writer list to each of the plurality of nodes; Further comprising: providing includes selecting one of the indexed writer lists; 2. The computer-implemented method of claim 1, wherein transmitting comprises transmitting the ordered set of agreements to the plurality of nodes along with, for each agreement in the ordered set of agreements, an index to a selected one of the pre-generated indexed writer lists.

13. 1. A network of servers configured to implement a distributed file system, said network of servers comprising: a plurality of data nodes, each configured to store data blocks of a client file; multiple nodes, each containing a server; a metadata service configured to maintain and update the state of the namespace of the network of servers in response to changes to data blocks of the client files; a distributed coordination engine embedded in the metadata service and configured to coordinate received proposals to update the data blocks by generating an ordered set of agreements corresponding to the received proposals, the ordered set of agreements specifying the order in which the nodes will make changes to data stored in the data nodes; and Including, the metadata service is further configured to, for each agreement in the generated ordered set of agreements, provide or generate an index to a corresponding author list including an ordered list of nodes to implement the agreement and cause corresponding changes to the namespace; the metadata service is further configured to send the ordered set of agreements to the plurality of nodes, along with, for each agreement in the ordered set of agreements, a corresponding author list or pre-generated index, such that each of the plurality of nodes enables execution of only agreements for which it is the first listed node in the author list; A network of servers.

14. 14. The network of servers of claim 13, wherein the metadata service is further configured to generate the contributor lists for at least some of the generated ordered sets of agreements.

15. 14. The network of servers of claim 13, wherein the metadata service is further configured to select from among a plurality of pre-generated contributor lists.

16. 14. The network of servers of claim 13, wherein the writer list includes an ordered list of preferred working nodes toward the top of the writer list and a list of failed or suspected failed nodes toward the bottom of the writer list.

17. 14. The network of servers of claim 13, wherein the metadata service is further configured to provide the same contributor list for a second proposal that depends on the execution of a previous said first proposal in addition to providing it for a first proposal.

18. 14. The network of servers of claim 13, wherein the metadata service is further configured to make the agreement enforceable to a next listed node in the contributor list when a first listed node in the contributor list fails.

19. 14. The network of servers of claim 13, wherein the metadata service is further configured to enable a predetermined replacement node to enforce the agreement when an initially listed node in the writer list fails.

20. 14. The network of servers of claim 13, wherein each node that has executed an agreement is further configured to broadcast information regarding the executed agreement to each of the plurality of nodes.

21. 14. The network of servers of claim 13, wherein upon executing an agreement, the node is further configured to update a deterministic state machine with information about the executed agreement, the deterministic state machine being coupled to each other node and serving as a persistent messaging service between the plurality of nodes.

22. 14. The network of servers of claim 13, wherein the metadata service is further configured to learn of failing or failed nodes and to place the failed or failing node at the bottom of any generated contributor list.

23. The metadata service pre-generating the indexed writer list for each possible combination of orderings of the plurality of nodes; distributing the pre-generated indexed writer list to each of the plurality of nodes; 14. The network of servers of claim 13, further configured to transmit the ordered set of agreements to the plurality of nodes along with, for each agreement in the ordered set of agreements, an index to a selected one of the pre-generated indexed writer lists.

Images