Method, apparatus and system for writer preselection in a distributed data system

Abstract

A computer-implemented method can include receiving a proposal to modify data stored in a distributed and replicated file system coupled to a network, the distributed and replicated data system including a plurality of nodes, each node including a server; a metadata service maintaining and updating a copy of a namespace of the distributed and replicated file system and coordinating updates to the data by generating an ordered set of protocols corresponding to the received proposal, the ordered set of protocols specifying an order of nodes to modify data stored in the data nodes and cause corresponding changes in a state of the namespace. For each protocol in the generated ordered set of protocols, a corresponding writer list can be provided, the corresponding writer list including an ordered list of nodes to execute the protocol and make the corresponding changes to the namespace. The ordered set of protocols can then be sent to the plurality of nodes along with the corresponding writer list or a pre-generated index thereof for each protocol in the ordered set of protocols, and each node in the plurality of nodes can be configured to only execute the protocol of the first listed node on the received writer list.

Description

Technical Field

[0001] This invention relates to the field of distributed replicated data systems, and in particular to methods, apparatus and systems for writer preselection in distributed data systems. Background Technology

[0002] The field of embodiments disclosed herein includes distributed replicated data systems. Some distributed data systems can define a logical structure called a region. Each such region can include servers whose task is to execute received commands by writing to and reading from their region's metadata service (hereinafter referred to as MDS). Many systems provide this metadata service through a single server. In applications where read consistency is sacrificed for performance improvements, read commands may optionally be configured to bypass the server. When multiple servers are used for failover in each region, a selection must be made to ensure that only one server in each region is designated to execute a given command at any given time, although different commands may have different designated servers. In this case, logic is provided to select a new server that allows writes when it believes that the currently write-enabled server has crashed. However, it can be difficult to reliably distinguish between a crashed server and a slow-running server. In fact, when no periodic heartbeat signal indicating that the server is continuing to operate normally is received within the expected time frame, the server may indeed have crashed, or it may simply be slowing down, or it may be delayed or unable to communicate with the server due to temporary network latency issues or other reasons. In this scenario, if new server writes are enabled programmatically, and the old server resumes operation for any reason, it's now possible to have two servers executing the same commands by writing to the backend storage. As a result, data integrity can no longer be guaranteed, because the two servers are now executing the same commands independently, which could lead to data corruption.

[0003] Therefore, some distributed systems require manual verification that the first server is terminated or inactive before the second server can write to the backend data store. This verifies that the server thought to be terminated is indeed terminated, thus preventing two active writers from existing in the same area simultaneously. This manual verification is not optimal because no data can be written until the currently active writer is manually verified (and thus disabled). Summary of the Invention

[0004] This invention provides a method, apparatus, and system for writer pre-selection in a distributed data system.

[0005] The technical solution provided by this invention is: a computer-implemented method, comprising:

[0006] The system receives proposals to modify data stored in a distributed and replicated file system coupled to a network. The distributed and replicated data system includes multiple nodes, each of which includes a server.

[0007] Configure the metadata service to maintain copies of the namespace of the distributed and replicated file system;

[0008] Data updates are coordinated by generating an ordered set of protocols corresponding to the received proposals. The ordered set of protocols specifies the order in which nodes modify the data stored in the data nodes and cause changes to the state of the namespace. Each node is configured to delay modifying the data stored in the data nodes and causing changes to the state of the namespace until the ordered set of protocols is received.

[0009] For each protocol in the generated ordered protocol set, an index is provided or identified for the corresponding writer list, which includes an ordered list of nodes to execute the protocol and make corresponding changes to the namespace. Each writer list includes a preferred running node toward a first end of the writer list and an ordered list of failed or suspected failed nodes toward a second end of the writer list.

[0010] An ordered protocol set, along with an index of the corresponding list of writers or the identifier of the corresponding list of writers for each protocol in the ordered protocol set, is sent to multiple nodes, each of which is configured to execute only the protocol of the first node listed on the provided or identified index of the corresponding list of writers.

[0011] When a previously listed node in the writer list fails or is suspected of failing, the next listed node in the writer list is able to execute the protocol; and

[0012] This enables the metadata service to update copies of the namespace it maintains.

[0013] Preferably, the method includes generating a list of writers for at least some of the generated ordered protocol sets.

[0014] Preferably, the option includes selecting from a list of multiple pre-generated writers.

[0015] Preferably, the provision also includes providing the same list of writers for the second proposal as for the first proposal after the first proposal is executed.

[0016] Preferably, it also includes the ability of a predetermined replacement node to execute the protocol when the first node listed in the writer list fails.

[0017] Preferably, it also includes each node that has executed the protocol propagating information related to the executed protocol to each of the multiple nodes.

[0018] Preferably, dissemination also includes ensuring the transmission of the disseminated information.

[0019] Preferably, the method further includes updating the deterministic state machine with information related to the protocol being executed when the node executes the protocol. The deterministic state machine is coupled to each of the other nodes and serves as a persistent message service between multiple nodes.

[0020] Preferably, it also includes identifying faulty or failed nodes and placing the faulty or failed nodes at the bottom of any generated writer list.

[0021] Preferred options also include:

[0022] For each of all possible combinations of the order of multiple nodes, pre-generate a list of writers for the index;

[0023] The list of writers for the pre-generated index is distributed to each of the multiple nodes.

[0024] It provides a list of writers, including the option to select one of the indexes, and

[0025] The sending process involves sending an ordered set of protocols along with an index for each protocol in the ordered set to multiple nodes, where the index points to a selected list of writers from a pre-generated list of indexes.

[0026] This invention also provides another technical solution: a network configured to implement a distributed file system, the distributed file system comprising:

[0027] Multiple data nodes, each configured to store data blocks of client files;

[0028] Multiple nodes, each node including a server;

[0029] Metadata service, configured to maintain and update the state of the namespace of the distributed file system in response to changes in data blocks of client files;

[0030] The distributed coordination engine, embedded in the metadata service, is configured to coordinate received proposals to update data blocks by generating an ordered set of protocols corresponding to the received proposals. The ordered set of protocols specifies the order in which nodes modify the data stored in the data nodes.

[0031] The metadata service is also configured to provide or generate an index to a corresponding writer list for each protocol in the generated ordered protocol set. The corresponding writer list includes an ordered list of nodes that execute the protocol and cause corresponding changes to the namespace. Each writer list includes an ordered list of preferred running nodes toward the first end of the writer list and an ordered list of failed or suspected failed nodes toward the second end of the writer list.

[0032] Furthermore, the metadata service is configured to send an ordered protocol set along with a list of writers or their pre-generated index for each protocol in the ordered protocol set to multiple nodes, such that each of the multiple nodes can only execute the protocol of the first node listed on the writer list, and when a previously listed node in the writer list has failed or is suspected of failing, the next listed node in the writer list is enabled to execute the protocol.

[0033] Preferably, the metadata service is also configured to generate a list of writers for at least some of the generated ordered protocol sets.

[0034] Preferably, the metadata service is also configured to select from a list of multiple pre-generated writers.

[0035] Preferably, the metadata service is also configured to provide the same list of writers for the second proposal as for the first proposal after the first proposal is executed.

[0036] Preferably, the metadata service is also configured such that a predetermined replacement node can execute the protocol when the first node listed in the writer list fails.

[0037] Preferably, each node that has executed the protocol is also configured to propagate information related to the executed protocol to each of the multiple nodes.

[0038] Preferably, when executing a protocol, the node is also configured to update the deterministic state machine with information related to the protocol being executed. The deterministic state machine is coupled to each of the other nodes and serves as a persistent message service among the multiple nodes.

[0039] Preferably, the metadata service is also configured to identify nodes that have failed or are in a faulty state and to place such nodes at the bottom of any generated list of writers.

[0040] Preferably, the metadata service is also configured as follows:

[0041] For each of all possible combinations of the order of multiple nodes, pre-generate a list of writers for the index;

[0042] Distribute the list of writers for the pre-generated index to each of the multiple nodes, and

[0043] The ordered protocol set, along with an index for each protocol in the ordered protocol set, is sent to multiple nodes, with the index pointing to a selected list of writers in a pre-generated index.

[0044] The beneficial effects of this invention after adopting the above technical solution are as follows:

[0045] This technical solution includes receiving proposals to modify data stored in a distributed and replicated file system coupled to a network. The distributed and replicated data system includes multiple nodes, each including a server. A metadata service maintains and updates copies of the namespace of the distributed and replicated file system and coordinates the updates to the data by generating an ordered set of protocols corresponding to the received proposals. The ordered set of protocols specifies the order in which nodes modify data stored in the data nodes, causing corresponding changes to the state of the namespace. For each protocol in the generated ordered set of protocols, a corresponding list of writers, comprising an ordered list of nodes, is provided to execute the protocol and make corresponding changes to the namespace. The ordered set of protocols, along with the corresponding list of writers or their pre-generated index for each protocol in the ordered set, can then be sent to multiple nodes. Each of the multiple nodes can be configured to execute only the protocol of the first node listed on the received list of writers, thereby significantly optimizing the prior art. Attached Figure Description

[0046] Figure 1 This is an illustration of a distributed replicated data system in which embodiments can be implemented.

[0047] Figure 2 This is a list of writers and illustrations of various aspects of the method implemented by this computer according to an embodiment.

[0048] Figure 3 This is a list of writers and illustrations of various aspects of the method implemented by this computer in the presence of a faulty (or suspected faulty) node, according to an embodiment.

[0049] Figure 4 This is a flowchart illustrating various aspects of a computer-implemented method according to an embodiment.

[0050] Figure 5 This is a flowchart illustrating various aspects of a computer-implemented method according to an embodiment.

[0051] Figure 6 This is a diagram illustrating a distributed state machine spanning a single proposer and multiple learners according to an embodiment.

[0052] Figure 7This is a diagram illustrating a pre-generated and indexed list of writers according to one embodiment.

[0053] Figure 8 This is a diagram illustrating a pre-generated and indexed list of writers according to one embodiment.

[0054] Figure 9 This is a flowchart of a computer-implemented method according to one embodiment.

[0055] Figure 10 This is a block diagram of a computing device that can be used to practice the embodiments shown and described herein. Detailed Implementation

[0056] definition

[0057] Distributed systems: A distributed system comprises a collection of different computing and / or storage processes and / or devices that can be spatially separated and can communicate with each other by exchanging messages or events.

[0058] Copying state machine: The replicated state machine approach is a method for achieving fault-tolerant services by replicating servers and coordinating client interactions with these replicas. These state machines are "replicated" because the states of the state machine evolve in the same ordered manner across all learners. A replica of the single server executes on an independent processor in the distributed system, and protocols are used to coordinate client interactions with these replicas. An instance and implementation of the replicated state machine is the deterministic state machine (DSM), which advances its state in a deterministic manner.

[0059] Proposers: According to one embodiment, the proposer is a process configured to suggest proposals, some of which can be configured to modify data.

[0060] Acceptors: According to one embodiment, acceptors are configured to participate in the process of determining the order in which proposals are made by proposers. According to one embodiment, a proposal becomes a protocol (e.g., a consensus proposal) only when a majority of acceptors have determined that the proposal occupies a specific position in the global sequence of protocols (described further below). According to one embodiment, acceptors can be configured to participate only in determining the order of protocols, regardless of the underlying content of the protocols (as described herein, the values ​​of the protocols are opaque to the distributed coordination engine, also as described below). Acceptors can be configured as application-independent entities.

[0061] Learners:According to one embodiment, learners learn the agreement reached between proposers and acceptors and apply the agreement to the application in a deterministic order through their output proposal sequence. In one embodiment, a protocol identifier is provided as persistent storage, which allows the sequence of protocols to be persistently recorded for each replicated state machine. Each proposal is guaranteed to be offered at least once to each learner among a particular member.

[0062] The Hadoop Compatible File System (HCFS) namespace is a hierarchical structure of files and directories. Hadoop is an open-source Java-based programming framework that supports the processing and storage of extremely large datasets in a distributed computing environment. It is part of the Apache project, sponsored by the Apache Software Foundation. Files and directories are represented by inodes on the namenode. Inodes record attributes or metadata, such as permissions, modification and access times, namespaces, and disk space quotas. File content is divided into large blocks (typically 128MB), and each block of a file is independently replicated on multiple datanodes (typically three). One implementation of HCFS is the Hadoop Distributed File System (HDFS). The namenode is the metadata service for HDFS, responsible for tracking changes in the namespace. The namenode maintains the namespace tree and the mapping of blocks to datanodes. That is, the namenode tracks the location of data in the Hadoop cluster and coordinates client access to that data. Typically, there is one namenode per cluster. Since each datanode can concurrently execute multiple application tasks, each cluster can have thousands of datanodes and tens of thousands of HDFS clients. The list of inodes and data blocks that defines the metadata of the namesystem is called an image. Name nodes store the entire namespace image in random access memory (RAM).

[0063] The roles of proposer (the process of proposing to change the state of a namespace to membership), acceptor (the process of voting to decide whether a proposal to change the state of a namespace should be accepted by membership), and learner (the process of learning in membership that an agreement has been reached) are defined, for example, in an implementation of the Paxos algorithm described in Lamport, L.: The Part-Time Parliament, ACM Transactions on Computer Systems 16,2 (May 1998), 133-169, the entire text of which is incorporated herein by reference. According to one embodiment, multiple nodes can be configured to perform each role. A distributed coordination engine (also known as DConE) can allow multiple learners, with the help of multiple acceptors, to agree on the order of events submitted to them by multiple proposers to achieve high availability. To achieve reliability, availability, and scalability, multiple concurrently active name nodes (collectively referred to herein as MDSs) can be provided by replicating the state of the namespace across multiple nodes, while requiring the state of the node containing the replicated namespace to remain consistent across these nodes. However, in one embodiment, as described below, only one MDS exists in each region.

[0064] Consistency between NameNodes in different regions can be guaranteed by a DConE, which can be configured to accept proposals to update the namespace, simplify the proposals into an ordered sequence of global updates, and then allow the MDS to learn and apply the updates to its respective state only in a specified, agreed-upon order. In this paper, "consistency" refers to One-Copy Equivalence, as described in "Concurrency Control & Recovery in Database Systems," published by Addison Wesley, 1987, Chapters 6, 7 & 8, by Bernstein et al., the entire contents of which are incorporated herein by reference. Because NameNodes start from the same state and apply the same deterministic updates in the same deterministic order, their respective states change identically over time, thus maintaining consistency.

[0065] Therefore, according to one embodiment, a namespace can be replicated across multiple NameNodes (or more generally, Metadata Servers or MDS), provided that...

[0066] a) Allow each MDS to modify its namespace copy, and

[0067] b) An update to a namespace copy must be propagated to namespace copies on other MDSs in other regions to ensure that namespace copies remain consistent across MDSs and regions.

[0068] Figure 1 A cluster running a single distributed file system 102 is shown, which spans different geographically (or other) separated regions. For example, a distributed file system can combine various aspects of HDFS. Each data node (in...) Figure 1 The DN (displayed as "DN") can be configured to communicate only within its own region (via the Data Node to Server Remote Process Call (RPC) protocol). That is, the DN of region 1 can communicate only with nodes (servers) 110, 112, 114… of region 1 (or nodes adjacent to region 1), and the DNs 132, 134, 136, 138… of region 2 can communicate only with nodes (services) 116, 118, 120… of region 2 (or servers adjacent to region 2). In one embodiment, only one metadata service (MDS) storing a copy in a namespace may exist in each region, as shown by MDS 103 for region 1 and MDS 105 for region 2. Conversely, the nodes (servers) of each region communicate only with the MDS of their respective region. In this way, nodes 110, 112, 114… communicate with MDS 103, and nodes 116, 118, 120… of region 2 communicate only with MDS 105. The MDSs in Region 1 and Region 2 can coordinate with each other using one or more (e.g., an odd number such as 3 for high availability (HA)) inter-region servers 140, 142 to keep the namespace state consistent across different regions of the distributed file system 102 through streaming namespace changes via WAN 108 across regions. These changes can be received by servers (nodes) in another region, which then modify the backend storage of the local write region, enabling all reads and writes to be performed as local operations, thus eliminating the need for cross-region security.

[0069] The DConE process 122 can be configured to guarantee identical deterministic updates to the state of a namespace, which is applied to all MDSs across all regions in the same deterministic order. In one embodiment, the DConE process 122 can be embedded within the MDS of each region. This deterministic order can be defined by a Global Sequence Number (GSN). Therefore, a key function of the DConE process 122 is to process agreed-upon proposals to modify or otherwise update the state of namespace replicas based on commands received from HDFS clients by servers in each region, and to transform them into a globally ordered sequence of protocols indexed by the GSN. Servers (or nodes, since the terms are interchangeable) can then sequentially apply protocols from this ordered sequence, which produces updates to the state of the replicas of the namespace in their region. The GSN can be configured as a unique, monotonically increasing number. However, as those skilled in the art will recognize, the GSN can be configured in other ways. In this way, the state of the replicas of the namespace stored in each region is kept consistent by executing the ordered (via the GSN mechanism) protocol set generated by the DConE process 122 in an ordered manner and by streaming changes from each region to all other regions. Since the MDS starts from the same state, this ordered application of updates ensures the consistency of replicas across regions, because snapshots of replicas on the MDS of the same protocol processed by the same GSN are identical both within and across regions.

[0070] Metadata in the namespace replicas maintained by MDS can be reconciled (or nearly reconciled, considering inherent bandwidth and latency in the network) on the fly during the DConE process 122 protocol delivery and during streaming changes between regions. Similarly, all file system data is automatically replicated within the distributed file system. In this way, consistent and continuous data replication occurs between file systems in a cluster (e.g., but not limited to Hadoop). Client applications can be configured to interact with a virtual file system that integrates the underlying storage across multiple regions. When changes are made to files in one region, these changes are consistently replicated to other regions. One embodiment may include a software application that allows Hadoop deployments to replicate HCFS data between clusters running different or even incompatible versions of Hadoop (e.g., Hadoop itself), such as CDH, HDP, EMC Isilon, Amazon S3 / EMRFS, and MapR. According to one implementation, replication can also be performed between different vendor distributions and Hadoop versions.

[0071] Advantageously, embodiments may provide a virtual file system compatible with all Hadoop applications and for Hadoop, a single virtual namespace integrating storage from different types of Hadoop, a globally distributed storage mechanism, and WAN replication using active-active replication technology, which delivers a single copy of consistent HDFS data replicated between remote data centers.

[0072] According to one embodiment, some or all of the functions described herein can be executed at a higher level in the distributed file system stack, within one or more servers adjacent to the MDS. In this way, an embodiment can be configured to operate as a proxy application for the distributed file system, rather than working in-depth at the namenode level.

[0073] In an agreed-upon, proposed, ordered global sequence of commands (published by a deterministic state machine (DSM) or other source), some commands in the sequence may depend on other commands. Dependent commands must be executed in the correct order. For example, consider commands A, B, and C, where command B depends on command A, and command C depends on both commands B and A. For load balancing or other purposes, the execution load of these commands may need to be distributed across multiple servers or nodes within the region. For example, node 1 might be assigned to execute command A, node 2 might be assigned to execute command B, and node 1 might be assigned to execute command C. For node 2 to execute command B, it must know that node 1 has completed executing command A. Similarly, when node 1 executes command C, it will need to verify that node 2 has completed executing command B. This approach risks introducing significant delays in processing client change requests (i.e., commands) when inter-node communication is delayed.

[0074] As mentioned above, the execution of dependent commands must be delayed until one or more of the commands they depend on have been executed. Independent commands can be executed in parallel. In the event of failure, commands can be re-executed, but under no circumstances can a sequence of commands be re-executed. That is, each command is idempotent (it will produce the same result if executed once or multiple times), but a sequence of more than one command is not idempotent because re-executing the same sequence will not produce the same result. Therefore, although each command can be executed more than once individually, a sequence of these commands may not be executed.

[0075] For scalability and high availability, multiple servers can execute commands, and each server within a region can execute multiple independent commands in parallel. Even if each node receives the entire ordered global command sequence, each command in the global sequence should be executed by only a single node. Here, nodes or servers are considered unreliable (because they are prone to failure and may or may not recover) and are configured to communicate with each other as described above.

[0076] One embodiment is a computer-implemented writer preselection method (i.e., preselecting (in one implementation, preferred) nodes that will execute commands configured to update the state of a namespace), which achieves scalability, high availability, and ensures rapid and reliable execution of the constituent commands of an ordered global sequence, while maintaining security to prevent the re-execution of the command sequence.

[0077] Writer list

[0078] According to one embodiment, and as Figure 2 As shown, before being inserted into the ordered global sequence generated by DConE 122, each command or instruction can be associated with a list of nodes (servers) to execute commands ordered according to execution preferences (in the case of consistency, the protocol). In one embodiment, this list of nodes can be ordered such that nodes appear in the list in a priority order of execution. This ordered list of nodes is also referred to herein as writer list 202. According to one embodiment, each node can be configured to execute only those commands from the first node in the received ordered writer list. According to one embodiment, if the first node in the writer list becomes inoperable (e.g., fails to receive the expected heartbeat signal in time), the next operable node or server in the ordered writer list 202 can be designated as the node to execute the command. Thereafter, nodes propagate information, including the commands they have executed, to all or a predetermined number of nodes in real-time or near real-time, ensuring that all nodes (including themselves) are promptly notified of all commands that have been executed.

[0079] like Figure 2As shown, Command 1 is a standalone command and is associated with Writer List 202, in which Server 112 is the first listed node. Therefore, Node 112 is the preferred server for executing Command 1. If Node 112 fails for any reason before or after executing Command 1, but before it can propagate the fact that it has executed Command 1, execution of Command 1 falls back to Node 110 (the next listed node in Writer List 202), and subsequently, if Server 110 fails, it falls back to the next listed node 114. Any known failed nodes are demoted to the bottom of the Writer List generated (or otherwise accessed) for subsequent commands. Since Command 2 depends on the execution of Command 1, Writer List 204 is therefore associated with it, and this Writer List is identical to Writer List 202. Command 3, as a standalone command, may have a different Writer List 206 associated with it. Note that the same writer lists as writer lists 202 and 204 may have been associated with command 3, but load balancing and / or other considerations may have recommended a different writer list for this command.

[0080] More specifically, according to one embodiment, the process placing commands into an ordered global protocol sequence can identify nodes suspected of failure and nodes that have already failed, and can track the operational status of all nodes. According to one embodiment, for each new execution of a client command, the writer list may include an ordered sequence of preferred runnable nodes that should execute the command, followed in one implementation by failed, failed, or suspected nodes. Therefore, when a node fails or is suspected of failing, such a node will be pushed to the bottom of the writer list.

[0081] In one embodiment, the runnable nodes in the writer list can be sorted such that slave commands will preferentially receive the same writer list (e.g., ...). Figure 2 As shown in 202 and 204 (in the original text), to reduce dependencies in execution between nodes, the nodes can be ordered such that all nodes have approximately the same location distribution. In practice, dependent commands will preferably receive the same list of writers to reduce dependencies in execution between nodes and subsequent latency. This is because the node chosen to execute the independent command is the best suited to execute the command that depends on it, avoiding the inherent latency of waiting for updates to propagate to other nodes to execute dependent commands. According to one embodiment, all nodes can have approximately the same location distribution within the generated list of writers to distribute the computational load evenly across all available (i.e., runnable) servers or nodes. In some cases, other considerations may determine a different order of nodes in the list of writers, thus modifying the preferences listed above.

[0082] Node failure

[0083] Servers are considered fault-prone. A node failure means it will stop executing commands, it will stop notifying other nodes of its execution status, or both. Other nodes cannot distinguish between these two situations (they cannot distinguish between slow, non-reporting, or failed nodes). According to one embodiment, when a node fails, its position in the writer list changes for at least all new or subsequent commands. In practice, according to this embodiment, the perceived node failure causes the node to be pushed back in the writer list for new or subsequent commands (i.e., away from the top, towards the bottom), so eventually the system will stop generating computational load for this perceived failed node until the server recovers and signals that it has returned to its nominal operating state. Figure 3 In this context, node 112 is either faulty or suspected of being faulty. When a new command is issued, node 112 will be demoted to the bottom or near the bottom of writer list 302, which is associated with the new command.

[0084] Commands previously designated to be executed by the failed node as the preferred writer will be stalled from the perspective of other nodes (and any commands dependent on them) if they have not yet been executed. According to an embodiment, if an external entity reliably confirms that a suspected failed node is inoperable and will not suddenly resume operation, these stalled commands can be handled in one of two ways:

[0085] 1) Dead node declaration

[0086] like Figure 4 As shown in 402, when node X fails, the failure of node X can be notified to all other nodes (in one embodiment, via the DConE process embedded in the MDS), as shown in 404. Upon receiving such a notification, according to an embodiment and as shown in 406, a node can remove node X from the writer list from all pending commands that have node X as the first preferred node for execution purposes, as... Figure 4 As shown, the command will be executed by another (in one embodiment, the next) node in the writer list, as shown in 408, now that node X has been removed from the top of the writer list.

[0087] 2) Replace Node

[0088] Similarly, such as Figure 5As shown in Figure 402, when node X fails, the failure of node X can be notified to all other nodes (in one embodiment, via a DConE process embedded in the MDS of a region, which includes the failed node X) as shown in Figure 502. Upon receiving such notification, according to an embodiment and as shown in Figure 406, the node can remove node X from the writer list of all pending commands that have node X as the first preferred node for execution purposes. According to one embodiment, a single alternative running node can be designated to take over the commands previously assigned to the failed node X for execution. This newly designated alternative node can be a new node (server), in which case the new node can query other nodes, as shown in Figure 504, to determine which commands were assigned to node X for execution, so that node X can execute these commands in place of the failed node X, as shown in Figure 506.

[0089] Execution state propagation

[0090] Each node propagates information about the commands it is executing for two reasons:

[0091] (1) Remove any subordinate commands assigned to other MDS, and

[0092] (2) To prevent the re-execution of non-idempotent commands in the event of node failure. Note that, according to one embodiment, if a command is idempotent, it is not necessary to prevent the immediate re-execution of the command without intervention from subordinate commands. Specifically, the embodiment does not require and does not defend against server failures after command execution but before the propagation of the information. Instead, according to one embodiment, non-idempotent commands are modified to be equivalent in result but idempotent.

[0093] The manner in which this propagation occurs does not need to be strictly limited. Regardless of how this propagation occurs, the following constraints or properties should be satisfied:

[0094] ● In order to satisfy reason (1), the propagation should occur rapidly, that is, close to real time.

[0095] ● The act of propagating information about executed commands should be persistent and guaranteed to deliver. In this way, even nodes that have failed and eventually recovered can and will eventually learn about the propagated information.

[0096] ●As long as the node that is propagating the information (the node executing the command that is propagating the information) is operational, the act of propagating information about the executed command should be operational.

[0097] ● To ensure continuity, each node should be able to acknowledge to other nodes that it has transmitted this propagation information.

[0098] One embodiment includes a deterministic finite state machine having a single proposer, a single receiver (the writing node itself), and numerous learners (other nodes) who are aware of the command's result. In such an embodiment, each execution server (node) has its own deterministic finite state machine. Such a finite state machine can then be used as a mechanism for information propagation between nodes, as described below.

[0099] Continuity guarantee

[0100] Even if a node fails, the execution state of any command must not be lost unless idempotency allows. Suppose command C is executed. Before this information is fully propagated, command D, which depends on C, is executed. At this point, even if C is idempotent, re-executing it would violate safety, but re-execution might occur due to insufficient propagation. For example, a node might execute a series of commands but fail before successfully propagating its "command complete" update state. Subsequently, another node might erroneously re-execute the command sequence, which would violate safety. Therefore, according to an embodiment, to tolerate the failure of N nodes, according to one embodiment, a node must confirm that propagation to at least N+1 nodes (including itself) has been successful before executing any later command in the command sequence that depends on the result of that command. Different strategies and risk tolerances will determine the number of nodes that must confirm propagation.

[0101] In the Paxos consensus algorithm, participants have the following roles: proposer (the process that proposes a command to change the state of a namespace to membership), acceptor (the process that votes to decide whether a command to change the state of a namespace should be agreed upon by membership), and learner (the process that learns from the agreement reached within membership). According to one embodiment, a node can be configured to perform any of these roles at any given time. As mentioned above, the DConE process can allow multiple learners, with the help of multiple acceptors, to reach consensus on the order of events submitted to the engine by multiple proposers, thereby achieving high availability.

[0102] Therefore, under Paxos, in the presence of multiple receivers, the proposed command will continue as long as a majority (i.e., most) of the multiple receivers are not faulty. However, according to one embodiment, in the special case of a single receiver node, the proposed command can continue (i.e., replace its position in the ordered sequence of protocols to be executed) without consulting the single receiver node. Thus, the proposed command can bypass the (single) receiver and proceed directly from the proposer in the deterministic state machine to the protocol with the learner. Therefore, in this case, Paxos-style consensus is unnecessary because the proposer "knows" that it has no conflict with itself.

[0103] Information about any executed command must be propagated to all learners. In the case of a single proposer node, this node both executes the command and must be aware of information about the command and its changes (e.g., from pending to executed). Therefore, this single proposer effectively acts as a persistent path (to itself) for propagating information about the executed command. In one embodiment, a persistent path for propagating information about the executed command can also be established for all learner nodes. According to one embodiment, the proposer may have an associated deterministic state machine (for which it is the sole proposer) that allows all learners to be aware of information about the executed command. Each other node also has a deterministic state machine (active when they are the sole proposer) that allows all other nodes, acting as learners, to obtain such information and change their state accordingly.

[0104] Therefore, in one embodiment, such as Figure 6 As shown, each node can have a deterministic state machine (DSM), for which it is the sole proposer, and every other node is a learner. As illustrated, node 602 has DSM 601, for which it is the sole proposer, and all other nodes 604, 606, 608, 610… are learners. Similarly, node 604 has DSM 603, for which it is the sole proposer, and all other nodes 602, 606, 608, 610… are learners. Likewise, node 606 has DSM 605, for which it is the sole proposer, and all other nodes 602, 604, 608, 610… are learners, and so on. In this context, DSM 601, 603, 605… act as pre-established and persistent message queues, which significantly reduces the amount of communication between nodes and the inherent latency in point-to-point communication between nodes.

[0105] Optimization of the list of optional writers

[0106] As described above, for each new command from the client commands, according to one embodiment, the writer list may include a preferred ordered sequence of runnable nodes that should execute the command, followed in one implementation by failed, failed, or suspected failed nodes. In this way, the preferred runnable node will be at the top of the writer list, followed by an ordered list of backup runnable nodes that will execute the command if the node at the top of the list becomes a non-running node, and then at the end of the list are known currently non-running nodes. Therefore, when a node fails or is suspected of failing, that node will be pushed to the bottom of the writer list. According to one embodiment, if all participants know the list of available nodes in advance, a separate and unique writer list can be pre-generated for each possible permutation of nodes. For example, if there are n nodes, then n! different writer lists can be pre-generated and numbered (indexed). These pre-generated and indexed writer lists can then be pre-distributed (i.e., before processing commands) to each node as a list of writer lists, such as... Figure 7 As shown in Figure 702. Figure 7 In the illustrative example shown, there are five available or potentially available nodes, and commands can be assigned to these nodes for preferred execution. There are five available nodes, such as... Figure 7 As shown, these five nodes have 120 different orders, and each of these unique node orders can form a writer list, where each writer list is identified by indices k=1 to k=120. Even with a large number of nodes, this simple writer list does not consume too much storage space on each node, nor does it prohibit the transmission of such a writer list in terms of bandwidth—especially since such propagation only needs to be performed once.

[0107] Instead of sending a list of writers with each command to be executed, detailing the preferred node (the first listed node) for executing the command, followed by backup nodes at the bottom and non-operating, failed, or suspected failed nodes, a simple index k is appended to a pre-generated and pre-distributed list of writers 702 to accompany the command to be executed. Sending index k to list 702 of writers instead of the complete list of writers significantly reduces traffic overhead and provides a readiness mechanism that specifies the preferred node or server to execute any given command. Upon receiving index k, only the node at the top of the writer list, corresponding to the received index, will execute the command. According to one embodiment, several factors may be considered when selecting index k into the list of writers, including, for example, load balancing, command dependency, geographical proximity, network conditions and network latency, as well as knowledge about which nodes are currently non-operating, failed, or suspected of failing. Other factors may also be considered when selecting index k. When the index k of the command to be executed is sent, the selected index k will correspond to a list at the bottom of which at least non-running, faulty, or suspected faulty nodes appear.

[0108] Therefore, as Figure 8 As shown, the DConE process 122 (which can be embedded in the MDS of each region) can propagate the index k of the writer list along with the GSN and the command to be executed to all nodes. The index k specifies a unique list of writers, which in turn specifies the preferred node (and the backup, lower-listed nodes) to execute the command, as shown in 802.

[0109] Figure 9This is a flowchart of a computer-implemented method according to one embodiment. As shown, block B902 requests to receive a proposal to modify data stored in a distributed and replicated file system coupled to a network. The distributed and replicated data system includes multiple servers (also referred to herein as nodes) and a metadata service configured to maintain and update copies of the namespace of the distributed and replicated file system. Block B904 can then be executed, whereby updates to the data can be coordinated by generating an ordered set of protocols corresponding to the received proposal, specifying the order in which nodes modify data stored in data nodes and cause corresponding changes to the state of the namespace. Each node can be configured to delay modifying the data and causing changes to the state of the namespace until the ordered set of protocols is received. As shown in B906, for each protocol in the generated ordered set of protocols, a corresponding list of writers can be provided or identified, the corresponding list of writers including an ordered list of nodes to perform the protocol to make the corresponding changes to the namespace. As shown in B908, the ordered set of protocols, along with the corresponding list of writers or their pre-generated index for each protocol in the ordered set, can then be sent to multiple nodes. According to one embodiment, each of the multiple nodes can be configured to execute the protocol of only the first node listed on the writer list.

[0110] According to another embodiment, the provision may include generating a list of writers for at least some of the generated ordered sets of protocols. The provision may also include selection from a plurality of pre-generated lists of writers. The writer list may include an ordered list of preferred runnable nodes toward the top of the writer list and may include a list of failed or suspected failed nodes toward the bottom of the writer list. The provision may also include providing or identifying the same list of writers for a second proposal, relative to the execution provided for a first proposal, the second proposal depending on a previous execution of the first proposal. The computer-implemented method may further include enabling the next listed node in the writer list to execute the protocol when the first listed node in the writer list has failed or is suspected of failing. A predetermined replacement node may also be capable of executing the protocol when the first listed node in the writer list has failed. Each node or server that has executed the protocol may also be configured to propagate information related to the executed protocol to each of the plurality of nodes. Propagation may also include guaranteeing the delivery of the propagated information. While a node executes a protocol, a deterministic state machine may be updated with information related to the executed protocol, the deterministic state machine being coupled to each of the other nodes and serving as a persistent messaging service among the plurality of nodes. The computer-implemented method may further include identifying failed or faulty nodes and placing them at the bottom of any generated writer list. In one embodiment, an indexed writer list may be pre-generated for each of all possible combinations of the order of multiple nodes and distributed to each of the multiple nodes. The method may then include selecting one of the indexed writer lists and sending an ordered set of protocols along with an index for each protocol in the ordered set to the multiple nodes, the index pointing to the selected list of pre-generated indexed writer lists.

[0111] Another embodiment is a network of nodes configured to implement a distributed file system. The cluster may include multiple data nodes, each configured to store data blocks of client files; multiple servers or nodes, each configured to read and / or modify data stored in the data nodes and, in response to changes to data blocks of client files, cause a corresponding update to the state of the cluster's namespace; and a distributed coordination engine embedded in a metadata service, configured to coordinate received proposals to modify data blocks by generating an ordered set of protocols corresponding to the received proposals, wherein the ordered set of protocols specifies the order in which nodes modify data stored in the data nodes and cause a corresponding change to the state of the namespace. The metadata service may also be configured to, for each protocol in the generated ordered set of protocols, provide a corresponding list of writers, including an ordered list of nodes, to execute the protocol and cause a corresponding change to the namespace, and send the ordered set of protocols along with the corresponding list of writers or their pre-generated index for each protocol in the ordered set to the multiple nodes. In this way, each of the multiple nodes may execute only the protocol of the first node listed on the writer list.

[0112] physical hardware

[0113] Figure 10 A block diagram of a computing device that can implement the embodiments is shown. Figure 10 The computing device may include a bus 1001 or other communication mechanism for communicating information, and one or more processors 1002 coupled to the bus 1001 for processing information. The computing device may also include random access memory (RAM) or other dynamic storage device 1004 (referred to as main memory), coupled to the bus 1001, for storing information and instructions to be executed by the processors 1002. The main memory (tangible and non-transitory; the term excludes signals themselves and waveforms) 1004 may also be used to store temporary variables or other intermediate information during the execution of instructions by the processors 1002. Figure 10 The computing device may also include a read-only memory (ROM) and / or other static storage device 1006 coupled to bus 1001 for storing static information and instructions for processor 1002. A data storage device 1007 (e.g., a disk and / or solid-state data storage device) may be coupled to bus 1001, the data storage device 1007 being used to store information and instructions—e.g., execution... Figures 1 to 9The computing device provides the information and instructions required for the functions shown and disclosed. The computing device can also be coupled to a display device 1021 via bus 1001, which is used to display information to the computer user. An alphanumeric input device 1022, including alphanumeric and other keys, can be coupled to bus 1001, which is used to transmit information and command selections to processor 1002. Another type of user input device is cursor control 1023, such as a mouse, trackball, or cursor arrow keys, which is used to transmit directional information and command selections to processor 1002 and to control cursor movement on display 1021. Figure 10 The computing device can be coupled to the network 1026 via a communication interface (such as a modem, network interface card or NIC) 1008.

[0114] As shown in the figure, storage device 1007 may include a direct-access data storage device, such as a disk 1030, a non-volatile semiconductor memory (EEPROM, flash memory, etc.) 1032, or a hybrid data storage device including a disk and non-volatile semiconductor memory (as shown in 1031). Reference numerals 1004, 1006, and 1007 are examples of tangible, non-transitory computer-readable media having data stored thereon representing a sequence of instructions that, when executed by one or more computing devices, implement aspects of the distributed systems and computer-implemented methods described and illustrated herein. Some of these instructions may be stored locally in a client computing device, while others may be stored remotely (and / or executed) and transmitted to the client computing device via network 1026. In other embodiments, all these instructions may be stored locally in a client or other independent computing device, while in other embodiments, all these instructions are stored and executed remotely (e.g., in one or more remote servers), and the results are transmitted to the client computing device. In yet another embodiment, the instructions (processing logic) may be stored on another form of tangible and non-transitory computer-readable medium, as shown in 1028. For example, reference numeral 1028 can be implemented as an optical disc (or some other storage technology), which can constitute a suitable data carrier to load instructions stored thereon onto one or more computing devices, thereby reconfiguring the computing devices as one or more embodiments described and illustrated herein. In other embodiments, reference numeral 1028 can be implemented as an encrypted solid-state drive. Other embodiments are also possible.

[0115] Embodiments of the present invention relate to using computing devices to implement the functions disclosed herein. According to one embodiment, the methods, apparatus, and systems described herein can be provided by one or more computing devices in response to processor 1002 executing a sequence of instructions contained in memory 1004, the sequence of instructions embodying aspects of the computer-implemented methods shown and described herein. These instructions may be read into memory 1004 from another computer-readable medium (e.g., data storage device 1007 or another (optical, magnetic, etc.) data carrier), as shown in 1028. Execution of the sequence of instructions contained in memory 1004 causes processor 1002 to perform these steps and have the functions described herein. In alternative embodiments, hardwired circuitry may be used instead of, or in combination with, software instructions to implement the described embodiments. Therefore, the embodiments are not limited to any particular combination of hardware circuitry and software. Indeed, those skilled in the art will understand that any suitable computer system can implement the functions described herein. The computing device may include one or more microprocessors for performing the desired functions. In one embodiment, instructions operable by one or more microprocessors cause the microprocessors to perform the steps described herein. The instructions may be stored in any computer-readable medium. In one embodiment, these instructions may be stored in non-volatile semiconductor memory external to the microprocessor or integrated with the microprocessor. In another embodiment, the instructions may be stored on a disk and read into volatile semiconductor memory before being executed by the microprocessor.

[0116] The foregoing detailed description describes the processes and symbolic representations of the operation of a computing device, which may include computer components, including a local processing unit, memory storage for the local processing unit, a display device, and an input device. Commands used in this disclosure may correspond to high-level instructions from client processes and may cause one or more computers to perform multiple operations. Operations may include single machine instructions. Furthermore, such processes and operations may utilize computer components in a heterogeneous distributed computing environment, including, for example, remote file servers, computer servers, and memory storage devices. These distributed computing components may access the local processing unit via a communication network.

[0117] The processing and operations performed by a computer include the manipulation of data bits by a local processing device and / or a remote server, and the maintenance of these bits within data structures residing in one or more local or remote memory storage devices. These data structures impose a physical organization on the set of data bits stored in the memory storage device and represent electromagnetic spectrum elements.

[0118] The process of a computer-implemented method, such as those described and illustrated herein, can generally be defined as a sequence of computer-executed steps that lead to a desired result. These steps typically require physical manipulation of physical quantities. Typically, though not always necessary, these quantities may take the form of electrical, magnetic, or optical signals that can be stored, transmitted, combined, compared, or otherwise manipulated. Those skilled in the art will generally refer to these signals as bits or bytes (when they have binary logic levels), pixel values, work, values, elements, symbols, characters, terms, numbers, dots, records, objects, images, files, directories, subdirectories, etc. However, it should be remembered that these terms and similar terms should be associated with the appropriate physical quantities used for computer commands, and these terms are merely conventional labels applied to physical quantities present within and during computer operation.

[0119] It should also be understood that operations within a computer are typically referred to using terms such as add, compare, move, position, place, illuminate, remove, and change. The commands described herein are machine operations that are executed in conjunction with various inputs provided by a human, AI agent operator, or user interacting with the computer. The machine used to perform the operations described herein includes local or remote general-purpose digital computers or other similar computing devices.

[0120] Furthermore, it should be noted that the programs, processes, methods, etc., described herein are not related to or limited to any particular computer or device, nor are they related to or limited to any particular communication network architecture. Instead, various types of general-purpose hardware machines can be used with program modules constructed in accordance with the teachings described herein. Similarly, it may prove advantageous to construct dedicated devices to perform the steps described herein using dedicated computer systems within a specific network architecture having hard-wired logic or programs stored in non-volatile memory (e.g., read-only memory).

[0121] While certain exemplary embodiments have been described, these embodiments are presented by way of example only and are not intended to limit the scope of this disclosure. Therefore, the foregoing description does not imply that any particular apparatus, feature, step, module, or block is necessary or indispensable. In fact, the innovative methods and systems described herein can be embodied in many other forms; furthermore, various omissions, substitutions, and changes can be made to the methods and systems described herein without departing from the spirit of the embodiments disclosed herein.

Claims

1. A computer-implemented method, comprising: The system receives proposals to modify data stored in a distributed and replicated file system coupled to a network. The distributed and replicated data system includes multiple nodes, each of which includes a server. Configure the metadata service to maintain copies of the namespace of the distributed and replicated file system; Data updates are coordinated by generating an ordered set of protocols corresponding to the received proposals. The ordered set of protocols specifies the order in which nodes modify the data stored in the data nodes and cause changes to the state of the namespace. Each node is configured to delay modifying the data stored in the data nodes and causing changes to the state of the namespace until the ordered set of protocols is received. For each protocol in the generated ordered protocol set, an index is provided or identified for the corresponding writer list, which includes an ordered list of nodes to execute the protocol and make corresponding changes to the namespace. Each writer list includes a preferred running node toward a first end of the writer list and an ordered list of failed or suspected failed nodes toward a second end of the writer list. An ordered protocol set, along with an index of the corresponding list of writers or the identifier of the corresponding list of writers for each protocol in the ordered protocol set, is sent to multiple nodes, each of which is configured to execute only the protocol of the first node listed on the provided or identified index of the corresponding list of writers. When a previously listed node in the writer list has failed or is suspected of failing, the next listed node in the writer list can execute the protocol. and This enables the metadata service to update copies of the namespace it maintains.

2. The computer-implemented method according to claim 1, wherein, Provides a list of generated writers for at least some of the generated ordered protocol sets.

3. The computer-implemented method according to claim 1, wherein, It offers the option to select from a list of multiple pre-generated writers.

4. The computer-implemented method according to claim 1, wherein, The provision also includes providing the same list of writers for the second proposal as for the first proposal after the first proposal is executed.

5. The computer-implemented method of claim 1 further includes a predetermined replacement node capable of executing the protocol when the first listed node in the writer list fails.

6. The computer-implemented method of claim 1 further includes propagating information related to the executed protocol from each node that has executed the protocol to each of the plurality of nodes.

7. The computer-implemented method according to claim 6, wherein, Communication also includes ensuring the delivery of the information being communicated.

8. The computer-implemented method of claim 1 further includes updating a deterministic state machine with information related to the protocol being executed when the node executes the protocol, the deterministic state machine being coupled to each of the other nodes and used as a persistent message service among the multiple nodes.

9. The computer-implemented method of claim 1 further includes identifying faulty or failed nodes and placing the faulty or failed nodes at the bottom of any generated writer list.

10. The computer-implemented method according to claim 1, further comprising: For each of all possible combinations of the order of multiple nodes, pre-generate a list of writers for the index; The list of writers for the pre-generated index is distributed to each of the multiple nodes. This provides a list of writers, including the option to select one of the indexes, and The sending process involves sending an ordered set of protocols along with an index for each protocol in the ordered set to multiple nodes, where the index points to a selected list of writers from a pre-generated list of indexes.

11. A network of servers configured to implement a distributed file system, the distributed file system comprising: Multiple data nodes, each configured to store data blocks of client files; Multiple nodes, each node including a server; Metadata service, configured to maintain and update the state of the namespace of the distributed file system in response to changes in data blocks of client files; The distributed coordination engine, embedded in the metadata service, is configured to coordinate received proposals to update data blocks by generating an ordered set of protocols corresponding to the received proposals. The ordered set of protocols specifies the order in which nodes modify the data stored in the data nodes. The metadata service is also configured to provide or generate an index to a corresponding writer list for each protocol in the generated ordered protocol set. The corresponding writer list includes an ordered list of nodes that execute the protocol and cause corresponding changes to the namespace. Each writer list includes an ordered list of preferred running nodes toward the first end of the writer list and an ordered list of failed or suspected failed nodes toward the second end of the writer list. Furthermore, the metadata service is configured to send an ordered protocol set along with a list of writers or their pre-generated index for each protocol in the ordered protocol set to multiple nodes, such that each of the multiple nodes can only execute the protocol of the first node listed on the writer list, and when a previously listed node in the writer list has failed or is suspected of failing, the next listed node in the writer list is enabled to execute the protocol.

12. The network of the server according to claim 11, wherein, The metadata service is also configured to generate a list of writers for at least some of the generated ordered protocol sets.

13. The network of the server according to claim 11, wherein, The metadata service is also configured to select from a list of multiple pre-generated writers.

14. The network of the server according to claim 11, wherein, The metadata service is also configured to provide the same list of writers for the second proposal as for the first proposal after the first proposal is executed.

15. The network of the server according to claim 11, wherein, The metadata service is also configured so that a pre-selected replacement node can execute the protocol when the first node listed in the writer list fails.

16. The network of the server according to claim 11, wherein, Each node that has executed the protocol is also configured to propagate information related to the executed protocol to each of the multiple nodes.

17. The network of the server according to claim 11, wherein, When executing a protocol, nodes are also configured to update a deterministic state machine with information related to the protocol being executed. The deterministic state machine is coupled to each of the other nodes and serves as a persistent message service between multiple nodes.

18. The network of the server according to claim 11, wherein, The metadata service is also configured to detect failed or faulty nodes and place them at the bottom of any generated list of writers.

19. The network of the server according to claim 11, wherein, The metadata service is also configured as follows: For each of all possible combinations of the order of multiple nodes, pre-generate a list of writers for the index; Distribute the list of writers for the pre-generated index to each of the multiple nodes, and The ordered protocol set, along with an index for each protocol in the ordered protocol set, is sent to multiple nodes, with the index pointing to a selected list of writers in the pre-generated index.

Images