Extended distributed storage system

Abstract

An extended distributed storage system is provided. The plurality of computing devices are communicatively coupled to each other via a network, and each of the plurality of computing devices is operably coupled to one or more of the plurality of storage devices. A plurality of faulty resilient strips are distributed between the plurality of storage devices such that each faulty resilient strip of the plurality of faulty resilient strips spans the plurality of storage devices. The plurality of computing devices maintain each failing resilient strip using the buckets. These buckets are operably split and transferred between computing devices to balance available computing power and storage access.

Description

[0001] This application is a divisional application of the Chinese application with application number 201980040816.8, application date June 4, 2019, and invention name “Extended Distributed Storage System”, all contents of which are incorporated herein by reference.

[0002] Priority claim

[0003] This application claims priority to U.S. Provisional Patent Application No. 62 / 686,956, filed on June 19, 2018, entitled “Expanding a Distributed Storage System,” and U.S. Patent Application No. 16 / 274,600, filed on February 13, 2019, entitled “Expanding a Distributed Storage System.” Background Art

[0004] The limitations and disadvantages of conventional methods of data storage will become apparent to those skilled in the art by comparing the conventional methods of data storage with some aspects of the present methods and systems set forth in the remainder of this disclosure with reference to the accompanying drawings.

[0005] Incorporated by Reference

[0006] US Patent Application No. 15 / 243,519, entitled "Distributed Erasure Coded Virtual Filesystem," is hereby incorporated by reference in its entirety. Summary of the invention

[0007] As more fully set forth in the claims, there is provided a method and system for extending a distributed file system substantially as shown and / or described in conjunction with at least one of the figures. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 Various exemplary configurations of distributed file systems according to aspects of the present disclosure are shown.

[0009] Figure 2 An exemplary configuration of a distributed file system node according to aspects of the present disclosure is shown.

[0010] Figure 3 Another representation of a distributed file system according to an exemplary implementation of the present disclosure is shown.

[0011] Figure 4 An embodiment of extending processing capability in a distributed file system according to an exemplary implementation of the present disclosure is shown.

[0012] Figure 5 An embodiment of extending storage capacity in a distributed file system according to an exemplary implementation of the present disclosure is shown.

[0013] Figure 6 is a flow chart illustrating an exemplary method for expanding processing capabilities in a distributed file system according to an exemplary implementation of the present disclosure.

[0014] Figure 7 is a flow chart illustrating an exemplary method for extending storage capacity in a distributed file system according to an exemplary implementation of the present disclosure.

[0015] Figure 8 An exemplary implementation is shown in which two distributed failure-resilient address spaces reside on multiple solid-state storage disks.

[0016] Fig. 9 A forward error correction scheme that can be used to protect data stored to a non-volatile memory of a virtual file system according to an exemplary implementation of the present disclosure is shown. DETAILED DESCRIPTION

[0017] Traditionally, file systems use centralized control over metadata structures (e.g., directories, files, attributes, file contents). If a local file system is accessible from a single server and the server fails, the file system's data may be lost because there is no further protection. To increase protection, some file systems (e.g., such as provided by NetApp) have used one or more pairs of controllers in an active-passive manner to replicate metadata across two or more computers. Other solutions have used multiple metadata servers in a clustered manner (e.g., such as provided by IBM GPFS, Dell EMC Isilon, Lustre, etc.). However, because the number of metadata servers in a traditional cluster system is limited to a small number, such a system cannot be expanded.

[0018] The system in the present disclosure is applicable to small clusters and can also be expanded to multiple, thousands of nodes. Exemplary embodiments of non-volatile memory (NVM) (e.g., flash memory in the form of a solid-state drive (SSD)) are discussed. NVM can be divided into 4kB blocks and 128MB blocks. Extensions can be stored in volatile memory, such as RAM for fast access, also backed up by NVM storage devices. Extensions can store pointers to blocks (e.g., 256 pointers to 1MB of data stored in blocks). In other embodiments, larger or smaller memory partitions can also be used. The metadata functions in the present disclosure can be effectively distributed on multiple servers. For example, in the case where the target of a large load is a "hot spot" of a specific part of the file system namespace, the load can be distributed across multiple nodes.

[0019] Figure 1 Various exemplary configurations of distributed file systems according to aspects of the present disclosure are shown. Figure 1 1 is a local area network (LAN) 102 including one or more nodes 120 (indexed by integers from 1 to J, j≥1), and optionally (indicated by dashed lines): one or more dedicated storage nodes 106 (indexed by integers from 1 to M, M≥1), one or more compute nodes 104 (indexed by integers from 1 to N, N≥1), and / or an edge router connecting the LAN 102 to a remote network 118. Optionally, the remote network 118 includes one or more storage services 114 (indexed by integers from 1 to K, K≥1), and / or one or more dedicated storage nodes 115 (indexed by integers from 1 to L, L≥1).

[0020] 120 per node j (j is an integer, where 1≤j≤J) is included for running directly located in the device 104 n The operating system and / or device 104 n A networked computing device (e.g., a server, a personal computer, etc.) that has circuits for executing processes (e.g., client processes) in one or more virtual machines.

[0021] Compute node 104 is a networking device that runs a virtual front end without a virtual back end. Compute node 104 can run a virtual front end by incorporating single root input / output virtualization (SR-IOV) into a network interface card (NIC) and consuming a full processor core. Alternatively, compute node 104 can route the network through the Linux kernel network stack and use kernel process scheduling to run the virtual front end, whereby there is no requirement for a full core. This is useful if the user does not want to allocate a full core for the file system or if the network hardware is incompatible with the file system requirements.

[0022] Figure 2 An exemplary configuration of a node according to various aspects of the present disclosure is shown. The node includes a front end 202 and a driver 208, a memory controller 204, a back end 206, and an SSD agent 214. The front end 202 can be a virtual front end; the memory controller 204 can be a virtual storage controller; the back end 106 can be a virtual back end; and the driver 208 can be a virtual driver. As used in the present disclosure, a virtual file system (VFS) process is a process that implements one or more of the following: the front end 202, the memory controller 204, the back end 206, and the SSD agent 214. Thus, in an exemplary implementation, resources (e.g., processing and memory resources) of a node can be shared between client processes and VFS processes. The VFS process can be configured to require a relatively small amount of resources to minimize the impact on the performance of client applications. The front end 202, the memory controller 204, and / or the back end 206, and / or the SSD agent 214 can run on a processor of the host 201 or a processor of the network adapter 218. For a multi-core processor, different VFS processes can run on different cores and can run different subsets of services. As far as the client process 212 is concerned, the interface with the virtual file system is independent of the specific physical machine on which the VFS process runs. The client process only needs to provide a driver 208 and a front end 202 to serve it.

[0023] The node can be implemented as a single-tenant server (e.g., bare metal) running directly on the operating system or a virtual machine (VM) and / or container (e.g., Linux container (LXC)) within a bare metal server. The VFS can run as a VM environment within an LXC container. Thus, within the VM, the only thing that can run is the LXC container that includes the VFS. In a traditional bare metal environment, there are user space applications and the VFS runs in an LXC container. If the server runs other containerized applications, the VFS can run in an LXC container that is outside the management scope of the container deployment environment (e.g., Docker).

[0024] The node can be served by an operating system and / or a virtual machine monitor (VMM) (e.g., a hypervisor). The VMM can be used to create a node on a host 201 and run the node. Multiple kernels can reside in a single LXC container running a VFS, and the VFS can run on a single host 201 using a single Linux kernel. Therefore, a single host 201 can include multiple front ends 202, multiple memory controllers 204, multiple back ends 206, and / or one or more drivers 208. The driver 208 can run in a kernel space outside the scope of the LXC container.

[0025] The network stack 210 can be run in user space 222 using SR-IOV PCIe virtual functions. SR-IOV allows isolation of PCI Express (PCI Express) so that a single physical PCI Express can be shared on a virtual environment and different virtual functions can be provided to different virtual components on a single physical service machine. The I / O stack 210 enables the VFS node to bypass the standard TCP / IP stack 220 and communicate directly with the network adapter 218. The VFS functions of the Portable Operating System Interface for uniX (POSIX) can be provided to the VFS driver 208 through a lock-free queue. Non-volatile memory express (NVMe) drivers 214 can also be run in user space 222 using SR-IOV or full PCIe physical function addresses, thereby completely bypassing the Linux IO stack. Non-volatile storage devices 216 attached via a PCI Express (PCIe) bus can be accessed using NVMe. For example, the non-volatile storage device 220 can be a flash memory in the form of a solid-state drive (SSD), or a storage class memory (SCM) in the form of an SSD or a storage module (DIMM). Other embodiments may include storage class memory technologies such as 3D-Xpoint.

[0026] By coupling the physical SSD 216 with the SSD agent 214 and the network 210, the SSD can be implemented as a networked device. Alternatively, by using a network protocol such as NVMe-oF (NVMe over Fabrics), the SSD can be implemented as a network-attached NVMe SSD 242 or 244. NVMe-oF can allow access to NVMe devices using redundant network links, thereby providing a higher level or resilience. Network adapters 226, 228, 230, and 232 can include hardware accelerators for connecting to NVMe SSDs 242 and 244 to convert the network adapter into a networked NVMe-oF device without using a server. NVMe SSDs 242 and 244 can each include two physical ports, and all data can be accessed through any of these ports.

[0027] Each client process / application 212 can run directly on the operating system or can run in a virtual machine and / or container served by the operating system and / or hypervisor. The client process 212 can read data from the memory and / or write data to the memory in the process of performing its main function. However, the main function of the client process 212 is independent of the memory (that is, the process only involves reliably storing its data and can be retrieved when needed, and does not involve where, when, or how to store data). Exemplary applications that generate this process include: email servers, web servers, office productivity applications, customer relationship management (CRM), animation video rendering, genomics computing, chip design, software construction, and enterprise resource planning (ERP).

[0028] The client application 212 can make a system call to the kernel 224 that communicates with the VFS driver 208. The VFS driver 208 places the corresponding request on the queue of the VFS front end 202. If there are several VFS front ends, the driver can load balanced access to different front ends. Thereby ensuring that a single file / directory is always accessed via the same front end. This operation can be completed by sharing the front end based on the ID of the file or directory. The VFS front end 202 provides an interface for routing file system requests to the appropriate VFS back end based on the storage bucket responsible for the operation. The appropriate VFS back end can be located on the same host or it can be located on another host.

[0029] The VFS backend 206 hosts several buckets, each of which services file system requests, receiving and performing tasks to manage the virtual file system in other ways (e.g., load balancing, logging, maintaining metadata, caching, data movement between tiers, removing stale data, correcting corrupted data, etc.).

[0030] The VFS SSD agent 214 handles the interaction with the corresponding storage device 216. For example, this may include translating addresses and generating commands that are issued to the storage device (e.g., via SATA, SAS, PCIe, or other suitable bus). Thus, the VFS SSD agent 214 operates as an intermediary between the storage device 216 and the VFS backend 206 of the virtual file system. The SSD agent 214 can also communicate with standard network storage devices that support standard protocols such as NVMe-oF (NVMe over Fabrics).

[0031] Figure 3 Another representation of a distributed file system according to an exemplary implementation of the present disclosure is shown. Figure 2 described in Figure 3, element 302 represents the memory resources (e.g., DRAM and / or other short-term memory) and processing (e.g., x86 processors, ARM processors, NICs, ASICs, FPGAs, etc.) of the various nodes (compute, storage, and / or VFS) on which the virtual file system resides. Element 308 represents one or more physical storage devices 216 that provide long-term storage for the virtual file system.

[0032] like Figure 3 As shown in , the physical storage device is organized into multiple distributed failure resilient address spaces (DFRAS) 318. Each failure resilient address space includes multiple blocks 310, which in turn include multiple blocks 312. In some implementations, the blocks 312 are organized into blocks 310 for convenience only, and it is not possible to implement this operation in all implementations. Each block 312 stores committed data 316 (which may be in various states discussed below) and / or metadata 314 that describes or references the committed data 316.

[0033] Storage device 308 is organized into multiple DFRAS to enable high performance parallel submission (e.g., Figure 1 All nodes 1041–104 in N , 1061–106 M , and 1201–120 J Concurrent commits may be performed in parallel.) In an exemplary implementation, each node of the virtual file system may own corresponding one or more DFRASs of the plurality of DFRASs and have exclusive read / commit access to the DFRASs it owns.

[0034] Each bucket has a DFRAS, and therefore, when writing to it, no coordination with any other node is required. Each bucket can be striped across multiple different blocks on multiple different SSDs, and therefore, each bucket with its DFRAS can choose which "block stripe" to currently write to based on multiple parameters, and once a block is assigned to that bucket, no coordination is required to achieve this. All buckets can effectively write to all SSDs without any coordination.

[0035] Each DFRAS owned and accessed only by its owner's bucket running on a particular node allows each node of the VFS to control a portion of storage 308 without the need to cooperate with any other node (except during initialization or during the (reallocation) of buckets holding DFRAS after a node failure, which can be performed asynchronously, for example, to actually read / commit to storage 308). Thus, in this implementation, each node can read / commit to its bucket's DFRAS independently of the operation of other nodes, without the need to reach any consensus when reading and committing to storage 308. Further, in the event of a particular node's failure, the fact that a particular node owns multiple buckets allows its workload to be more intelligently and efficiently redistributed to other nodes (rather than having to distribute the entire workload to a single node, thereby creating a "hotspot"). In view of this, in some implementations, the number of buckets may be large relative to the number of nodes in the system, so that any one bucket may be a relatively small load placed on another node. This allows fine-grained redistribution of the load of failed nodes based on the power and capacity of other nodes (e.g. nodes with greater power and capacity may be given a higher percentage of failed node buckets)

[0036] To allow this, metadata may be maintained that maps each bucket to its current owning node so that reads and commits to storage 308 may be redirected to the appropriate node.

[0037] Load distribution is possible because the entire file system metadata space (e.g., directories, file attributes, content ranges of files, etc.) can be split (e.g., cut or sliced) into small, uniform pieces (e.g., fragments). For example, a large system with 30k servers can cut the metadata space into 128k or 256k fragments.

[0038] Each of these metadata shards can be maintained in a "bucket". Each VFS node can be responsible for several buckets. A bucket is considered "active" or a "leader" in a bucket when it serves metadata shards on a given backend. Typically, there are more buckets than VFS nodes. For example, a small system with 6 nodes can have 120 buckets, and a large system with 1000 nodes can have 8k buckets.

[0039] Each bucket can be active in a small set of nodes (usually, the 5 nodes that make up the 5-tuple for that bucket). The cluster configuration keeps all participating nodes up to date with the allocations for each bucket's 5-tuple.

[0040] Each quintuple monitors itself. For example, if the cluster has 10k servers and each server has 6 buckets, each server only needs to talk to 30 different servers to maintain the state of its bucket (6 buckets have 6 quintuples, so 6*5=30). This is a much smaller number if a central entity has to monitor all nodes and maintain cluster-wide state. As the cluster size increases, the use of quintuples allows performance to scale in larger clusters because the nodes do not perform more work. This can have disadvantages: in "dumb" mode, small clusters actually generate more communication than if there were physical nodes, but this disadvantage is overcome by sending only a single heartbeat between two servers with all the buckets they share (as the cluster grows, this will change to only one bucket, but if you have a small 5-server cluster, the cluster only includes all buckets in all messages and each server only talks to 4 other servers). The quintuple can make decisions (i.e., reach consensus) using an algorithm similar to the Raft consensus algorithm.

[0041] Each bucket can have a set of compute nodes that can run on it. For example, five VFS nodes can run a bucket. However, at any given moment, only one node in the group is the controller / leader. Further, for sufficiently large clusters, no two buckets share the same group. If there are only 5 or 6 nodes in the cluster, most buckets may share a backend. In fairly large clusters, there are multiple distinct groups of nodes. For example, for 26 nodes, there are more than possible five-node groups (ie, quintuples).

[0042] All nodes in the group learn and agree (i.e., reach consensus) which node is the actual active controller (i.e., leader) for the bucket. Nodes accessing a bucket can remember ("cache") the last node that was the leader of the bucket among the group members (e.g., five). If the node accesses the bucket leader, the bucket leader performs the requested operation. If the node accesses a node that is not the current leader, the node instructs the leader to "redirect" the access. If there is a timeout for accessing the cached leader node, the contacted node can try a different node in the same five-tuple. All nodes in the cluster share a common "configuration" of the cluster to allow nodes to know which server can run each bucket.

[0043] Each bucket may have a load / usage value that indicates the extent to which the bucket is being used by applications running on the file system. For example, a server node with 11 lightly used buckets may receive another bucket of metadata to run before a server with 9 heavily used buckets, even though there will be an imbalance in the number of buckets used. The load value may be determined based on average response latency, the number of concurrently running operations, memory consumed, or other metrics.

[0044] Even if the VFS node does not fail, redistribution can occur. If the system identifies that one node is busier than other nodes based on the tracked load metrics, the system can move one of its buckets to (i.e., "failover") another server that is less busy. However, before the buckets are actually relocated to different hosts, load balancing can be achieved by transferring writes and reads. Because each write can end on a different node group determined by DFRAS, nodes with higher loads may not be selected in the stripe where data is being written. The system can also choose not to read from highly loaded nodes. For example, a "degraded mode read" can be performed, in which blocks in highly loaded nodes are reconstructed from other blocks in the same stripe. A reduced mode read is a read performed via the remaining nodes in the same stripe, and the data is reconstructed via fault protection. When the read latency is too high, a degraded mode read can be performed because the initiator of the read can assume that the node has stopped operating. If the load is high enough to establish a higher read latency, the cluster can revert to reading the data from other nodes and using a degraded mode read to reconstruct the required data.

[0045] Each bucket manages its own distributed erasure coding instance (i.e., DFRAS 518) and does not need to coordinate with other buckets to perform read or write operations. For each different bucket, there are potentially thousands of concurrent, distributed erasure coding instances working concurrently. This forms an integral part of performance scaling because it effectively allows arbitrarily large file systems to be split into independent fragments that do not need to coordinate, thereby providing high performance regardless of scale.

[0046] Each bucket handles all file system operations that fall into its fragment. For example, directory structures, file attributes, and file data ranges will fall within the jurisdiction of a specific bucket.

[0047] An operation done from any frontend begins by finding out which bucket owns the operation. Then, the backend leader and node for the bucket is determined. This determination can be performed by trying the last known leader. If the last known leader is not the current leader, the node can learn which node is the current leader. If the last known leader is no longer part of the bucket's quintuple, the backend will let the frontend know that it should return the configuration to find out the members of the bucket's quintuple. The distribution of operations allows complex operations to be processed by multiple servers rather than by a single computer in a standard system.

[0048] If the cluster size is small (e.g., 5) and quintuples are used, there are buckets that share the same group. As the cluster size grows, the buckets are redistributed so that no two groups are the same.

[0049] The storage system can be expanded by adding more computing power and by adding more storage capacity. More computing power supports more metadata operations and IOPS. When expanding computing power, users can choose to add new servers with more cores or can decide to allocate more cores running on current servers.

[0050] Figure 4 An embodiment of the extended processing capability of a distributed file system according to an exemplary implementation of the present disclosure is shown. The distributed file system initially includes a computing device 401 (e.g., a server) and a plurality of storage devices 409a, 409b, 409c, and 409d. The computing device 401 includes a first front end 403 and a first back end 405. The first back end 405 includes at least one storage bucket 407. The plurality of storage devices 409a, 409b, 409c, and 409d may include a plurality of solid-state devices. The plurality of storage devices 409a, 409b, 409c, and 409d may be configured into a plurality of blocks, for example, block a1, block a2, block a3, block b1, block b2, and block b3.

[0051] Each bucket in the backend is operable to establish a fault-resilient stripe comprising a plurality of blocks. For example, for 10 blocks, 8 data blocks may be protected by 2 error protection / correction blocks (i.e., using 8+2 stripes). Similarly, for 10 fault domains, 6 data blocks may be protected by 4 error protection / correction blocks (i.e., using 6+4 stripes).

[0052] To illustrate, in Figure 44 storage devices and 2 stripes are described in FIG. 4 . A plurality of different storage devices and stripes may be used without departing from the present disclosure. Bucket 407 is operable to establish a plurality of fault resilient stripes 411a and 411b. A first fault resilient stripe 411a includes a block a1, a block a2, and a block a3. A second fault resilient stripe 411b includes a block b1, a block b2, and a block b3. Each of the plurality of storage blocks in a particular fault resilient stripe may be located in a different storage device of the plurality of storage devices.

[0053] Bucket load balancing allows the cluster to be expanded. When processing resources (e.g., computing resources 421) are added to the file system, a new front end 423 and a new back end 425 may be generated. If the first back end 405 includes at least two buckets, the first computing device may be operable to transfer a bucket of the at least two buckets to the second back end 425 of the second computing device 421. If there are more buckets than back ends, the existing buckets may be moved to the new back end.

[0054] like Figure 4 As shown in , the first backend 405 initially includes only one storage bucket 407. If the load on the file system reaches a state where there is only one storage bucket that becomes a hot spot, the storage bucket can be split. The splitting method of the storage bucket begins by splitting the metadata space managed by the storage bucket into two halves (or four equal parts, or eight equal parts), splitting the registry into two halves (or four equal parts or eight equal parts), and splitting all the on-disk hashing stored in the registry into two halves (or four equal parts or eight equal parts). The calculation can be performed in RAM. Splitting into two halves or smaller fragments (four equal parts, or eight equal parts) may be equally difficult, so the file system can perform a higher level of splitting in case it has to be superimposed on the split. After the storage bucket is split, the new storage bucket can be balanced and loaded to the new server, while ensuring that each server maintains approximately the same amount of computing load.

[0055] When the new server requires additional storage buckets, the computing device 401 is operable to split the storage bucket 407 on the first backend 405 into a first split storage bucket 407a and a second split storage bucket 407b. The first split storage bucket 407a can be retained in the first backend 405. The second split storage bucket 407b can be transferred to a new backend 425 of the second computing device 421. Distributed among the first split storage bucket 407a and the second split storage bucket 407b are a plurality of failure resilient stripes 411a and 411b associated with the storage bucket 407 on the first backend 405.

[0056] Each bucket 407a and 407b is operable to construct a new failure resilient stripe based on the available capacity of the plurality of storage devices 409a, 409b, 409c, and 409d. The most recently added storage devices in the plurality of storage devices are typically the least utilized and are therefore selected for the new failure resilient stripe.

[0057] Figure 5 An embodiment of extending storage capacity in a distributed file system according to an exemplary implementation of the present disclosure is shown. Figure 5 The file system in initially includes a first computing device 501 and two storage devices 509a and 509b. The first computing device 501 includes a front end 503 and a back end 505. The back end 505 includes a storage bucket 507 operable to establish a failure resilient stripe 511a.

[0058] Afterwards, two additional storage devices 509c and 509d are added to Figure 5 These recently added storage devices 509c and 509d of the plurality of storage devices are typically the least utilized and are therefore selected for the new failure resilient stripe 511b. The file system determines / selects the blocks of the storage devices based on the occupancy of the storage devices.

[0059] When a new SSD is added and configured in its fault domain, the system can use the new SSD for data placement. The SSD can be added to the current server or a new server. The new server also has increased computing power.

[0060] Once the system adds an SSD, it can start using that SSD for new writes. Since most reads come from the current device, the system directs new writes to the new device to distribute access across all devices.

[0061] When old files are deleted or data is overridden, the old capacity is released. The new SSD will also be accessible for read requests. Once this happens, the system's automatic load balancing begins to shift to writing back to the old SSD.

[0062] A plurality of fault-resilient stripes may be constructed using storage buckets on a computing device, such that each of the plurality of fault-resilient stripes includes a plurality of storage blocks and each of the storage blocks is located in a different storage device.

[0063] Figure 66 is a flow chart illustrating an exemplary method of expanding processing power in a distributed file system according to an exemplary implementation of the present disclosure. In box 601, more computing power can be added by adding new processors and / or servers and associated backends. In box 603, the number of buckets is compared to the number of backends. If the number of buckets exceeds the number of backends, the existing buckets can be transferred to the new backend in box 607. For example, if the existing computing device backend includes at least two buckets, one of the buckets can be transferred to a new backend of the new computing device.

[0064] However, if the number of buckets does not exceed the number of backends, one or more existing buckets may be split into two or more new buckets in block 605. In block 607, once the number of buckets resulting from the split exceeds the number of backends, the buckets may be moved to the new backends.

[0065] Figure 7 is a flow chart illustrating an exemplary method of expanding storage capacity in a distributed file system according to an exemplary implementation of the present disclosure.

[0066] In block 701, more storage capacity is added to the file system by adding a new SSD. When building a stripe, different storage devices are selected based on the available capacity of each of the multiple storage devices. In block 703, the availability of the SSD is evaluated. If the previous SSD is available, then in block 705, the write can be distributed to all (newly added and old) SSDs. When the storage device is not available, then in block 707, the write is directed to one or more newly added SSDs.

[0067] Figure 8 An exemplary implementation is shown in which two distributed failure-resilient address spaces reside on multiple solid-state storage disks.

[0068] Block 910 can be 1,1 –910 D,C Organized into multiple block stripes 9201–920 S (S is an integer). In an exemplary implementation, each block stripe 920 is corrected using forward error correction (eg, erasure coding). s (s is an integer, where 1≤s≤S) is independently protected. Thus, any specific block stripe 920 can be determined based on the desired level of data protection. s Block 910 in d,c The number of

[0069] For the purpose of illustration, assume that each block stripe is 920 sThe number of blocks 910 is N=M+K (where each of N, M, and K is an integer). d,c , then N blocks 910 d,c The M blocks in the memory can store data digits (usually binary digits or "bits" in current storage devices) and the N blocks 910 d,c The K blocks in may store protection numbers (again, typically bits). The virtual file system may then store N blocks 910 from N different failure domains. d,c Assign to each stripe 920 s .

[0070] As used herein, a "fault domain" refers to a group of components where the failure of any single component (a component loses power, becomes unresponsive, etc.) can cause all components to fail. For example, if a rack has a single top-of-rack switch, then the failure of the single top-of-rack switch will cause the connection to all components (e.g., compute, storage, and / or VFS nodes) on the rack to be disconnected. Thus, for the rest of the system, this is equivalent to all components on the rack failing together. A virtual file system according to the present disclosure may include fewer fault domains than block 910.

[0071] In an exemplary implementation in which nodes of a virtual file system are connected and powered in a fully redundant manner, each node has only a single storage device 906, and the failure domain may be only the single storage device 906. Thus, in the exemplary implementation, each block stripe 920 s Including residing in N storage devices 9061–906 D (thus, D is greater than or equal to N) a plurality of blocks on each storage device 910 d,c . Fig. 9 An embodiment of this implementation is shown in FIG.

[0072] exist Figure 8 In the example, D=7, N=5, M=4, K=1, and the storage is organized into two DFRAS. These numbers are for illustration only and are not intended to be limiting. For the sake of illustration, these block stripes of the first DFRAS are arbitrarily called 920. The first block stripe 9201 consists of blocks 910 1,1 ,910 2,2 ,910 3,3 ,910 4,5 , and 910 5,6 The second block stripe 9202 is composed of block 910 3,2 ,910 4,3 ,910 5,3 ,910 6,2 , and 910 7,3and the third block stripe 9203 is composed of block 910 1,4 ,910 2,4 ,910 3,5 ,910 5,7 , and 910 7,5 constitute.

[0073] Despite Figure 8 In a simple embodiment of the present invention, D=7 and N=5, however, in actual implementations, D can be much larger than N (e.g., a multiple of an integer greater than 1, and can be as high as multiple orders of magnitude) and the two values ​​can be selected so that the probability of any two block stripes 920 of a single DFRAS residing on the same set of N storage devices 906 (or more generally, on the same set of N failure domains) is below a desired threshold. In this way, any single storage device 906 d (or more generally, any single failure domain) will result in the loss of any specific stripe 920 s At most one block 910 b,c (The expected statistics will likely be determined based on the chosen values ​​of D and N, the size of the N storage devices 906, and the arrangement of the failure domains.) Even further, a double failure will result in most stripes losing at most a single block 910. b,c And only a small number of stripes (determined based on the values ​​of D and N) lose two blocks in any particular stripe 920s (eg, the number of two failed stripes may be exponentially smaller than the number of one failed stripe).

[0074] For example, if each storage device 906 d is 1 TB, and each block is 128 MB, then storage device 906 d The failure of 7812 (=1TB / 128MB) block stripes 920 will cause the loss of one block 910 (the expected statistics are likely to be determined based on the following items: the selected values ​​of D and N, the size of the N storage devices 906, and the arrangement of the failure domains). s , appropriate forward error correction algorithms and block stripes 920 can be used s The other N-1 blocks in the block are used to correct the lost block 910 d,c In addition, because the affected 7812 blocks are striped across all storage devices 9061–906 D Evenly distributed, so for the 7812 lost blocks 910 d,c Performing a rebuild involves (expected statistics are likely to be determined based on the selected values ​​of D and N, the size of the N storage devices 906, and the arrangement of the failure domains) DReading the same amount of data (i.e., the burden of rebuilding the lost data) is across all storage devices 9061–906 D Evenly distributed to provide very fast recovery from failures).

[0075] Next, turn to storage device 9061–906 D In the case of concurrent failures of two fault domains (or more generally, concurrent failures of two fault domains), since each DFRAS has block stripes 9201–920 S On all storage devices 9061–906 D Evenly distributed, so only a very small number of block strips 9201–920 S Loss of two of its N blocks. The virtual file system may operate based on indicating block stripes 9201-920 S With storage device 9061–906 D The two lost block stripes can be quickly identified by using the metadata of the mapping between the two lost block stripes. Once the two lost block stripes are identified, the virtual file system can prioritize the reconstruction of the two lost block stripes before starting the reconstruction of one lost block stripe. The remaining block stripes have only a single lost block, and for them (most of the affected block stripes), the two storage devices 906 d Concurrent failure with only one storage device 906 d A similar principle applies to the third concurrent failure (in the case of two concurrent failures, the number of block stripes with three failed blocks is even less than the number of block stripes with two failed blocks), and so on. In an exemplary implementation, based on block stripe 920 s The number of losses in can control the execution block stripe 920 s For example, this operation can be achieved by controlling the rate at which the reconstruction is read and submitted, the rate at which the FEC calculation for the reconstruction is performed, the rate at which the network message for the reconstruction is transmitted, etc.

[0076] Fig. 9 FIG. 1 shows a forward error correction scheme that can be used to protect data stored in a non-volatile memory of a virtual file system according to an exemplary implementation of the present disclosure. FIG. 1 shows a storage block 902 of a block stripe 9301-9304 of a DFRAS. 1,1 –902 7,7 .exist Fig. 9 In the protection scheme of , five blocks of each stripe are used to store data digits and two blocks of each stripe are used to store data of the protection digits (i.e., M=5 and K=2). Fig. 9 The protection figures are calculated using the following formulas (1)-(9):

[0077]

[0078]

[0079] thus, Fig. 9 The four stripes 9301-9304 in are part of a multi-strip (in this case, four stripes) FEC protection domain and various combinations of the above formulas (1)-(9) can be used to recover from the loss of any two or fewer blocks in any of the block stripes 9301-9304. For comparison, an embodiment of a single-strip protection domain is as D11, D22, D33, D44, D54 are protected only by P1 and D11, and D22, D33, D44, D54, and P1 are all written to stripe 9301 (9301 is a single-strip FEC protection domain).

[0080] According to an exemplary implementation of the present disclosure, a plurality of computing devices are communicatively coupled to each other via a network, and each computing device in the plurality of computing devices includes one or more storage devices in a plurality of storage devices. A plurality of failure resilient address spaces are distributed across the plurality of storage devices, so that each of the plurality of failure resilient address spaces spans across the plurality of storage devices. Each of the plurality of failure resilient address spaces is organized into a plurality of stripes (e.g., Figure 8 and Fig. 9 Each one or more of the plurality of stripes is a plurality of forward error correction (FEC) protection domains (e.g., such as Figure 8 Each stripe in the plurality of stripes may include a plurality of storage blocks (e.g., a plurality of 912). Each block of a particular stripe in the plurality of stripes may reside on a different storage device of the plurality of storage devices. A first portion of the plurality of storage blocks (e.g., Fig. 9 Strip 9301 of 902 1,2 –902 1,6 The number of five) can be used to store data digits, and the second part of the plurality of memory blocks (eg, Fig. 9 Two 902 in the strip 9301 1,1 and 902 1,7 A number of ) can be used to store a protection number calculated at least in part based on the data number.

[0081] A plurality of computing devices may be operated to sort a plurality of stripes. The sorting may be used to select which of the plurality of stripes is used for the next commit operation of a fault-resilient address space in a plurality of fault-resilient address spaces. The sorting may be based on how many protected and / or unprotected storage blocks exist in each of the plurality of stripes. For any particular stripe in the plurality of stripes, the sorting may be based on a bitmap stored on a plurality of storage devices having a particular stripe in the plurality of stripes. The sorting may be based on how many blocks of storage data currently exist in each of the plurality of stripes. The sorting may be based on the read and write overheads committed to each of the plurality of stripes. At any given time, only one of the plurality of computing devices may own each fault-resilient address space, and only its owner may read and write each of the plurality of fault-resilient address spaces. Each computing device may own a plurality of fault-resilient address spaces. A plurality of storage devices may be organized into a plurality of fault domains. Each of the plurality of stripes may span a plurality of fault domains. Each failure resilient address space may span all of the multiple fault domains such that upon a failure of any specific one of the multiple fault domains, a workload for reconstructing lost data is distributed between each other of the multiple fault domains. Multiple stripes may be distributed across multiple fault domains such that in the event of a concurrent failure of two of the multiple fault domains, the chance that two blocks of any specific one of the multiple stripes reside on two of the multiple fault domains is exponentially less than the chance that only one block of any specific one of the multiple stripes resides on two of the multiple fault domains.

[0082] A plurality of computing devices may be operated to first reconstruct any of a plurality of stripes having two faulty blocks, and then reconstruct any of a plurality of stripes having only one faulty block. A plurality of computing devices may be operated to reconstruct a plurality of stripes having two faulty blocks at a higher rate than the reconstruct rate of a plurality of stripes having only one faulty block (e.g., a greater percentage of CPU clock cycles dedicated to reconstructing, a greater percentage of network transmission probability dedicated to reconstructing, etc.). In the event of a failure in one or more fault domains, a plurality of computing devices may be operated to determine a rate at which any specific lost block is reconstructed based on how many other blocks of the same stripe in the plurality of stripes have been lost. One or more of the plurality of fault domains include a plurality of storage devices. Each of the plurality of FEC protection domains may span a plurality of stripes in the plurality of stripes.

[0083] Multiple strips can be organized into groups (e.g. Figure 8 Block strips 9201–920 in S), wherein each of the plurality of groups includes one or more of the plurality of stripes, and for each group, the plurality of computing devices are operable to sort the one or more of the plurality of stripes in the group. The plurality of computing devices are operable to: perform continuous commit operations on a selected one of the plurality of groups until the one or more of the plurality of stripes in the group no longer satisfies a determination criterion, and once the selected one of the plurality of groups no longer satisfies the determination criterion, select a different one of the plurality of groups. The criterion may be based on how many blocks are available for new data to be written.

[0084] Although the present method and / or system has been described with reference to specific implementations, it will be appreciated by those skilled in the art that various changes may be made and equivalents may be substituted without departing from the scope of the present method and / or system. In addition, various modifications may be made to adapt specific situations or materials to the teachings of the present disclosure without departing from the scope of the present disclosure. Therefore, the present method and / or system is not intended to be limited to the specific implementations disclosed, but rather the present method and / or system shall include all implementations falling within the scope of the appended claims.

[0085] As used herein, the terms "circuits" and "circuitry" refer to physical electronic components (i.e., hardware) and any software and / or firmware ("code") that can configure hardware, be executed by hardware, and / or be otherwise associated with hardware. As used herein, for example, a specific processor and memory may include a first "circuit" when executing a first line or more lines of code, and may include a second "circuit" when executing a second line or more lines of code. As used herein, "and / or" refers to any one or more items in a list connected by "and / or". As an embodiment, "x and / or y" refers to any element in a three-element set {(x), (y), (x, y)}. In other words, "x and / or y" refers to "one or two of x and y". As another embodiment, "x, y, and / or z" refers to any element in a seven-element set {(x), (y), (z), (x, y), (x, z), (y, z), (x, y, z)}. In other words, "x, y, and / or z" means "one or more of x, y, and z". As used herein, the term "exemplary" refers to serving as a non-limiting embodiment, instance, or illustration. As used herein, the terms "eg" and "for example" list a list of one or more non-limiting embodiments, instances, or illustrations. As used herein, a circuit is "operable" to perform a function as long as the circuit includes the necessary hardware and code (if necessary) to perform the function, regardless of whether the function's performance is disabled or not enabled (e.g., by user configuration settings, factory trim, etc.).

Claims

1. A system comprising: Buckets; A first backend device; as well as The second backend device, wherein: The first backend device is operable to control the storage bucket, The storage bucket is initially located entirely on the first backend device, The first backend device is operable to split the storage bucket into two storage buckets, and The first computing device is operable to transfer one or both of the two buckets to the second backend device.

2. The system according to claim 1, wherein: The computing device includes the first backend device.

3. The system according to claim 1, wherein: The buckets are operable to construct multiple failure-resilient stripes.

4. The system according to claim 1, wherein: The first backend device is operable to split the bucket into three or more buckets.

5. The system according to claim 1, wherein: Each of the two storage buckets is operable to construct a failure-resilient stripe, and wherein a most recently added storage device of the plurality of storage devices is selected for the failure-resilient stripe.

6. A system comprising: A first computing device including a first backend; as well as Multiple storage devices, including: a storage bucket located entirely on the first back end of the first computing device, The storage bucket is operable to construct a failure resilient stripe across a group of storage devices selected from the plurality of storage devices, The first computing device is operable to split the bucket into two buckets, and A most recently added storage device of the plurality of storage devices is selected for the storage device group before an occupied storage device.

7. The system according to claim 6, wherein: The first computing device is operable to transfer any buckets to a second backend on a second computing device.

8. The system according to claim 7, wherein: The transferred bucket is located entirely on the second backend on the second computing device.

9. The system according to claim 6, wherein: All buckets are operable to build multiple failure-resilient stripes.

10. The system according to claim 6, wherein: The first computing device is operable to split a bucket on the first backend into three or more split buckets.

11. A method comprising: constructing a failure resilient stripe using the storage buckets of the back end of the first computing device, such that the failure resilient stripe spans a storage device group selected from the plurality of storage devices; splitting the storage bucket completely located on the first computing device into two storage buckets; and One or both of the two buckets are transferred to a second computing device back end.

12. The method according to claim 11, wherein: The first computing device backend is an element of a computing device.

13. The method according to claim 11, wherein: The method includes building a plurality of failure-resilient stripes using the storage buckets of the first computing device backend.

14. The method according to claim 11, wherein: The method includes splitting the bucket on the first computing device back end into three or more split buckets.

15. The method according to claim 11, wherein: The method comprises: selecting a most recently added storage device among the plurality of storage devices for a new failure resilient stripe; and The new fault-resilient strip is constructed.

16. A method comprising: forming a storage device group on a first computing device by selecting a storage device from a plurality of storage devices based on a time when the storage device was added to the plurality of storage devices; constructing a failure-resilient stripe across the group of storage devices via storage buckets located entirely on the first computing device; and The bucket is split into two buckets.

17. The method according to claim 16, wherein: The method includes transferring the bucket to a second computing device.

18. The method according to claim 16, wherein: The first computing device includes a front end and a back end.

19. The method according to claim 16, wherein: The method includes constructing a plurality of failure-resilient stripes using the buckets.

20. The method according to claim 16, wherein: The first computing device is operable to split the bucket into three or more split buckets.

Images