Distributed erasure coding virtual file system

Abstract

This application relates to a distributed erasure coding virtual file system. A plurality of computing devices are communicatively coupled to each other via a network, and each of the plurality of computing devices includes one or more of a plurality of storage devices. A plurality of failure resilient address spaces are distributed across the plurality of storage devices such that each of the plurality of failure resilient address spaces spans the plurality of storage devices. Each of the plurality of failure resilient address spaces is organized as a plurality of stripes. Each of the plurality of stripes or a plurality of stripes is part of a respective one of a plurality of forward error correction (FEC) protection domains. Each of the plurality of stripes can include a plurality of storage blocks. Each block of a particular stripe of the plurality of stripes can reside on a different storage device of the plurality of storage devices.

Description

[0001] This application is a divisional application of PCT application No. PCT / IB2016 / 001177, filed on July 26, 2016, entitled "Distributed Erasure Coding Virtual File System", which entered the national phase in the People's Republic of China on February 13, 2018, as application No. 201680048284.9, the entire contents of which are incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to virtual file systems, and in particular, to distributed erasure coding virtual file systems. BACKGROUND

[0003] The limitations and disadvantages of conventional approaches to data storage will become apparent to one of ordinary skill in the art with reference to the certain aspects of the present methods and systems set forth in the remainder of this disclosure by reference to the accompanying drawings.

[0004] CROSS-REFERENCE TO RELATED APPLICATIONS

[0005] U.S. Patent Application No. 14 / 789,422, entitled "Virtual File System Supporting Multi-tier Storage", is hereby incorporated by reference in its entirety. SUMMARY

[0006] Methods and systems for distributed erasure coding virtual file systems are provided, substantially as shown and / or described by way of reference to at least one of the accompanying drawings, and as more fully set forth in the claims. BRIEF DESCRIPTION OF DRAWINGS

[0007] Figure 1 Various exemplary configurations of virtual file systems in accordance with some aspects of the present disclosure are shown.

[0008] Figure 2 Various exemplary configurations of computing nodes using virtual file systems in accordance with some aspects of the present disclosure are shown.

[0009] Figure 3 Various exemplary configurations of specialized virtual file system nodes in accordance with some aspects of the present disclosure are shown.

[0010] Figure 4 Various exemplary configurations of specialized storage nodes in accordance with some aspects of the present disclosure are shown.

[0011] Figure 5A Another representation of a virtual file system in accordance with an exemplary implementation of the present disclosure is shown.

[0012] Figure 5B is shownFigure 5A An example implementation of a memory.

[0013] Figure 5C An example implementation is shown in which two distributed failure resilient address spaces are saved on multiple solid state disks.

[0014] Figure 5D An example chunk stripe is shown in accordance with an example implementation of the present disclosure.

[0015] Figure 6A Storing an example file to a virtual file system is shown in accordance with an example implementation of the present disclosure.

[0016] Figure 6B Merging data to commit data to a single stripe is shown.

[0017] Figure 7 Components of a virtual file system for managing commits to memory are shown.

[0018] Figure 8A is a flowchart showing an example process of preparing a chunk stripe for commit.

[0019] Figure 8B is a flowchart showing an example process of committing data to a prepared chunk stripe.

[0020] Figure 9 A forward error correction scheme that can be used to protect data stored to non-volatile memory of a virtual file system is shown in accordance with an example implementation of the present disclosure.

[0021] Figure 10 A forward error correction scheme that can be used to protect data stored to non-volatile memory of a virtual file system is shown in accordance with an example implementation of the present disclosure.

[0022] Figure 11 A forward error correction scheme that can be used to protect data stored to non-volatile memory of a virtual file system is shown in accordance with an example implementation of the present disclosure.

[0023] Figure 12A and Figure 12B An efficient commit operation that can be used by a virtual file system in accordance with an example implementation of the present disclosure is shown. DETAILED DESCRIPTION

[0024] Figure 1 Various example configurations of a virtual file system in accordance with some aspects of the present disclosure are shown. Figure 1The diagram shows a local area network (LAN) 102 comprising one or more virtual file system (VFS) nodes 120 (indexed by integers from 1 to J, j≥1), and optionally including (indicated by dashed lines): one or more dedicated storage nodes 106 (indexed by integers from 1 to M, M≥1), one or more compute nodes 104 (indexed by integers from 1 to N, N≥1), and / or an edge router connecting LAN 102 to a remote network 118. The remote network 118 optionally includes one or more storage services 114 (indexed by integers from 1 to K, K≥1), and / or one or more dedicated storage nodes 115 (indexed by integers from 1 to L, L≥1).

[0025] 104 per compute node n (n is an integer, where 1 ≤ n ≤ N) is a networked computing device (e.g., a server, personal computer, etc.) that includes circuitry used to run various client processes (directly on device 104). n On the operating system and / or on device 104 n (Running in one or more virtual machines / containers) and used for interface connections to one or more VFS nodes 120. As used in this disclosure, a "client process" is a process that reads data from and / or writes data to memory in the course of performing its primary function, but whose primary function is not memory-related (i.e., the process is only concerned with ensuring that its data is reliably stored and retrieved when necessary, without concern with where, when, or how the data is stored). Exemplary applications that give rise to such processes include: email server applications, web server applications, office productivity applications, customer relationship management (CRM) applications, and enterprise resource planning (ERP) applications, to name just a few. References below Figure 2 Description of compute node 104 n An example configuration.

[0026] 120 per VFS node j (j is an integer, where 1 ≤ j ≤ J) is a networked computing device (e.g., a server, personal computer, etc.) that includes circuitry for running the VFS process, and optionally, client processes (directly on device 104). n On the operating system and / or on device 104 n (In one or more virtual machines running). As used in this disclosure, a "VFS process" is a process that executes one or more of the VFS drivers, VFS frontends, VFS backends, and VFS memory controllers described below in this disclosure. References below Figure 3 Description of VFS node 120 jof an exemplary configuration of a VFS process. Thus, in exemplary implementations, the client processes and the VFS processes can share resources (e.g., processing resources and storage resources) of the VFS node 120 j The processes of the virtual file system can be configured to require a relatively small amount of resources to minimize the impact on the performance of the client applications. From the perspective of the client processes, the interface to the virtual file system is independent of the particular physical machine on which the VFS processes are running.

[0027] Each on-premise storage node 106 m (m is an integer where 1 < m < M) is a networked computing device and includes one or more storage devices and associated circuitry that makes the storage devices accessible via the LAN 102. Exemplary configurations of the on-premise storage nodes 106 Figure 4 are described below with reference to m

[0028] Each storage service 114 k (k is an integer where 1 < k < K) can be a cloud-based service such as Amazon S3, Microsoft Azure, Google Cloud, Rackspace, Amazon Glacier, and Google Nearline.

[0029] Each remote on-premise storage node 1151(1 is an integer where 1 < 1 < L) can be similar to or identical to the on-premise storage nodes 106. In exemplary implementations, the remote on-premise storage nodes 1151may store data in a different format and / or use a different protocol for access than the on-premise storage nodes 106 (e.g., HTTP rather than an Ethernet-based protocol or an RDMA-based protocol).

[0030] Figure 2 Various exemplary configurations of a computing node that uses a virtual file system are shown in accordance with some aspects of the present disclosure. The exemplary computing node 104 n includes hardware 202, which in turn includes a processor chipset 204 and a network adapter 208.

[0031] The processor chipset 204 can include, for example, an x86-based chipset that includes a single- or multi-core processor system-on-a-chip, one or more RAM ICs, and a platform controller hub IC. The chipset 204 can include one or more various types of bus adapters for connecting to other components of the hardware 202 (e.g., PCIe, USB, SATA, and / or others).

[0032] ​The network adapter 208 may, for example, include circuitry for interfacing to an Ethernet-based network and / or an RDMA-based network. In an example implementation, the network adapter 208 can include a processor (e.g., an ARM-based processor) and one or more of the illustrated software components can run on the processor. The network adapter 208 interfaces with other components of the LAN 100 via a (wired, wireless, or optical) link 226. In an example implementation, the network adapter 208 can be integrated with the chipset 204.

[0033] The software running on the hardware 202 includes at least: an operating system and / or a hypervisor 212, one or more client processes 218 (indexed by an integer from 1 to Q, Q > 1), and one or more instances of a VFS driver 221 and / or a VFS frontend 220. The computing node 104 n Additional software optionally running on the computing node 104 includes one or more virtual machines (VMs) and / or containers 216 (indexed by an integer from 1 to R, R > 1).

[0034] Each client process 218 q (q is an integer where 1 < q < Q) can run directly on the operating system 212 or can run in a virtual machine and / or container 216 r (r is an integer where 1 < r < R) that is serviced by the OS and / or hypervisor 212. Each client process 218 is a process that reads data from and / or writes data to memory in the course of performing its primary function, but whose primary function is not memory-related (i.e., the process is concerned only with its data getting reliably stored and retrievable when necessary, not with where, when, and how the data is stored). Exemplary applications that give rise to such processes include: email server applications, web server applications, office productivity applications, customer relationship management (CRM) applications, and enterprise resource planning (ERP) applications, to name a few.

[0035] Each VFS frontend instance 220 s (S is an integer where 1 < s < S if there is at least one frontend instance on the computing node 104 n provides an interface to an appropriate VFS backend instance (running on a VFS node) for routing file system requests that can originate from one or more client processes 218, one or more VMs and / or containers 216, and / or the OS and / or hypervisor 212. Each VFS frontend instance 220 sThe VFS front end 220 can run on a processor of the chipset 204 or on a processor of the network adapter 208. For a multi-core processor of the chipset 204, different instances of the VFS front end 220 can run on different cores.

[0036] Figure 3 Various exemplary configurations of a dedicated virtual file system node are shown in accordance with some aspects of the present disclosure. The exemplary VFS node 120 j includes hardware 302, which in turn includes a processor chipset 304, a network adapter 308, and optionally, one or more storage devices 306 (indexed by an integer from 1 to W, W > 1).

[0037] Each storage device 306 p (p is an integer, where 1 < p < P if there is at least one storage device) can include any suitable storage device for implementing a memory tier desired to be implemented within the VFS node 120 j .

[0038] The processor chipset 304 can be similar to the chipset 204 described above with reference to Figure 2 . The network adapter 308 can be similar to the network adapter 208 described above with reference to Figure 2 and can interface with other nodes of the LAN 100 via a link 326.

[0039] The software running on the hardware 302 includes at least an operating system and / or a hypervisor 212, and includes at least one of: one or more instances of the VFS front end 220 (indexed by an integer from 1 to W, W > 1), one or more instances of the VFS back end 222 (indexed by an integer from 1 to X, X > 1), and one or more instances of the VFS memory controller 224 (indexed by an integer from 1 to Y, Y > 1). Additional software optionally running on the hardware 302 includes: one or more virtual machines (VMs) and / or containers 216 (indexed by an integer from 1 to R, R > 1), and / or one or more client processes 318 (indexed by an integer from 1 to Q, Q > 1). Thus, as described above, VFS processes and client processes can share resources on a VFS node and / or can be saved on separate nodes.

[0040] The client processes 218 and the VMs and / or containers 216 can be as described above with reference to Figure 2 .

[0041] Each VFS front end instance 220 w (w is an integer, where 1 < w < W if there is at least one VFS front end instance 220 jThere exists at least one front-end instance (1 ≤ w ≤ W) that provides an interface for routing file system requests to the appropriate VFS back-end instance (running on the same VFS node or different VFS nodes), wherein the file system requests may originate from one or more client processes 218, one or more VMs and / or containers 216, and / or OS and / or hypervisors 212. Each VFS front-end instance 220 w It can run on the processor of chipset 304 or on the processor of network adapter 308. For the multi-core processor of chipset 304, different instances of VFS front-end 220 can run on different cores.

[0042] 222 VFS backend instances per VFS x (x is an integer, where, if in VFS node 120) j There exists at least one backend instance (1 ≤ x ≤ X) that serves the file system requests it receives and performs tasks that otherwise manage the virtual file system (e.g., load balancing, journaling, maintaining metadata, caching, moving data between tiers, removing expired data, correcting corrupted data, etc.). Each VFS backend instance has 222 instances. x It can run on the processor of chipset 304 or on the processor of network adapter 308. For the multi-core processor of chipset 304, different instances of VFS backend 222 can run on different cores.

[0043] 224 VFS memory controller instances u (u is an integer, where, if in VFS node 120) j There exists at least one VFS memory controller instance (1 ≤ u ≤ U) that processes various storage devices 306 (possibly existing in VFS node 120). j Or interaction with another VFS node 120 or storage node 106. This may include, for example, address translation and generating commands to be issued to storage devices (e.g., SATA, PCIe, or other suitable buses). Therefore, VFS memory controller instance 224 u The operation acts as an intermediary between storage devices and various VFS backend instances of the virtual file system.

[0044] Figure 4 Various exemplary configurations of a dedicated storage node according to some aspects of this disclosure are shown. Exemplary dedicated storage node 106 m This includes hardware 402, which in turn includes a network adapter 408 and at least one storage device 306 (indexed by integers from 1 to Z, where Z ≥ 1). Each storage device 306 z This can be used as a reference above.Figure 3 The storage devices 306 described w The network adapters 408 can include circuitry (e.g., ARM-based processors) and bus (e.g., SATA, PCIe, or other) adapters that are operable to access (read, write, etc.) the storage devices 4061-406 Z These commands can conform to a standard protocol. For example, the special-purpose storage nodes 106 m may support RDMA-based protocols (e.g., Infiniband, RoCE, iWARP, etc.) and / or RDMA-reliant protocols (e.g., NVMe over fabrics).

[0045] In example implementations, tier 1 memory is distributed across one or more storage devices 306 (e.g., FLASH devices) that reside in one or more storage nodes 106 and / or one or more VFS nodes 120. Data written to the VFS is initially stored to tier 1 memory and then migrated to one or more other tiers as dictated by a data migration protocol, which can be user-defined and / or adaptive based on machine learning.

[0046] Figure 5A Another representation of a virtual file system is shown in accordance with example implementations of the present disclosure. In Figure 5A Element 502 represents the processing (e.g., x86 processors, ARM processors, NICs, ASICs, FPGAs, and / or other) and memory resources (e.g., DRAM and / or other short-term storage) of the various nodes (compute, storage, and / or VFS) that reside the virtual file system as described above. Element 508 represents one or more physical storage devices 306 that provide long-term storage for the virtual file system.

[0047] As shown in Figure 5A Physical storage is organized into a plurality of distributed failure resilient address spaces (DFRAS) 514. Each failure resilient address space includes a plurality of chunks 510, which in turn include a plurality of blocks 512. Organizing blocks 512 into chunks 510 is convenient in some implementations and can not be used for all implementations. Each block 512 stores committed data 514 (which can assume various states described below) and / or metadata 504 that describes the committed data 514.

[0048] Organizing memory 308 into a plurality of DFRAS allows for high performance parallel commits from multiple (or perhaps all) nodes of the virtual file system (e.g., Figure 1 All nodes 1041-104 N, 1061-106 M , and 1201-120 J may be submitted concurrently). In an example implementation, each of the nodes of the virtual file system can own respective one or more of the plurality of DFRAS and have exclusive read / commit access to the DFRAS it owns. Each DFRAS that only its own node owns / has access to allows each of the nodes of the virtual file system to control a portion of the memory 308 without having to coordinate with any other node (except during initialization or during (re)allocation of DFRAS after a node failure, e.g., which can be performed asynchronously from the actual reads / commits to the memory 308). Thus, in such an implementation, each node can read / commit to its DFRAS independently of what the other nodes are doing, without needing to reach any consensus in reading and committing to the memory 308. To allow for such operation, the metadata can be maintained as mapping each DFRAS to its current owning node so that reads and commits to the memory 308 can be redirected to the appropriate node. Moreover, if a particular node fails, the fact that the particular node owns multiple DFRAS allows its workload to be more intelligently / efficiently distributed to other nodes (rather than having to allocate the entire workload to a single node, which can create a “hot spot”). In light of this, in some implementations, the number of DFRAS is large relative to the number of nodes in the system, such that any one DFRAS can be placed on another node as a relatively small load. This allows for fine-grained redistribution of the failed node’s load according to the capabilities / capacities of the other nodes (e.g., nodes with greater capabilities / capacities can be given a higher percentage of the failed node’s DFRAS).

[0049] Referring briefly to Figure 5B , shown is an example implementation in which the virtual file system includes D (integer) SSDs 5081-508 D Each SSD 508 d (d is an integer, where 1≤d≤D) includes C (integer) chunks 510 d，1 -510 d，C , and each chunk 510 d，c (c is an integer, where 1≤c≤C) includes B (integer) blocks 512 d，c，1 -512 d，c，B . As discussed further below, the chunks 510 1，1 -510 D，C may be further organized into a plurality of chunk stripes 5201-520 S (S is an integer). In an example implementation, each chunk stripe 520 s(S is an integer, where 1 < s < S). The number of chunks 510 in any particular chunk stripe 520 can thus be determined based on the desired level of data protection d，c .

[0050] For purposes of illustration, assume that each chunk stripe 520 s includes N = M + K (where each of N, M, and K is an integer) chunks 510 d，c , then the N chunks 510 d，c store M storable data digits (typically, binary digits or "bits" for the current storage device) and the N chunks 510 d，c store K storable protection digits (again, typically bits). The virtual file system can then assign each stripe 520 s N chunks 508 d，c from N different failure domains.

[0051] As used herein, a "failure domain" refers to a group of components for which failure of any single component (the component loses power, becomes unresponsive, and / or otherwise) can cause failure of all components. For example, if a rack has a single rack-top switch, failure of that switch will cause disconnection of all components (e.g., compute, storage, and / or VFS nodes) on that rack. Thus, it is equivalent to all components on that rack failing simultaneously, as far as the rest of the system is concerned. A virtual file system according to the present disclosure can include fewer failure domains than chunks 510.

[0052] In an example implementation in which nodes of the virtual file system are connected and powered in a fully redundant manner and each such node has only a single storage device 306, the failure domain can be exactly that single storage device 306. Thus, in an example implementation, each chunk stripe 520 s includes a plurality of chunks 510 D residing on each of the N storage devices 306 d，c (thus, D is greater than or equal to N). An example of such an implementation is shown in Figure 5C .

[0053] In Figure 5C , D = 7, N = 5, M = 4, K = 1, and the memory is organized as two DFRAS. These numbers are for illustration only and are not intended to be limiting. The three chunk stripes 520 of the first DFRAS are arbitrarily named for purposes of illustration. The first chunk stripe 520i includes chunks 510 1，1 , 510 2，2 , 510 3，3 , 510 4，5 , and 510 5，6; the second chunk stripe 5202 includes chunks 510 3，2 , 510 4，3 , 510 5，3 , 510 6，2 , and 510 7，3 ; and the third chunk stripe 5203 includes 510 1，4 , 510 2，4 , 510 3，5 , 510 5，7 , and 510 7，5 .

[0054] Although in the simple example of Figure 5C D = 7 and N = 5, in a practical implementation D can be much larger than N (e.g., an integer multiple of greater than 1 and possibly by orders of magnitude) and the two values can be chosen such that the probability of any two chunk stripes 520 of a single DFRAS residing on the same set of N storage devices 306 (or, more generally, on the same set of N failure domains) is below a desired threshold. In this way, a failure of any single storage device 306 d (or, more generally, any single failure domain) will (expectation statistics can be determined based on the chosen values of D and N, the size of the N storage devices 306, and the arrangement of failure domains) result in the loss of at most one chunk 510 s of any particular stripe 520 b，c . Still further, a double failure will result in the loss of at most a single chunk 510 s in the majority of any particular stripe 520 b，c , and only a small number of stripes (determined based on the values of D and N) will lose two chunks (e.g., the number of stripes that fail twice can be exponentially smaller than the number of stripes that fail once).

[0055] For example, if each storage device 306 d is 1 TB, and each chunk is 128 MB, then a failure of a storage device 306 d will result (expectation statistics can be determined based on the chosen values of D and N, the size of the N storage devices 306, and the arrangement of failure domains) in 7812 (= 1 TB / 128 MB) chunk stripes 520 losing one chunk 510. For each chunk stripe 520 s so affected, the lost chunk 510 s can be quickly reconstructed using an appropriate forward error correction algorithm and the other N-1 chunks of the chunk stripe 520 d，c . Moreover, because the 7812 chunk stripes affected are evenly distributed across all of the storage devices 3061-306 D , the reconstruction of the lost 7812 chunks 510 d，cwill involve reading the same amount of data from each of the storage devices 3061-306 D (i.e., the burden of reconstructing the missing data is evenly distributed across all of the storage devices 3061-306 D to provide for rapid recovery from a failure).

[0056] Next, turning to the case of two simultaneous failures of storage devices 3061-306 D (or, more generally, two failure domains failing simultaneously), since the chunk stripes 5201-520 S of each DFRAS are evenly distributed across all of the storage devices 3061-306 D , only a very small number of chunk stripes 5201-520 S will lose two of their N chunks. Based on the metadata representing the mapping of chunk stripes 5201-520 S to storage devices 3061-306 D , the virtual file system can be operable to quickly identify these two missing chunk stripes. Once these two missing chunk stripes are identified, the virtual file system can prioritize reconstructing these two missing chunk stripes before starting to reconstruct one missing chunk stripe. The remaining chunk stripes will only have a single missing chunk and, for them (most of the affected chunk stripes), a simultaneous failure of two storage devices 306 d is the same as a failure of only one storage device 306 d . Similar principles apply to three simultaneous failures (the number of chunk stripes with three failed chunks will be less than the number of chunk stripes with two failed chunks in the two simultaneous failure case), etc. In an example implementation, the rate at which reconstruction of chunk stripes 520s is performed can be controlled based on the number of chunk stripes 520s that are missing. For example, this can be accomplished by controlling the rate at which reads and commits for reconstruction are performed, the rate at which FEC calculations for reconstruction are performed, the rate at which network messages for reconstruction are communicated, etc.

[0057] Reference is now made to Figure 5D , which shows an example implementation of chunk stripe 5201of Figure 5C . The chunks 512 d，c of chunk 510 d，c，b are organized into chunk stripes 5301-530 B , where each chunk stripe 530 b includes chunks 510 1，1 , 510 2，2 , 510 3，3 , 510 4，5 , and 510 5，6Block 512 with a specific bias within d，c，b (For example, block 530b includes block 510) 1，1 510 2，2 510 3，3 510 4，5 and 510 5，6 Each block in the array is biased by b × L, where L is the size of the block (e.g., 4kB). In the exemplary implementation where B = 1, the grouping strip is demoted to the block strip, and the term can be used synonymously. Each block is 512. d，c，b It can store data, protect data, and / or metadata. In an exemplary implementation, this metadata includes one or more bitmaps, each targeting a block 530. b Which block 512 of block strip 530b is being referred to? d，c，b Currently protected and block 530 b Which block 512? d，c，b Currently unprotected. Unprotected blocks may include: blocks that have never been written with data; blocks that store data that the owner has indicated is no longer needed (e.g., the owner has deleted the file corresponding to the data or the data has been committed elsewhere in the virtual file system); and blocks for which there are redundant, protected copies elsewhere in the virtual file system (e.g., on the target storage) (here referring to "read cache" blocks).

[0058] Return to Figure 5A During operation, data 506 to be committed to storage 508 is generated and / or stored (staged) using the processing and storage resources of 502 until it is committed to storage 508. When the virtual file system is ready to commit (de-stage) data 506, data 506 is transferred to storage 508 via LAN 102. The specific block 512 of block 520 to which the data is committed can be determined based on metadata 504 and / or metadata 505 describing the current state of the virtual file system (including, for example, which block of storage 508 is currently used, unused, protected, and / or unprotected).

[0059] Figure 6A An exemplary implementation of storing an exemplary file to a virtual file system is shown, according to this disclosure. Figure 6A In this process, a 4MB file 602 is submitted to a specific DFRAS. File 602 is divided into 1024 parts 610, each part 610 being 4kB in size. Then, using methods such as the following... Figure 9 to Figure 11The protection scheme described writes each 4 kB portion 610 to a respective chunk 512 of the DFRAS. Each 1 MB of the file is associated with a respective one of the ranges 6041-6044. The sizes of the file 602, ranges 604, portions 610, and chunks 512 are arbitrary choices and are not intended to be limiting. Each range 604 e In Figure 6A , e is an integer between 1 and 4, includes metadata 606 e which can be a portion of the metadata 504 and / or 505 in Figure 5A , or used to generate the metadata 504 and / or 505 in Figure 5A . The metadata 606 e includes information used to locate the portions 610 of the file on the storage 508. For example, this can include a pointer to each chunk 512 of one of the portions 610 associated with the range 604e. The 1024 chunks 512 of the 1024 portions can be (with a predetermined probability) evenly distributed across the failure domains of the DFRAS. Keeping a pointer to each chunk to which a portion of the file 602 is committed allows any particular portion of the file 602 to be committed to any particular chunk of the DFRAS, regardless of which storage device and / or stripe the particular chunk resides on. This effectively decouples a portion of data from the logical address / offset of the chunk on the storage 308 to which it will be committed. This allows data from different client application write operations to be merged into a single commit to the most attractive stripe. Thus even in the case of 4 kB or less multiple writes from many possibly unrelated client applications, the virtual file system can still merge the data and commit the data to a single stripe by simply updating the corresponding range to point to that stripe. Thus, many "stranger to each other" writes can share the same total commit overhead (e.g., the average of the 'U's described below with reference to Figure 12A and Figure 12B). Thus, the storage device and / or the particular stripe to which one or more portions of a particular file are committed can change over time as portions of the file are deleted and / or overwritten. An example is shown in Figure 6B .

[0060] In Figure 6BIn this example, each of the client processes 2181-2814 issues a write to the virtual file system for a respective data portion D1-D4. Each of the client processes 2181-2814 can be unaware of the other client processes 2181-2814. Similarly, the data generated by each can be completely independent of the data generated by the others (i.e., D1 can be completely independent of each of D2, D3, and D4; D2 can be completely independent of D1, D3, and D4; and so on). Any two or more of the writes can occur simultaneously, or all four of the writes can be spread out in time. In the example, D1 is a file (or a portion of a file) associated with extent 6041, D2 is a file (or a portion of a file) associated with extent 6042, D3 is a file (or a portion of a file) associated with extent 6043, and D4 is a file (or a portion of a file) associated with extent 6044. The data from the four writes is merged into a single commit that ends in block 5301. To illustrate, this example assumes that M = 4 and K = 1, and that block 5301 is the same as that shown in Figure 5D D1 is committed to block 512 1，1，1 and the corresponding pointer 6081 for extent 6041 is updated to point to 512 1，1，1 D2 is committed to block 512 2，2，1 and the corresponding pointer 6082 for extent 6042 is updated to point to 512 2，2，1 D3 is committed to block 512 3，3，1 and the corresponding pointer 6083 for extent 6043 is updated to point to 512 3，3，1 D4 is committed to block 512 5，4，1 and the corresponding pointer 6083 for extent 6043 is updated to point to 512 5，4，1 and P1 (e.g., equal to ) is committed to block 512 5，6，1 .

[0061] Now referring to Figure 7 again, it is shown that Figure 5AThe memory 508 and memory and processing resources 502 are shown. Also shown are a strip queue 704 and a strip queue manager 702. Strip queue 704 stores a list (e.g., pointers) of elements corresponding to one or more DFRAS blocks 520 and / or block 530 on memory 508. Elements in strip queue 704 can be ranked based on characteristics of block bars 520 and / or block bars 530 (e.g., by classifying the queue and / or storing ranking metadata of the queue), thereby making elements corresponding to the “most attractive” block bars and / or block bars 530 readily available (e.g., simply by “popping” the next element from strip queue 704). Characteristics can be determined based on metadata 504 and / or metadata 505. For example, the “most attractive” block bar 520 may correspond to the block bar 520 with the most (or at least the threshold number) specific states of block 512. s Block strip 520 corresponding to block strip 530 having at most (or at least a threshold number) blocks 512 in at least a certain number of specific states. s This corresponds to chunk bar 520 that performed the fewest (or at least below the threshold) commits. s For example, the "most attractive" block 530. b Block strip 530 can correspond to block 512 having the most (or at least the threshold number) specific states. b This corresponds to block 530, which has performed the fewest (or at least fewer than the threshold) commits. b Etc. Any specific block 512 can be determined by characteristics such as the following: d，c，b State: Block 512 d，c，b Currently, is data being stored? It is currently stored in block 512. d，c，b Whether the data in is currently protected (i.e., included in the calculation of the corresponding protection number), is currently stored in block 512. d，c，b Whether the data is stored elsewhere (e.g., in the target storage), etc.

[0062] In an example implementation, a separate stripe queue 704 can be maintained for each chunk stripe of a particular DFRAS such that any particular stripe queue 704 need only be aware of a relatively small portion of all chunk stripes of the particular DFRAS. In such an implementation, the chunk stripes of the current chunk stripe can be ranked and used to commit until the current chunk stripe no longer meets some criteria (or criteria). For example, the criteria (or criteria) can be based on the status of the individual chunks and / or chunk stripes of the chunk stripes. For example, the criteria can be whether the chunk stripe includes at least a threshold number of suitable chunk stripes, where a suitable stripe can be one that currently has at least a predetermined threshold number of chunks in a particular status (e.g., at least a threshold number of unprotected chunks and / or at least a threshold number of chunks that can be used to write (first or re-write) new data). Once the current chunk stripe no longer meets the criteria (or criteria), the node owning the DFRAS can move to the next chunk stripe (e.g., one selected at random or based on the ranking of the chunk stripes of the DFRAS). In this way, the average cost of commit operations (as discussed below with reference to FIG. 8B) can remain low. If none of the chunk stripes of the DFRAS contain a sufficient number of suitable chunk stripes, a scrub of the DFRAS can be performed to convert unsuitable chunk stripes to suitable chunk stripes. Figure 8A and Figure 8B Further discussed below), can remain low. If none of the chunk stripes of the DFRAS contain a sufficient number of suitable chunk stripes, a scrub of the DFRAS can be performed to convert unsuitable chunk stripes to suitable chunk stripes.

[0063] In an example implementation, the stripe queue 704 can also represent the status of each chunk stripe 530 b corresponding to the elements in the queue 704. In an example implementation, the status of a chunk stripe 530 can be "ready" or "unready." An unready chunk stripe 530 b is one that needs to have some operation performed on it before it is ready to accept commit of data that has not previously been written to it. A ready chunk stripe 530 b is one that has had such an operation performed and is ready to accept commit of data that has not previously been written to it. One example of such an operation is described with reference to FIG. 8A, attention now turning to Figure 8A .

[0064] In block 802, the stripe queue manager 702 pops the next most attractive unprepared block stripe 530 ("first stripe") from the stripe queue 704. In block 804, the virtual file system reads the blocks 512 in the first stripe that currently hold data ("first data") from the first stripe, holding and protecting that data during and after the next commit to the first stripe. In block 806, the virtual file system computes a protection number ("first protection number") for the first data. In block 808, the virtual file system commits the first protection number to the first stripe. Based on the completion of block 808, the first stripe is ready to accept new data. The queue manager 702 can then move on to preparing the next most attractive unprepared block stripe 530. The queue manager 702 can perform such queue preparation operations as background processing using excess resources (processor cycles, available network bandwidth, etc.) so as not to impede other operations.

[0065] Figure 8B is a flowchart illustrating an exemplary process for committing data to a block stripe prepared in Figure 8A In block 822, data ("second data") is prepared for demotion to the memory 508. In block 824, the stripe queue manager 702 pops the next most attractive prepared block stripe 530 ("first stripe") from the stripe queue 704. In block 826, the second data is written to the blocks of the first stripe that do not store data that is to be persisted by a commit operation. In block 828, the virtual file system computes a protection number ("second protection number") for the first stripe based on the second data and other data in the first stripe that is to be persisted and protected by committing the second data. In block 830, the first protection number for the first stripe is overwritten with the second protection number.

[0066] Figure 8A and Figure 8B The processing of Figure 8B makes the commit operations performed in Figure 12A and Figure 12B very efficient. This is further described below with reference to

[0067] Figure 9 A forward error correction scheme that can be used to protect data stored to a virtual file system's non-volatile memory is shown in accordance with an exemplary implementation of the present disclosure. Shown are the storage blocks 902 of a block stripe 5301-5304 of a DFRAS 1，1 - 902 7，7 Each of which can be a block 512 described above. In the protection scheme of Figure 9 , five blocks of each stripe are used to store data numbers and two blocks of each stripe are used to store data for protection numbers (i.e., M = 5 and K = 2). In Figure 9 , the protection numbers are computed using the following equations (1) through (9):

[0068]

[0069]

[0070]

[0071]

[0072]

[0073]

[0074]

[0075]

[0076]

[0077] Thus, Figure 9 The four stripes 5301-5304 in are part of a multi-stripe (in this case, four stripe) FEC protection domain and any two or fewer blocks lost in any of the block stripes 5301-5304 can be recovered using different combinations of the above equations (1) through (9). For comparison, an example of a single stripe protection domain would be if D11, D22, D33, D44, D54 were protected by only P1, then D11, D22, D33, D44, D54, and P1 would all be written to stripe 5301 (5301 would be a single stripe FEC protection domain).

[0078] Figure 10 A forward error correction scheme that can be used to protect data stored to a virtual file system's non-volatile memory is shown in accordance with an example implementation of the present disclosure. Figure 10 A scheme is shown in which the protection can be described as ((M+K) x R) + O, where M is the number of data blocks in a block stripe, K is the number of protection blocks in a block stripe, R is the redundancy factor, and O is the number of protection blocks calculated from the (M+K) x R data blocks to compute the protection numbers. In Figure 10 In, M = 6, K = 2, R = 3, and O = 1. These numbers are arbitrarily chosen and are not intended to be limiting.

[0079] Shown is data 1002 based on a commit to be stored in one or more stripe of blocks. Data 1002 includes 12 portions, each of which can correspond to data and / or new data that has been written to a stripe of blocks to which data 1002 is to be committed. Shown as 1004 are calculations of Pl and Ql based on data portions 1-6, and calculations of P2 and Q2 based on data portions 7-12. Shown as 1006 are calculations of P3 by data portions 1-6 and Pl and Ql, and calculations of P4 by data portions 7-12 and P2 and Q2.

[0080] Shown as 1008 is an example of data and protection written to two stripes on memory 508. The first stripe spans blocks 512 1，1 to 512 1，25 (and thus internal failure domains 1-25) and the second stripe spans blocks 512 1，26 to 512 2，18 (and thus internal failure domains 26-18). In this example, memory 508 includes 32 internal failure domains and four external failure domains. For example, each internal failure domain can be a respective one of 32 SSDs and each external failure domain can be a server that houses a respective 8 of the 32 SSDs. Thus, if any single server fails, its 8 SSDs are taken offline. As shown, if an external failure domain fails, Figure 10 the protection scheme of Figure 10 may be used in conjunction with multi-stripe FEC protection domains, as used in the example protection scheme of FIG. 9.

[0081] Figure 11 An example forward error correction scheme that can be used to protect data stored to non-volatile memory of a virtual file system is shown in accordance with an example implementation of the present disclosure. Figure 11 The protection scheme of may be described as a "nested" or "layered" protection scheme, in which the data is first encoded using Ml + Kl and the resulting encoded data is again encoded using M2 + K2 protection. This is not intended to be limiting, although only two layers are encoded for illustrative purposes.

[0082] In Figure 11 Ml = 6, Kl = 2, M2 = 3, K2 = 1. These numbers are arbitrarily chosen and not intended to be limiting.

[0083] Shown is data 1102 to be stored in one or more stripe based on a commit being completed. Data 1102 includes 24 portions, each of which can correspond to data and / or new data that has been written to a stripe to which data 1102 is to be committed. Shown as 1104 are calculations of Pl and Ql based on data portions 1-6, calculations of P2 and Q2 based on data portions 7-12, calculations of P3 and Q3 based on data portions 13-18, and calculations of P4 and Q4 based on data portions 19-24. Shown as 1106 are calculations of P5 through data portions 1, 7, and 13; P6 through data portions 19, 2, and 8; P7 through data portions 14, 20, and 3; P8 through data portions 9, 15, and 21; P9 through data portions 4, 10, and 16; P10 through data portions 22, 5, and 11; P11 through data portions 17, 23, and 6; P12 through data portions 12, 18, and 24; P13 through Pl, P2, and P3; P14 through P4, Ql, and Q2; and P15 through Q3 and Q4.

[0084] Shown as 1108 is an example of data and protection for eleven stripes on storage 508 (the first two of which are shown in the figure). The first stripe spans internal failure domains 1, 9, 17, and 25; the second stripe spans internal failure domains 2, 10, 18, and 26; and so on. In this example, storage 508 includes 32 internal failure domains and four external failure domains. For example, each internal failure domain can be a respective one of 32 SSDs and each external failure domain can be a server that houses a respective 8 of the 32 SSDs. Thus, if any single server fails, its 8 SSDs are taken down. As shown, if an external failure domain fails, Figure 10 The protection scheme of Figure 11 The protection scheme of Figure 9 may be used in conjunction with multiple FEC protection domains, as used in the example protection scheme of

[0085] Figure 12A and Figure 12B High efficiency commit operations that can be used by a virtual file system in accordance with example implementations of the present disclosure are shown. In Figure 12A and Figure 12B Shown is a commit to an example stripe 530 that includes blocks 1202-1220 (each of which is an example of block 512 d，c，b in FIG. 11A).

[0086] Figure 12AThe first way in which data can be written to stripe 530 is shown. For this commit process, to commit 1 new chunk to stripe 530, 7 other chunks are read, new parity (Q2 and P2) is computed based on existing data (D1 through D9) and new data (D10), and then 3 writes are performed. Thus, committing 1 new chunk requires 7 read operations, one parity computation, and 3 write operations. Generalizing to U new chunks to be written, the cost of committing U new chunks using this process can be described as: (M-U) reads + (U+K) writes + compute parity.

[0087] Figure 12B The second way in which data can be written to stripe 530 is shown. For this commit process, to commit 1 new chunk to stripe 530, the following steps are performed: 3 chunks are read: a parity chunk and chunks to be rewritten (1212 in the example shown); new parity chunks (Q2 and P2) are computed based on data to be rewritten (D5), new data (D10), and old parity chunks (Q1 and P1); 3 chunks are written (new data D10 through 1212 and new parity Q2 and P2 through 1218 and 1220, respectively). Thus, committing 1 new chunk requires 3 read operations, one parity computation, and 3 write operations. Generalizing to U new chunks to be written, the cost of committing U new chunks using this process can be described as: U+K reads + U+K+ compute parity.

[0088] To compare the two processes, an example implementation can assume that the parity operation costs are equally high in Figure 12A and Figure 12B and that the read and write costs are equally high. With these assumptions, and expressing the cost of the parity computation as 'Y', Figure 12A the cost of the process of can be written as (M+K+Y) / U and Figure 12B the cost of the process of can be written as (2x(U+K)+Y) / U. By setting the costs equal to: (M+K+Y) / U = (2x(U+K)+Y)) / U, a threshold for determining which of the two processes to use for any particular commit can be obtained. Solving for U as (M-K) / 2. Thus, when U is less than (M-K) / 2, Figure 12B the process of can have less total overhead, and when U is greater than (M-K) / 2, Figure 12A the process of can have less total overhead.

[0089] Of course, in other implementations, reads can not be equally as costly as writes and / or different parity calculations can cost differently. Such implementations can take these factors into account when determining which commit procedure to use for any particular commit operation. For example, writes can be G times more costly than reads, in which case the costs of the two procedures can be rewritten as: (M-U) x G + (U+K) + calculating parity and (U+K) x G + (U+K) + calculating parity. From these expressions, different criteria (e.g., a threshold for U) can be derived to determine when to use the Figure 12A procedure and when to use the Figure 12B procedure. Similarly, different parity calculations can apply different weight factors for use in the criteria used for the commit procedure. Moreover, because the characteristics of NAND FLASH found in most SSDs cause writes to be significantly more expensive than reads, and because the ability to balance random writes of the same stripe (as discussed above with reference to Figure 6A and Figure 6B ), a virtual file system utilizing these aspects of the present disclosure can achieve significantly stronger total write overhead than a non-merging model in which K protection blocks must be calculated and stored for each write (i.e., U can typically be 1 for a non-merging model, as in the case of random workloads). That is, aspects of the present disclosure achieve a greater typical or average value of U as compared to a non-merging model.

[0090] As can be seen from the costs of Figure 12A (M+K+Y) / U and Figure 12B ((2 x (U+K) + Y)) / U), the cost / total overhead of a commit operation decreases as the number of new blocks of the stripe to be written (U) increases. Thus, in exemplary implementations, the stripe queue 704 (described above with reference to Figure 7 , Figure 8A , and Figure 8B ) can be managed to maximize some metric based on U (e.g., to maximize U for any particular commit operation and / or to maximize the average value of U over a series of commit operations). In exemplary implementations, this can be achieved by buffering data from multiple write instructions of one or more client applications and committing the data to the memory 308 in a single commit, as discussed above with reference to Figure 6A .

[0091] According to an exemplary implementation of this disclosure, multiple computing devices (e.g., two or more of 104, 106, and 120) are communicatively coupled to each other via a network (e.g., 102), and each of the multiple computing devices includes one or more of a plurality of storage devices (e.g., 306). Distributing multiple fault-tolerant address spaces (e.g., multiple 514) across the multiple storage devices ensures that each of the multiple fault-tolerant address spaces spans multiple storage devices. Each of the multiple fault-tolerant address spaces is organized into multiple stripes (e.g., multiple 530). Each or more of the multiple stripes is a plurality of forward error correction (FEC) protection domains (e.g., ...). Figure 9 A portion of a corresponding one of multiple FEC domains. Each of the multiple domains may include multiple storage blocks (e.g., multiple 512). Each block of a particular domain in the multiple domains may reside on different storage devices in multiple storage devices. The first portion of the multiple storage blocks (e.g., by...) Figure 9 The number of items 5301 is five out of 902 1，2 Up to 902 1，6 Composed of multiple storage blocks, the second part (e.g., ...) can be used to store data numbers, and the second part of multiple storage blocks... Figure 9 The number of items 5301 is two of 902. 1，1 and 902 1，7) can be used to store a number of protected numbers computed based at least in part on a number of data digits. A number of computing devices can operate to rank a number of stripes. The ranking can be used to select which of the number of stripes to use for a next commit operation for one of a number of failure resilient address spaces. The ranking can be based on how many protected and / or unprotected storage blocks in each of the number of stripes. For any particular stripe of the number of stripes, the ranking can be based on a bitmap stored on a number of storage devices with the particular stripe of the number of stripes. The ranking can be based on how many blocks in each of the number of stripes currently store data. The ranking can be based on a total overhead of reads and writes committed to each of the number of stripes. At any given time, only one of the number of computing devices can own each of the number of failure resilient address spaces, and each of the number of failure resilient address spaces can only be read and written by its owner. Each of the computing devices can own a number of failure resilient address spaces. A number of storage devices can be organized into a number of failure domains. Each of the number of stripes can span the number of failure domains. Each of the failure resilient address spaces can span all of the number of failure domains, such that upon a failure of any particular failure domain of the number of failure domains, a workload to reconstruct lost data is distributed in each of the others of the number of failure domains. The number of stripes can be distributed across the number of failure domains such that if two of the number of failure domains fail simultaneously, the probability that two blocks of any particular stripe of the number of stripes reside on the two that fail is exponentially smaller than the probability that only one block of any particular stripe of the number of stripes resides on the two that fail. The number of computing devices can operate to first reconstruct any of the number of stripes that have two failed blocks, and then reconstruct any of the number of stripes that have only one failed block. The number of computing devices can operate to perform reconstruction of the number of stripes that have two failed blocks at a higher rate than the number of stripes that have only one failed block (e.g., with a greater percentage of CPU clock cycles dedicated to reconstruction, a greater percentage of network transmission opportunities dedicated to reconstruction, etc.). If one or more of the failure domains fail, the number of computing devices can operate to determine a rate at which to reconstruct any particular lost block based on a number of other blocks in the same stripe of the number of stripes that are lost. Wherein the one or more of the number of failure domains comprise a number of storage devices. Each of the number of FEC protection domains can span a number of stripes of the number of stripes. The number of stripes can be organized into a number of groups (e.g., group striped stripes 5201 through 520 S), wherein each of the plurality of groups includes one or more of the plurality of stripes, and the plurality of computing devices are operable to rank, for each group, the one or more of the plurality of stripes in the group. The plurality of computing devices are operable to perform a continuous commit operation to a selected one of the plurality of groups until the one or more of the plurality of stripes of the selected one of the plurality of groups no longer satisfy a predetermined criterion, and to select a different one of the plurality of groups once the selected one of the plurality of groups no longer satisfies the predetermined criterion. The criterion can be based on a number of chunks of new data available to be written. Each of the plurality of FEC protection domains (e.g., the first stripe in Figure 10 ) can include a plurality of data storage chunks (e.g., 512 1，1 to 512 1，6 , 512 1，9 to 512 1，14 , and 512 1，17 to 512 1，22 ), a first protection storage chunk (e.g., any of 512 1，7 , 512 1，8 , 512 1，15 , 512 1，16 , 512 1，23 , and 512 1，23 ), and a second protection storage chunk (e.g., 512 1，25 ). A number stored in the first protection storage chunk can be computed based on contents of a subset of the plurality of data storage chunks. A number stored in the second protection storage chunk can be computed based on at least contents of a subset of the plurality of data storage chunks and the first protection chunk (e.g., P3 is computed based on a subset of contents of 5121to 512 24 ). The plurality of computing devices are operable to compute a first protection portion (e.g., P1 and Q1 of Figure 10 ) based on a plurality of data portions (e.g., data portions 1 to 6 of Figure 10 ), to replicate the plurality of data portions and the first protection portion, the replication resulting in a first set including the plurality of data portions, one or more replicas of the plurality of data portions, the first protection portion, and one or more replicas of the first protection portion (e.g., the first set includes data portions 1 to 6 and three occurrences of protection portions P1 and Q1, as shown at 1006 of Figure 10 ), to compute a second protection portion based on all of the data portions and all of the first protection portions of the first set (e.g., P3 is computed based on data portions 1 to 6 and all three occurrences of protection portions P1 and Q1, as shown at 1006 of Figure 10 ), to commit the first set and the second protection portion to one of the plurality of FEC protection domains (e.g., to the first stripe in Figure 10 ). The one of the plurality of FEC protection domains (e.g., the first stripe in Figure 11The first one in the list may include multiple first storage blocks (e.g., 512). 1，1 512 1，9 and 512 1，17 ) and second storage block (e.g., 512) 1，25 ), wherein, stored in each of the plurality of first storage blocks, can be: one of the plurality of data portions (e.g., Figure 11 One of the data sections 1 to 24), or a protection number calculated based on multiple data sections (e.g., Figure 11 (One of P1, Q1, P2, Q2, P3, Q3, P4, and Q4 in the first storage block). The protection portion stored in the second storage block can be calculated based on the contents of multiple first storage blocks (e.g., Figure 11 (One of P5 to P15). According to the system described in Statement 1, multiple computing devices are capable of operating based on multiple data portions (e.g., Figure 11 Data sections 1 to 6) calculate the first protection section (e.g., Figure 11 The second protection part (e.g., in P1) is calculated based on multiple data parts and a first subset of the first protection part. Figure 11 In the process, P5 is calculated based on data portions 1, 7, and 13. Multiple data portions and a first subset of the first protection portion, along with the first protection portion, are submitted to the first of multiple FEC protection domains (e.g., data portions 1, 7, 3, and P5 are submitted to the first domain in Figure 11). A second protection portion is calculated based on a second subset of the multiple data portions and the first protection portion (e.g., in...). Figure 11 In the process, P13 is calculated based on P1, P2, and P3, and multiple data portions and a second subset of the first protection portion, as well as the second protection portion, are submitted to the second of multiple FEC protection domains (e.g., submitting P1, P2, P3, and P13 to...). Figure 11 Article 9 of the law.

[0092] Accordingly, the present methods and systems can be implemented in hardware, software, or a combination of hardware and software. The present methods and / or systems can be implemented in a centralized fashion in at least one computer system or in a distributed fashion where different elements are spread across several interconnected computer systems. Any kind of computing system or other apparatus adapted for carrying out the methods described herein is suited. A typical combination of hardware and software can be a general purpose computing system with a program or other code that, when being loaded and executed, controls the computing system such that it carries out the methods described herein. Another typical implementation includes a special purpose computer or a data processor. Some implementations can include a network of several computer systems that are connected together to enable the distributed use of the present virtual file system. Some implementations can include a general purpose computer system that is programmed to behave in certain aspects like a special purpose computer system. Some implementations can include a special purpose computer system that is programmed to behave in certain aspects like a general purpose computer system. Some implementations can include physical, hardware, and / or tangible components or devices, such as microprocessors, storage media, logic circuits, etc. Some implementations can include software components or entities, such as a program, routine, app, application, operating system, etc. Some implementations can include a combination of hardware and software components or entities. Some implementations can include a non-transitory machine-readable medium or media having instructions stored thereon or therein which, when executed by a machine, cause the machine to perform any of the methods described herein. Some implementations can include a machine or machines, which, when

[0093] While the present methods and / or systems have been described with reference to certain implementations, it is to be understood that various other specific arrangements can be devised in

[0094] As used in this application, the terms "circuit" and "circuitry" refer to physical electronic components (i.e. hardware) and any software and / or firmware ("code") which can configure the hardware, whether such software and / or firmware is executed on the hardware or not, in addition to any other aspects associated with a hardware implementation. As used herein, a particular processor and memory can comprise a first "circuit" when executing one or more first codes and can comprise a second "circuit" when executing one or more second codes. As used herein, "and / or" means any one or more of the items in the list joined by "and / or". As an example, "x and / or y" means any element of the three-element set {(x), (y), (x, y)}. In other words, "x and / or y" means one or both of x and y. As another example, "x, y, and / or z" means any element of the seven-element set {(x), (y), (z), (x, y), (x, z), (y, z), (x, y, z)}. In other words, "x, y, and / or z" means one or more of x, y and z. As used herein, the term "exemplary" means serving as a non-limiting example, instance, or illustration. As used herein, the terms "e.g." and "for example" set off a list of one or more non-limiting examples, instances, or illustrations. As used herein, circuitry is "operable to" perform a function whenever the circuitry comprises the necessary hardware and / or code (if any is needed) to perform the function, regardless of whether performance of the function is enabled or disabled by some user- configurable setting, factory tweak, etc.

Claims

1. A system comprising: One of a plurality of computing devices, wherein: Each of the plurality of computing devices is associated with one or more of the plurality of storage devices; Each of the multiple bars spans a group of two or more storage devices across the multiple storage devices, wherein the group is selected from the multiple groups based on predetermined criteria; The computing device is capable of operating as follows: Select one of the multiple bars; Write the data to the first part of the selected bar; and Write error protection information into the second part of the selected bar.

2. The system according to claim 1, wherein, The selection of a bar is based on how many storage blocks are currently in each of the plurality of bars.

3. The system according to claim 1, wherein, The selection is based on the bitmaps stored in the plurality of storage devices.

4. The system according to claim 1, wherein, The selection of a bar is based on how many storage blocks in each of the plurality of bars currently store data.

5. The system according to claim 1, wherein, The selection of a bar is based on the total read and write overhead for each of the plurality of bars.

6. The system according to claim 1, wherein, Each of the multiple clauses spans multiple fault domains.

7. The system according to claim 1, wherein, The computing device is operable to rebuild the specific stripe in the event of a failure of one or more storage devices in the group of two or more storage devices spanned by the specific stripe.

8. The system according to claim 1, wherein, The computing device is operable to prioritize the reconstruction of multiple stripes.

9. The system according to claim 1, wherein, The computing device is operable to prioritize the reconstruction of multiple strips based on the number of faulty storage devices in each of the multiple strips.

10. The system according to claim 1, wherein, The computing device can operate to rebuild a strip with more faulty storage devices at a higher rate than a strip with fewer faulty storage devices.

11. The system according to claim 1, wherein, The computing device is operable to write the data into the selected bar until the criteria are met.

12. The system according to claim 11, wherein, The standard is based on the number of blocks that can be used to write new data.

13. The system according to claim 1, wherein, Each stripe includes one or more storage blocks on each of the two or more storage devices it spans.

Images