Data migration in distributed file system

Abstract

Data migration in a distributed file system is provided. The plurality of computing devices are communicatively coupled to each other via a network, and each computing device of the plurality of computing devices is operably coupled to one or more of the plurality of storage devices. The plurality of fault tolerant address spaces are distributed across the plurality of storage devices such that each of the plurality of fault tolerant address spaces spans the plurality of storage devices. The plurality of computing devices maintain metadata that maps each fault tolerant address space to one of the plurality of computing devices. Metadata is grouped into buckets. Each bucket is stored in a back end of the computing device. Data may be migrated from an external file system to a plurality of storage devices using an index node stub to represent directories and files of the external file system. When the content of the external file system is copied, the index node stub is replaced by the real index node.

Description

[0001] This application is a divisional application of the Chinese application with application number 201980056627.X, application date June 5, 2019, and invention name “Data Migration in Distributed File Systems”, all contents of which are incorporated herein by reference.

[0002] Priority claim

[0003] This application claims priority to U.S. Provisional Patent Application No. 62 / 691,732, filed on June 29, 2018, entitled “Data Migration in a Distributive File System,” and U.S. Patent Application No. 16 / 273,762, filed on February 12, 2019, entitled “Data Migration in a Distributive File System.” Technical Field

[0004] The present application relates to data migration, and in particular, to data migration in a distributed file system. Background Art

[0005] By comparing conventional methods for data storage with some aspects of the present method and system set forth in the remainder of this disclosure with reference to the accompanying drawings, limitations and disadvantages of conventional methods for data storage will become apparent to those skilled in the art.

[0006] Incorporated by Reference

[0007] U.S. Patent Application No. 15 / 243,519, entitled "Coded Virtual File System," is incorporated herein by reference in its entirety. Summary of the invention

[0008] The present application provides a method and system for data migration in a distributed file system substantially as shown and / or described in conjunction with at least one of the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 Various example configurations of distributed file systems according to aspects of the present disclosure are shown.

[0010] Figure 2 An example configuration of a distributed file system node according to aspects of the present disclosure is shown.

[0011] Figure 3 Another representation of a distributed file system according to an example implementation of the present disclosure is shown.

[0012] Figure 4AAn example of an external file system and an internal file system according to an example implementation of the present disclosure is shown.

[0013] Figure 4B An example of data migration from an external file system to an internal file system according to an example implementation of the present disclosure is shown.

[0014] Figure 4C An example of an internal file system after data migration according to an example implementation of the present disclosure is shown.

[0015] Figure 5 is a flow chart illustrating an example method for migrating data to a distributed file system according to the present disclosure.

[0016] Figure 6 An exemplary implementation in which two distributed fault-tolerant address spaces reside on multiple solid-state storage disks is shown.

[0017] Figure 7 A forward error correction scheme that may be used to protect data stored to a non-volatile memory of a virtual file system according to an example implementation of the present disclosure is shown. DETAILED DESCRIPTION

[0018] Traditionally, file systems use centralized control of metadata structures (e.g., directories, files, attributes, file contents). If a local file system is accessible from a single server and that server fails, the file system's data may be lost if no further protection exists. To add protection, some file systems (e.g., as provided by NetApp) use one or more pairs of controllers in an active-passive manner to replicate metadata across two or more computers. Other solutions use multiple metadata servers in a clustered manner (e.g., as provided by IBM GPFS, DellEMC Isilon, Lustre, etc.). However, because the number of metadata servers in a traditional cluster system is limited to a small number, such a system cannot be scaled.

[0019] The system in the present disclosure can be applied to small clusters and can also be scaled to many, thousands of nodes. An exemplary embodiment of a non-volatile memory (NVM) is discussed, for example, flash memory in the form of a solid-state drive (SSD). The NVM can be divided into 4kB blocks and 128MB blocks. Extents can be stored in volatile memory (e.g., RAM for fast access) and backed up by NVM memory. Extents can store pointers for blocks, for example, 256 pointers to 1MB of data stored in a block. In other embodiments, larger or smaller memory partitions can also be used. The metadata functions in the present disclosure can be effectively distributed across many servers. For example, in the case where the target of a large load is a "hot spot" of a specific part of the file system namespace, the load can be distributed across multiple nodes.

[0020] Figure 1 Various example configurations of distributed file systems according to aspects of the present disclosure are shown. Figure 1 1 is a local area network (LAN) 102 that includes one or more nodes 120 (indexed by integers from 1 to J, where j≥1), and optionally includes (indicated by dashed lines): one or more dedicated storage nodes 106 (indexed by integers from 1 to M, where M≥1), one or more compute nodes 104 (indexed by integers from 1 to N, where N≥1), and / or an edge router that connects the LAN 102 to a remote network 118. The remote network 118 optionally includes one or more storage services 114 (indexed by integers from 1 to K, where K≥1) and / or one or more dedicated storage nodes 115 (indexed by integers from 1 to L, where L≥1).

[0021] 120 per node j (j is an integer, where 1≤j≤J) is a networked computing device (e.g., a server, a personal computer, etc.) that includes a computer for directly accessing the device 104. n operating system and / or on the device 104 n A circuit for running a process (e.g., a client process) in one or more virtual machines running in.

[0022] The compute node 104 is a networking device that can run a virtual front end without a virtual back end. The compute node 104 can run a virtual front end by incorporating single root input / output virtualization (SR-IOV) into a network interface card (NIC) and consuming a full processor core. Alternatively, the compute node 104 can run a virtual front end by routing the network via the Linux kernel network stack and using kernel process scheduling, so there is no requirement for a full core. This is useful if the user does not want to allocate a full core for the file system or if the network hardware is incompatible with the file system requirements.

[0023] Figure 2 An example configuration of a node according to aspects of the present disclosure is shown. The node includes a front end 202 and a driver 208, a memory controller 204, a back end 206, and an SSD agent 214. The front end 202 can be a virtual front end; the memory controller 204 can be a virtual memory controller; the back end 206 can be a virtual back end; and the driver 208 can be a virtual driver. As used in the present disclosure, a virtual file system (VFS) process is a process that implements one or more of the front end 202, the memory controller 204, the back end 206, and the SSD agent 214. Therefore, in an example implementation, the resources (e.g., processing and memory resources) of the node can be shared among the client process and the VFS process. The VFS process can be configured to require a relatively small amount of resources to minimize the impact on the performance of the client application. The front end 202, the memory controller 204 and / or the back end 206 and / or the SSD agent 214 can run on a processor of the host 201 or on a processor of the network adapter 218. For a multi-core processor, different VFS processes can run on different cores and can run different subsets of services. As far as the client process 212 is concerned, the interface with the virtual file system is independent of the specific physical machine on which the VFS process runs. The client process only requires the driver 208 and the front end 202 to exist in order to service the client process.

[0024] The node can be implemented as a single tenant server (e.g., bare-metal) running directly on the operating system or as a virtual machine (VM) and / or container (e.g., Linux container (LXC)) within a bare-metal server. The VFS can run as a VM environment within an LXC container. Therefore, inside the VM, the only thing that can run is the LXC container that includes the VFS. In a traditional bare-metal environment, there are user space applications and the VFS runs in the LXC container. If the server is running other containerized applications, the VFS can run inside an LXC container that is outside the management scope of the container deployment environment (e.g., Docker).

[0025] The node may be served by an operating system and / or a virtual machine monitor (VMM) (e.g., a hypervisor). The VMM may be used to create and run nodes on a host 201. Multiple cores may reside within a single LXC container running a VFS, and the VFS may run on a single host 201 using a single Linux kernel. Thus, a single host 201 may include multiple front ends 202, multiple memory controllers 204, multiple back ends 206, and / or one or more drivers 208. The driver 208 may run inside kernel space outside the scope of the LXC container.

[0026] SR-IOV PCIe virtual functions can be used to run the network stack 210 in user space 222. SR-IOV allows isolation of PCI Express so that a single physical PCI Express can be shared on a virtual environment and different virtual functions can be provided to different virtual components on a single physical server machine. The I / O stack 210 enables the VFS node to bypass the standard TCP / IP stack 220 and communicate directly with the network adapter 218. A portable operating system interface for uniX (POSIX) VFS functionality can be provided to the VFS driver 208 through a lock-free queue. SR-IOV or full PCIe physical function addresses can also be used to run a non-volatile memory express (NVMe) driver 214 in user space 222, thereby completely bypassing the Linux IO stack. NVMe can be used to access a non-volatile storage device 216 attached via a PCI Express (PCIe) bus. The non-volatile storage device 220 may be, for example, flash memory in the form of a solid state drive (SSD) or storage class memory (SCM) in the form of an SSD or memory module (DIMM). Other examples may include storage class memory technologies such as 3D-XPoint.

[0027] By coupling the physical SSD 216 with the SSD agent 214 and the network 210, the SSD can be implemented as a networked device. Alternatively, by using a network protocol such as NVMe-oF (NVMe over Fabrics (architecture of all-flash arrays)), the SSD can be implemented as a network-attached NVMe SSD 242 or NVMe SSD 244. NVMe-oF can allow the use of redundant network links to access NVMe devices, thereby providing a higher level or fault tolerance. Network adapter 226, network adapter 228, network adapter 230, and network adapter 232 may include hardware acceleration for connecting to NVMe SSD 242 and NVMe SSD 244 to convert the NVMe SSD 242 and NVMe SSD 244 into networked NVMe-oF devices without using a server. NVMe SSD 242 and NVMe SSD 244 can each include two physical ports, and all data can be accessed through any one of these ports.

[0028] Each client process / application 212 can run directly on the operating system, or can run in a virtual machine and / or container served by the operating system and / or hypervisor. The client process 212 can read data from the memory and / or write data to the memory in the process of performing its main function. However, the main function of the client process 212 is independent of storage (that is, the process only pays attention to the data of the process being reliably stored and retrievable when needed, and does not pay attention to where the data is stored, when the data is stored, or how the data is stored). Example applications that cause such processes include: email servers, web servers, office productivity applications, customer relationship management (CRM), animation video rendering, genome computing, chip design, software construction, and enterprise resource planning (ERP).

[0029] The client application 212 can make a system call to the kernel 224 that communicates with the VFS driver 208. The VFS driver 208 places the corresponding request on the queue of the VFS front end 202. If there are several VFS front ends, the driver can load balanced access to different front ends to ensure that a single file / directory is always accessed via the same front end. This can be done by sharding the front end based on the ID of the file or directory. The VFS front end 202 provides an interface for routing file system requests to the appropriate VFS back end based on the bucket responsible for the operation. The appropriate VFS back end can be on the same host or can be on another host.

[0030] The VFS backend 206 hosts several buckets, each of which services the file system requests it receives and performs tasks, thereby managing the virtual file system in other ways (e.g., load balancing, logging, maintaining metadata, caching, moving data between tiers, removing expired data, correcting corrupted data, etc.).

[0031] The VFS SSD agent 214 handles the interaction with the corresponding storage device 216. This may include, for example, translating addresses, and generating commands issued to the storage device (e.g., via SATA, SAS, PCIe, or other suitable buses). Thus, the VFS SSD agent 214 operates as an intermediary between the storage device 216 and the VFS backend 206 of the virtual file system. The SSD agent 214 may also communicate with standard network storage devices that support standard protocols such as NVMe-oF (NVMe over Fabrics).

[0032] Figure 3 Another representation of a distributed file system according to an example implementation of the present disclosure is shown. Figure 3 , element 302 represents memory resources (e.g., DRAM and / or other short-term memory) and processing (e.g., x86 processors, ARM processors, NICs, ASICs, FPGAs, etc.) resources of various nodes (compute, storage, and / or VFS), such as those described above with respect to Figure 2 The described virtual file system resides on these nodes. Element 308 represents one or more physical storage devices 216 that provide long-term storage for the virtual file system.

[0033] like Figure 3 As shown, physical memory is organized into a plurality of distributed fault tolerant address spaces (DFRAS) 318. Each of the plurality of distributed fault tolerant address spaces (DFRAS) 318 includes a plurality of blocks 310, which in turn includes a plurality of blocks 312. In some implementations, the blocks 312 are organized into blocks 310 for convenience only, and in all implementations, the blocks 312 may not be organized into blocks 310. Each block 312 stores committed data 316 (which may take various states, as discussed below) and / or metadata 314 that describes or references the committed data 316.

[0034] Storage 308 is organized into multiple DFRAS to enable high performance concurrent submissions from many (perhaps all) nodes of the virtual file system (e.g., Figure 1 All nodes 1041 to 104 N , Node 1061 to Node 106 M and node 1201 to node 120 JConcurrent commits may be performed in parallel.) In an example implementation, each node of the virtual file system may own a corresponding one or more of the plurality of DFRASs and have exclusive read / commit access to the DFRASs owned by the node.

[0035] Each bucket has a DFRAS and therefore does not need to coordinate with any other node when writing to that bucket. Each bucket can be striped across many different blocks on many different SSDs, so each bucket and its DFRAS can choose what "block stripe" to write to at the moment based on many parameters, and once a block is assigned to that bucket, no coordination is required to do so. All buckets can effectively write to all SSDs. Without any coordination.

[0036] Each DFRAS owned and accessible only by the owner bucket of each DFRAS running on a particular node allows each node of the VFS to control a portion of memory 308 without having to cooperate with any other node (except during initialization or during [reallocation] of the bucket holding the DFRAS after a node failure, for example, which can be performed asynchronously to actually read / commit memory 308). Therefore, in such an implementation, each node can read / commit to the DFRAS of the bucket of each node independently of what other nodes are doing without the need to achieve any consistency when reading and submitting to memory 308. In addition, in the event of a particular node failure, the fact that a particular node owns multiple buckets allows the workload of the particular node to be more intelligently and efficiently redistributed to other nodes (in contrast, the entire workload must be assigned to a single node, which can create a "hot spot"). In this regard, in some implementations, the number of buckets can be larger relative to the number of nodes in the system, so that any bucket can be a relatively small load placed on another node. This allows for fine-grained redistribution of the load of a failed node based on the power and capacity of other nodes (eg, nodes with greater power and capacity may be given a higher percentage of failed node buckets).

[0037] To allow such operations, metadata may be maintained that maps each bucket to the current owning node of each bucket so that reads and commits to memory 308 may be redirected to the appropriate node.

[0038] Because the entire file system metadata space (e.g., directories, file attributes, content ranges within files, etc.) can be divided (e.g., sliced ​​or sharded) into small, uniform pieces (e.g., "shards"), load distribution is possible. For example, a large system with 30k servers can slice the metadata space into 128k or 256k shards.

[0039] Each such metadata shard can be maintained in a "bucket". Each VFS node can be responsible for several buckets. When a bucket on a given backend is serving metadata shards, the bucket is considered "active" or the "leader" of that bucket. Typically, there are many more buckets than VFS nodes. For example, a small system with 6 nodes may have 120 buckets, and a larger system with 1,000 nodes may have 8k buckets.

[0040] Each bucket can be active on a small set of nodes (typically 5 nodes, which form a penta-group for that bucket). The cluster configuration is kept up to date on all participating nodes about the penta-group assignment for each bucket.

[0041] Each quintuple monitors itself. For example, if the cluster has 10k servers and each server has 6 buckets, each server will only need to talk to 30 (6 buckets will have 6 quintuples, so 6*5=30) different servers to maintain the state of the server's bucket. This is a much smaller number than if a centralized entity had to monitor all nodes and maintain cluster-wide state. Because nodes do not perform more work as the cluster size increases, the use of quintuples allows performance to be scaled with larger clusters. This may result in the following disadvantages: in "dumb" mode, a small cluster can actually generate more communication than physical nodes, but this disadvantage can be overcome by utilizing all buckets shared by two servers to send only a single heartbeat between the two servers (as the cluster grows, this will change to only one bucket, but if you have a small cluster of 5 servers, the cluster of 5 servers only includes all buckets in all messages and each server will only talk to the other 4 servers). Quintuples can be decided (i.e., reach consensus) using an algorithm similar to the Raft consensus algorithm.

[0042] Each bucket can have a set of compute nodes that can run it. For example, five VFS nodes can run a bucket. However, only one of the nodes in the group is the controller / leader at any given moment. Furthermore, for sufficiently large clusters, no two buckets share the same group. If there are only 5 or 6 nodes in the cluster, most buckets can share a backend. In fairly large clusters, there are many different node groups. For example, for 26 nodes, there are more than possible groups of five nodes (ie, quintuples).

[0043] All nodes in the group know and agree (i.e., reach consensus) which node is the actual active controller (i.e., leader) for the bucket. A node accessing a bucket can remember ("cache") the last node that was the leader among the (e.g., five) members of the group for that bucket. If it accesses the bucket leader, the bucket leader performs the requested operation. If it accesses a node that is not the current leader, the node instructs the leader node to "redirect" the access. If there is a timeout accessing the cached leader node, the contacting node can try a different node for the same five-tuple. All nodes in the cluster share a common "configuration" of the cluster, which allows the node to know which server can run each bucket.

[0044] Each bucket may have a load / usage value that indicates the extent to which the bucket is used by applications running on the file system. For example, a server node with 11 lightly used buckets may receive another bucket of metadata to run before a server with 9 highly used buckets, even though there will be an imbalance in the number of buckets used. The load value may be determined based on average response latency, the number of concurrently running operations, memory consumed, or other metrics.

[0045] Even when the VFS node is not faulty, reallocation can also occur. If the system determines that one node is busier than other nodes based on the tracked load metrics, the system can move one of the buckets in the node's bucket (i.e., "failover") to another server that is not too busy. However, before actually relocating the buckets to different hosts, load balancing can be achieved by transferring writes and reads. Since each write can end on a different group of nodes determined by DFRAS, nodes with higher loads may not be selected in the stripe where the data is being written. The system can also choose not to serve reads from highly loaded nodes. For example, a "degraded mode read" can be performed, in which blocks in a high-load node are reconstructed from other blocks in the same stripe. Degraded mode reads are reads performed via the remaining nodes in the same stripe, and the data is reconstructed via fault protection. Because the initiator of the read can assume that the node has failed, the degraded mode read can be performed when the read latency is too high. If the load is high enough to create a higher read latency, the cluster can revert to reading the data from other nodes and using degraded mode reads to reconstruct the required data.

[0046] Each bucket manages its own distributed erasure coding instance (i.e., DFRAS 518) and does not need to cooperate with other buckets to perform read or write operations. There are potentially thousands of concurrent distributed erasure coding instances working concurrently, each targeting a different bucket. This is a major part of the performance scaling, as it effectively allows any large file system to be divided into independent pieces that do not need to cooperate, thus providing high performance regardless of scale.

[0047] Each bucket handles all file system operations that fall into its shard. For example, directory structures, file attributes, and file data ranges will fall into the jurisdiction of a specific bucket.

[0048] An operation done from any frontend starts by finding out the bucket that owns the operation. Then, the backend leader and node for the bucket is determined. This determination can be performed by trying the last known leader. If the last known leader is not the current leader, the node can learn which node is the current leader. If the last known leader is no longer part of the bucket's quintuple, the backend will let the frontend know that the frontend should go back to the configuration to find the members of the bucket's quintuple. The distribution of operations allows complex operations to be handled by multiple servers rather than by a single computer in a standard system.

[0049] If the cluster size is small (e.g., 5) and quintuples are used, there will be buckets that share the same group. As the cluster size increases, the buckets are redistributed so that no two groups are the same.

[0050] Figure 4A An example of a cluster 427 and an external file system 421 according to an example implementation of the present disclosure is shown. The cluster 427 includes an internal file system that may reside on one or more network interface cards (NICs). The external file system 421 includes an external client 423 and an external memory.

[0051] For illustration, as shown, cluster 427 includes two computing devices 403 and 413 and four storage devices 409a, 409b, 409c, and 409d. Different numbers of computing devices (e.g., CPU-based servers) and storage devices (e.g., SSDs) may be used without departing from the present disclosure. Computing devices 403 and 413 and storage devices 409a, 409b, 409c, and 409d may be operably coupled via one or more NICs. For example, each of computing devices 403 and 413 may be on a separate NIC, and storage devices 409a, 409b, 409c, and 409d may be on one or more NICs.

[0052] The computing device 401 includes a front end 403 and a back end 405. The back end 405 includes at least one bucket 407. The computing device 411 includes a front end 413 and a back end 415. The back end 415 includes at least one bucket 417.

[0053] Each bucket in the backend is operable to construct one or more fault-tolerant stripes comprising a plurality of blocks. For example, for 10 blocks, 8 data blocks may be protected with 2 error protection / correction blocks (i.e., using 8+2 stripes). Similarly, for 10 fault domains, 6 data blocks may be protected with 4 error protection / correction blocks (i.e., using 6+4 stripes).

[0054] Bucket 407 is operable to construct a fault-tolerant stripe 419 including block a1, block a2, and block a3. Typically, each of the plurality of storage blocks in a particular fault-tolerant (or fault-protective) stripe may be located in a different storage device in the plurality of storage devices. Alternatively, the plurality of storage blocks of the fault-tolerant stripe may be distributed across at least two storage devices in the plurality of storage devices.

[0055] Bucket 407 can generate, manage and control fault-tolerant stripes 419 via index nodes and one or more disk areas. An index node is a data structure that stores all information about a file or directory other than the naming of the file or directory and its actual data. Each 4k block in a file can be handled by an object called a disk area. The disk area ID can be based on the index node ID and the offset within the file (e.g., 1MB). And, the disk area ID is managed by a bucket. The disk area can be stored in the registry of each bucket. The disk area range can be protected by a read-write lock. Each disk area can manage the continuous 1MB of content for the file by managing pointers to up to 256 4k blocks responsible for 1MB of data. In other embodiments, larger or smaller memory partitions can also be used. The disk area can also be stored in a volatile memory (e.g., RAM for fast access) and is also backed up by NVM memory.

[0056] Figure 4B An example of migrating data from an external file system 421 to an internal file system in a cluster 427 according to an example implementation of the present disclosure is shown. Migrating a large file system can be a very lengthy process, especially if the user has multiple file systems layered. In addition, the user may also need to access the external file system before the migration can be completed.

[0057] During the migration, the external file system 421 may or may not change. The migration includes a mapping process to traverse the namespace of the external memory 425 and create a directory hierarchy with stub index nodes (e.g., stub index nodes 431). The stub index nodes 431 are substitutes for the actual nodes, which are coupled to the memory 409a to the memory 409d via one or more disk areas 433. The stub index nodes 431 can quickly allow access to the pre-existing external memory 425 while migrating the data to the memory 409a to the memory 409d at the back end. The mapping process can generate index node stubs for directories as well as files.

[0058] The user can be notified when the entire migration process is complete. Once completed, all stub index nodes will be replaced by real nodes. Alternatively, each stub index node will be replaced by a real node as the data is migrated. The opendir function can be run to block access operations to files on a specific directory until the backend process completes the entire directory migration. For a directory, opendir can block access until all contained file stubs and directory stubs are generated. Thereafter, opendir will return and readdir can begin.

[0059] When a file inode stub (or directory inode stub) is opened, the system can pause until the file (or directory) is pulled from the filer. When the file inode stub is opened, migration can create all possible extents based on the associated external files. These extents 433 are associated with locations in the extent stripe 437.

[0060] The backend process may traverse the queue of all stubs and proceed to create the namespace 429 and copy the files. The backend process may traverse all directory inode stubs and then traverse all file inode stubs. After all (or each) file inodes are created, they may be converted into real inodes and the contents of the external file may be downloaded to the disk area.

[0061] The system may maintain the migration status and may be able to notify the user when all files have been copied to the cluster system 427. Each migration process may need to know whether the corresponding namespace 429 still has some stubs (index nodes or extents) or whether all stubs have been converted.

[0062] Even if the external file system 421 is large, the data migration is tolerant to restarts and failures. It is not possible to access the root of the namespace 429 until all files in the root directory have been migrated.

[0063] Data migration can be run concurrently from several nodes. Each backend will be responsible for its own directories and files. If one computing device node 411 cannot directly mount the migrated external file system 421 and another computing device node 401 can directly mount the migrated external file system 421, a tunnel 435 can be generated from the backend 415 of the one computing device node 411 to the backend 405 of the other computing device node 401.

[0064] During the migration, progress can be indicated (e.g., in terms of number or capacity of files moved). The state of disk extents to be extracted and inode stubs to be converted into real directories / files can be maintained. Migrations can continue to run across bucket failovers and system restarts.

[0065] Figure 4C FIG. 4 shows an example of an internal file system of a cluster 427 after data migration according to an example implementation of the present disclosure. Figure 4C In the example of , stripe 437 has been filled with blocks b1 and b2. The external file has been migrated to block b1. Block b2 includes data that can be used to recover block b1. A sparse file indicates that the space required by the external file system is less than the allocated space. If an extent is read for 0 seconds when reading an external file, the extent will be deleted to create a sparse file. Extents corresponding to sparse files can be added to the queue via tags in the registry.

[0066] Figure 5 5 is a flow chart illustrating an example method for migrating data from an external file system to an internal distributed file system according to the present disclosure. In block 501, index node stubs are generated for files and directories of the external file system. For example, buckets in a computing device backend may generate these stub index node data structures.

[0067] In block 503, one or more stripes are allocated. These stripes span multiple storage devices in the internal file system, and each stripe includes multiple storage blocks. Each stripe can be generated by a bucket in the computing device backend.

[0068] In block 505, the contents of the files and directories of the external file system are migrated to the storage blocks of the stripe. During the migration, data can be accessed from the external storage via the stub index node. When migrating a specific file, access can be blocked by an open command.

[0069] In block 507, one or more error correction data blocks are generated for each stripe based on the migrated data. The error correction data is stored in the corresponding stripe. During a degraded read in which data on one or more storage devices is unavailable, the error correction data is used in conjunction with available data to regenerate the unavailable data. A storage block in the plurality of storage blocks is designated as a sparse file that is not used for the migrated data or the associated error correction data.

[0070] Figure 6 An exemplary implementation in which two distributed fault-tolerant address spaces reside on multiple solid-state storage disks is shown.

[0071] Block 910 1,1 To block 910 D,C Can be organized into multiple block strips 9201 to 920 S (S is an integer). In an example implementation, each block stripe 920 is individually protected using forward error correction (eg, erasure coding). s (s is an integer, where 1≤s≤S). Any particular block stripe 920 s Block 910 d,c The number can therefore be determined based on the desired level of data protection.

[0072] For the purpose of illustration, assume that each block stripe is 920 s The number of blocks 910 is N=M+K (where each of N, M and K is an integer). d,c , then N blocks 910 d,c M blocks in 910 d,c Data bits (usually binary digits or "bits" of current storage devices) can be stored, and N blocks 910 d,c K blocks in 910 d,c A protection bit (again, typically a bit) may be stored. The virtual file system may then assign a value to each stripe 920. s Allocate N blocks from N different fault domains 910 d,c .

[0073] As used herein, a "fault domain" refers to a group of components in which the failure of any single component in the group of components (component loses power, becomes unresponsive, etc.) can cause the failure of all components. For example, if a rack has a single top-of-rack switch, the failure of that switch will disrupt connectivity to all components on that rack (e.g., compute, storage, and / or VFS nodes). Thus, to the rest of the system, this is equivalent to if all components on that rack failed together. A virtual file system according to the present disclosure may include fewer fault domains than block 910.

[0074] In an example implementation where the nodes of the virtual file system are connected and powered in a fully redundant manner with only a single storage device 906 per such node, the failure domain may be only that single storage device 906. Thus, in the example implementation, each block stripe 920s includes N storage devices 9061 to 9066 residing on the N storage devices 9061 to 9066. D A plurality of blocks 910 on each of d,c , (so D is greater than or equal to N). Figure 6 An example of such an implementation is shown in .

[0075] exist Figure 6 In FIG. 1 , D=7, N=5, M=4, K=1, and the memory is organized into two DFRAS. These numbers are for illustration only and are not intended to be limiting. For illustration, the three block stripes 920 of the first DFRAS are arbitrarily called. The first block stripe 9201 consists of blocks 910 1,1 ,910 2,2 ,910 3,3 ,910 4,5 and 910 5,6 The second block stripe 9202 is composed of block 910 3,2 ,910 4,3 ,910 5,3 ,910 6,2 and 910 7,3 and the third block stripe 9203 is composed of block 910 1,4 ,910 2,4 ,910 3,5 ,910 5,7 and 910 7,5 composition.

[0076] Despite Figure 6 In the simple example of , D=7 and N=5, but in actual implementations, D can be much larger than N (e.g., a multiple of an integer greater than 1 and can be as high as multiple orders of magnitude), and the two values ​​can be selected so that the probability that any two block stripes 920 of a single DFRAS reside on the same set of N storage devices 906 (or more generally, on the same set of N failure domains) is below a desired threshold. In this way, any single storage device 906 d Failure of any single fault domain (or more generally, any single fault domain) will result in the loss of any particular stripe 920 s At most one block in 910 b,c (The expected statistics are likely to be determined based on the selected values ​​of D and N, the size of the N storage devices 906, and the arrangement of the failure domains.) Further, a double failure will result in most stripes losing at most a single block 910. b,c, and only a small number of stripes (determined based on the values ​​of D and N) are lost from any particular stripe 920 s two blocks in (e.g., the number of two failed stripes may be exponentially less than the number of one failed stripe).

[0077] For example, if each storage device 906 d is 1 TB, and each block is 128 MB, then storage device 906 d A failure of 7812 (=1TB / 128MB) block stripes 920 will result in the loss of one block 910 (the expected statistics are likely to be determined based on the following items: the selected values ​​of D and N, the size of the N storage devices 906, and the arrangement of the failure domains). For each such affected block stripe 920 s , appropriate forward error correction algorithms and block striping 920 can be used s The other N-1 blocks in the block are used to correct the lost block 910 d,c In addition, since the affected 7812 block strips are evenly distributed across all storage devices 9061 to 906 D So for the lost 7812 blocks 910 d,c Performing the reconstruction will involve (expected statistics will likely be determined based on the selected values ​​of D and N, the size of the N storage devices 906, and the arrangement of the failure domains) D Each of the storage devices 9061 to 9064 reads the same amount of data (i.e., the burden of reconstructing the lost data is across all storage devices 9061 to 9064). D evenly distributed to allow very fast recovery from failures).

[0078] Next, turn to storage device 9061 to storage device 906 D In the case of concurrent failures of two of the fault domains (or more generally, concurrent failures of two fault domains), since each DFRAS block stripe 9201 to block stripe 920 S In all storage devices 9061 to 906 D uniform distribution on the block stripe 9201, so only a very small amount of block stripe 9201 to block stripe 920 S The virtual file system may be operated based on indicating block stripe 9201 to block stripe 920 S and storage device 9061 to storage device 906 DThe metadata of the mapping between the two lost block stripes can be used to quickly identify such two lost block stripes. Once such two lost block stripes are identified, the virtual file system can prioritize the reconstruction of the two lost block stripes before starting to reconstruct the one lost block stripe. The remaining block stripes will have only a single lost block, and for them (most of the affected block stripes), the two storage devices 906 d Concurrent failure with only one storage device 906 d A similar principle applies to the third concurrent failure (the number of block stripes with three failed blocks is even less than the number with two failed blocks in the two concurrent failure scenario), and so on. In an example implementation, the block stripes 920 may be based on s The number of losses in the control execution block stripe 920 s This can be achieved by, for example, controlling the rate at which reads and commits for reconstruction are performed, the rate at which FEC calculations for reconstruction are performed, the rate at which network messages for reconstruction are transmitted, etc.

[0079] Figure 7 FIG. 1 shows a forward error correction scheme that can be used to protect data stored in a non-volatile memory of a virtual file system according to an example implementation of the present disclosure. FIG. 1 shows a storage block 902 of a block stripe 9301 to a block stripe 9304 of a DFRAS. 1,1 To storage block 902 7,7 .exist Figure 7 In the protection scheme of , five blocks per stripe are used to store data bits, and two blocks per stripe are used to store protection bits (i.e., M=5 and K=2). Figure 7 In the calculation, the protection digits are calculated using the following formulas (1) to (9):

[0080]

[0081] therefore, Figure 7 The four stripes 9301 to 9304 in are part of a multi-strip (in this case, four stripes) FEC protection domain, and various combinations of the above formulas (1) to (9) can be used to recover the loss of any two or fewer blocks in any block stripe 9301 to block stripe 9304. For comparison, an example of a single stripe protection domain would be D11, D22, D33, D44, D54 protected only by P1, and D11, D22, D33, D44, D54 and P1 are all written to stripe 9301 (9301 is a single stripe FEC protection domain).

[0082] According to an example implementation of the present disclosure, a plurality of computing devices are communicatively coupled to each other via a network, and each of the plurality of computing devices includes one or more of a plurality of storage devices. A plurality of fault-tolerant address spaces are distributed across the plurality of storage devices, such that each of the plurality of fault-tolerant address spaces spans across the plurality of storage devices. Each of the plurality of fault-tolerant address spaces is organized into a plurality of stripes (e.g., Figure 6 and Figure 7 Each of the plurality of stripes or multiple stripes is a plurality of forward error correction (FEC) protection domains (e.g., such as Figure 6 Each of the plurality of stripes may include a plurality of storage blocks (e.g., a plurality of 912). Each block of a particular stripe of the plurality of stripes may reside on a different storage device of the plurality of storage devices. A first portion of the plurality of storage blocks (e.g., Figure 7 902 of 9301 strips 12 To 902 16 The number of five) can be used to store data bits, and the second part of the plurality of storage blocks (eg, Figure 7 Two 902 in strip 9301 1,1 and 902 1,7 The number of digits) can be used to store protection digits calculated at least in part based on the data digits.

[0083] A plurality of computing devices may be operable to sort a plurality of stripes. The sorting may be used to select which of the plurality of stripes is used for the next commit operation of a fault-tolerant address space in a plurality of fault-tolerant address spaces. The sorting may be based on how many protected and / or unprotected storage blocks exist in each of the plurality of stripes. For any particular stripe in the plurality of stripes, the sorting may be based on a bitmap stored on a plurality of storage devices having a particular stripe in the plurality of stripes. The sorting may be based on how many blocks of stored data currently exist in each of the plurality of stripes. The sorting may be based on the read and write overheads committed to each of the plurality of stripes. At any given time, each of the fault-tolerant address spaces may be owned by only one of the plurality of computing devices, and each of the plurality of fault-tolerant address spaces may be read and written by only its owner. Each computing device may own a plurality of fault-tolerant address spaces. The plurality of storage devices may be organized into a plurality of fault domains. Each of the plurality of stripes may span a plurality of fault domains. Each of the fault-tolerant address spaces may span all of the plurality of fault domains, such that upon a failure of any particular one of the plurality of fault domains, the workload for reconstructing lost data is distributed among each of the other one of the plurality of fault domains. The plurality of stripes may be distributed across the plurality of fault domains, such that in the event of a concurrent failure of two of the plurality of fault domains, the chance that two blocks of any particular one of the plurality of stripes reside on both of the failed one of the plurality of fault domains is exponentially less than the chance that only one block of any particular one of the plurality of stripes resides on two of the failed one of the plurality of fault domains.

[0084] The plurality of computing devices may be operable to first reconstruct any of the plurality of stripes having two failed blocks, and then reconstruct any of the plurality of stripes having only one failed block. The plurality of computing devices may be operable to perform reconstruction of the plurality of stripes having two failed blocks at a higher rate than the reconstruction rate of the plurality of stripes having only one failed block (e.g., a greater percentage of CPU clock cycles dedicated to reconstruction, a greater percentage of network transmission probability dedicated to reconstruction, etc.). In the event of a failure in one or more of the failure domains, the plurality of computing devices may be operable to determine a rate of reconstruction of any particular lost block based on how many other blocks of the same stripe in the plurality of stripes have been lost. Wherein, one or more of the plurality of failure domains include a plurality of storage devices. Each of the plurality of FEC protection domains may span a plurality of stripes in the plurality of stripes.

[0085] Multiple strips can be organized into multiple groups (e.g. Figure 6 Block stripe 9201 to block stripe 920 S), wherein each of the plurality of groups includes one or more of the plurality of stripes, and, for each group, the plurality of computing devices are operable to sort the one or more of the plurality of stripes in the group. The plurality of computing devices are operable to: perform continuous commit operations on a selected one of the plurality of groups until one or more of the plurality of stripes in the group no longer satisfies a determination criterion, and once the selected one of the plurality of groups no longer satisfies the determination criterion, select a different one of the plurality of groups. The criterion may be based on how many blocks are available for new data to be written.

[0086] Although the present method and / or system has been described with reference to certain implementations, it will be appreciated by those skilled in the art that various changes may be made and equivalents may be substituted without departing from the scope of the present method and / or system. In addition, many modifications may be made to adapt specific circumstances or materials to the teachings of the present disclosure without departing from the scope of the present disclosure. Therefore, the present method and / or system is not intended to be limited to the specific implementation disclosed, but rather the present method and / or system includes all implementations that fall within the scope of the appended claims.

[0087] As used herein, the terms "circuit" and "circuitry" refer to physical electronic components (i.e., hardware) and any software and / or firmware ("code") that can configure hardware, be executed by hardware, and / or otherwise be associated with hardware. As used herein, for example, a specific processor and memory may include a first "circuit" when executing a first line or more lines of code, and may include a second "circuit" when executing a second line or more lines of code. As used herein, "and / or" refers to any one or more items in a list connected by "and / or". As an example, "x and / or y" refers to any element in a three-element set {(x), (y), (x, y)}. In other words, "x and / or y" refers to "one or both of x and y". As another example, "x, y, and / or z" refers to any element in a seven-element set {(x), (y), (z), (x, y), (x, z), (y, z), (x, y, z)}. In other words, "x, y, and / or z" means "one or more of x, y, and z." As used herein, the term "exemplary" means serving as a non-limiting example, instance, or illustration. As used herein, the terms "eg" and "for example" set forth a list of one or more non-limiting examples, instances, or illustrations. As utilized herein, a circuit is "operable" to perform a function whenever the circuit includes the necessary hardware and code (if necessary) for performing the function, regardless of whether the function's performance (e.g., by user-configurable settings, factory trim, etc.) is disabled or not enabled.

Claims

1. A system comprising: multiple storage devices; as well as A computing device, comprising a bucket, wherein: The bucket is operable to read data from an external memory, The bucket is operable to write data to a fail-safe stripe, The failure protection stripe includes a plurality of storage blocks distributed across a group of storage devices selected from the plurality of storage devices, and The plurality of storage blocks are determined before accessing data from the external memory.

2. The system according to claim 1, wherein: The storage blocks of the fail-protection stripe are distributed across at least two storage devices of the plurality of storage devices.

3. The system according to claim 1, wherein: At least one memory block of the plurality of memory blocks of the fail-protection stripe is reserved for error correction data associated with other memory blocks of the plurality of memory blocks of the fail-protection stripe.

4. The system according to claim 1, wherein: The bucket is operable to generate an index node and one or more disk areas pointing to the plurality of storage blocks of the fail-safe stripe.

5. The system according to claim 1, wherein: If a storage block among the plurality of storage blocks is not used for migrated data or associated error correction data, the storage block is designated as a sparse file.

6. The system according to claim 1, wherein: The bucket is operable to generate a stub index node.

7. The system according to claim 6, wherein: The stub index node corresponds to one or both of a file in the external storage and a directory in the external storage.

8. The system according to claim 1, wherein: The system includes a network interface card (NIC).

9. A method comprising: Generate stub index nodes from buckets; specifying, by the bucket, a stripe, the stripe comprising a plurality of storage blocks across a set of storage devices selected from a plurality of storage devices; Migrating data from the external memory to the stripe, wherein before migrating the data from the external memory to the plurality of storage devices, determining the plurality of storage blocks in the plurality of storage devices; accessing the data from the external storage via the stub index node during data migration; and The data is accessed from the stripe after the data migration.

10. The method according to claim 9, wherein: The plurality of storage blocks of the stripe are distributed across at least two storage devices of the plurality of storage devices.

11. The method according to claim 9, wherein: At least one storage block among the plurality of storage blocks of the stripe is reserved for an error correction data block generated from other storage blocks among the plurality of storage blocks of the stripe.

12. The method according to claim 9, wherein: A data structure including an index node pointing to the plurality of storage blocks of the stripe and one or more disk areas is used to access the data from the stripe after data migration.

13. The method according to claim 9, wherein: If a storage block among the plurality of storage blocks is not used for migrated data or associated error correction data, the storage block is designated as a sparse file.

14. The method according to claim 9, wherein: The stub index node corresponds to one or both of a file in the external storage and a directory in the external storage.

15. The method according to claim 9, wherein: The stripe is a failsafe stripe.

Images