Distributed file system and method thereof, and non-transitory machine readable medium

Abstract

The invention relates to a distributed file system, a method thereof and a non-transitory machine readable medium. The plurality of computing devices are communicatively coupled to each other via a network, and each of the plurality of computing devices is operably coupled to one or more of the plurality of storage devices. The plurality of faulty resilient strips are distributed over the plurality of storage devices such that each of the plurality of faulty resilient strips spans the plurality of storage devices. The graphics processing unit is operable to access the data file from the faulty resilient stripe while bypassing the kernel page cache. In addition, these data files may be accessed in parallel by a graphics processing unit.

Description

[0001] This application is a divisional application of PCT application No. PCT / IB2019 / 000742, filed on June 5, 2019, entitled “GPU Based Server in a Distributed File System,” which entered the national phase in the People’s Republic of China on December 10, 2020, as application No. 201980038773.X, the entire contents of which are incorporated herein by reference.

[0002] Priority

[0003] This application claims priority to U.S. Provisional Patent Application No. 62 / 686,964, filed on June 19, 2018, entitled “GPU Based Server in a Distributed File System,” and U.S. Patent Application No. 16 / 273,749, filed on February 12, 2019, entitled “GPU Based Server in a Distributed File System.” BACKGROUND

[0004] The limitations and disadvantages of conventional data storage approaches will become apparent to one of ordinary skill in the art through comparison of such approaches with the aspects of the present methods and systems set forth in the remainder of this disclosure, with reference to the drawings.

[0005] INCORPORATION BY REFERENCE

[0006] U.S. Patent Application No. 15 / 243,519, entitled “Distributed Erasure Coded Virtual File System,” is incorporated by reference in its entirety. SUMMARY

[0007] Methods and systems are provided for a distributed file system including GPUs, substantially as shown in, and / or described in connection with, at least one of the figures, as more fully described in the claims. BRIEF DESCRIPTION OF DRAWINGS

[0008] Figure 1 Various example configurations of a distributed file system in accordance with aspects of the present disclosure are shown.

[0009] Figure 2 An example configuration of a distributed file system node in accordance with aspects of the present disclosure is shown.

[0010] Figure 3 Another representation of a distributed file system according to example implementations of the present disclosure is shown.

[0011] Figure 4 An example of a distributed file system including GPU-based servers according to example implementations of the present disclosure is shown.

[0012] Figure 5 is a flowchart showing an example method for using GPU-based servers in a distributed file system.

[0013] Figure 6 An example implementation of two distributed failure resilient address spaces residing on multiple solid state storage disks is shown.

[0014] Figure 7 A forward error correction scheme that can be used to protect data stored to non-volatile memory of a virtual file system according to example implementations of the present disclosure is shown. DETAILED DESCRIPTION

[0015] Traditionally, file systems use centralized control of metadata structures (e.g., directories, files, attributes, file contents). If a local file system is accessible from a single server, and that server fails, the data of the file system can be lost due to lack of further protection. To increase protection, some file systems (e.g., as provided by NetApp) have used one or more pairs of controllers to replicate metadata across two or more computers in an active-passive manner. Other solutions have used multiple metadata servers in a clustered manner (e.g., as provided by IBM GPFS, Dell EMC Isilon, Lustre, etc.). However, such systems cannot scale due to the number of metadata servers in traditional clustered systems being limited to a small number.

[0016] The system in this disclosure is suitable for small clusters and can also scale to many, thousands of nodes. Example implementations are discussed with respect to non-volatile memory (NVM) (e.g., flash memory in the form of a solid state drive (SSD)). The NVM can be divided into 4 kB blocks and 128 MB chunks. Extents can be stored in volatile memory (e.g., RAM for fast access) and can also be backed up by NVM. Extents can store pointers to blocks, e.g., 256 pointers to 1 MB data stored in a block. In other implementations, larger or smaller memory divisions can also be used. Metadata functions in this disclosure can be efficiently distributed across many servers. For example, in the case of a "hot spot" where a large load targets a specific portion of the namespace of the file system, the load can be distributed across multiple nodes.

[0017] Figure 1 Various example configurations of a distributed file system are shown in accordance with aspects of the disclosure. Figure 1 A local area network (LAN) 102 is shown in FIG. 1, which includes one or more nodes 120 (for J > 1, indexed by integers from 1 to J) and, optionally, one or more dedicated storage nodes 106 (for M > 1, indexed by integers from 1 to M), one or more compute nodes 104 (for N > 1, indexed by integers from 1 to N), and / or an edge router connecting the LAN 102 to a remote network 118. The remote network 118 optionally includes one or more storage services 114 (for K > 1, indexed by integers from 1 to K) and / or one or more dedicated storage nodes 115 (for L > 1, indexed by integers from 1 to L).

[0018] Each node 120 j (j is an integer, where 1 < j < J) is a networked computing device (e.g., a server, personal computer, etc.) that includes circuitry for running processes (e.g., client processes) directly on the operating system of the device 104 n and / or in one or more virtual machines running in the device 104 n .

[0019] The compute node 104 is a networked device that can run a virtual frontend without a virtual backend. The compute node 104 can run a virtual frontend by bringing single root input / output virtualization (SR-IOV) into a network interface card (NIC) and consuming a full processor core. Alternatively, the compute node 104 can run a virtual frontend by routing networking via the Linux kernel network stack and using kernel process scheduling, thus without the full core requirement. This is useful if the user does not want to allocate a full core for the file system or if the networking hardware is not compatible with the file system requirements.

[0020] Figure 2 An example configuration of a node is shown in accordance with aspects of the present disclosure. The node includes a frontend 202 and driver 208, a memory controller 204, a backend 206, and an SSD agent 214. The frontend 202 can be a virtual frontend; the memory controller 204 can be a virtual memory controller; the backend 206 can be a virtual backend; and the driver 208 can be a virtual driver. As used in this disclosure, a virtual file system (VFS) process is a process that implements one or more of the frontend 202, the memory controller 204, the backend 206, and the SSD agent 214. Thus, in example implementations, resources of the node (e.g., processing and memory resources) can be shared between client processes and VFS processes. The processes of the VFS can be configured to require a relatively small amount of resources to minimize the impact on the performance of client applications. The frontend 202, the memory controller 204, and / or the backend 206 and / or the SSD agent 214 can run on a processor of the host 201 or on a processor of the network adapter 218. For multi-core processors, different VFS processes can run on different cores, and can run different subsets of the services. From the perspective of the client processes 212, the interface to the virtual file system is independent of the particular physical machine on which the VFS process is running. The client processes only need the driver 208 and the frontend 202 to exist in order for them to be served.

[0021] The node can be implemented as a single-tenant server (e.g., bare metal) running directly on an operating system or as a virtual machine (VM) and / or container (e.g., Linux container (LXC)) within a bare metal server. The VFS can run within an LXC container that is a VM environment. Thus, inside the VM, the only thing that can run is the LXC container that includes the VFS. In a classic bare metal environment, there is a user space application, and the VFS runs in an LXC container. If the server is running other containerized applications, the VFS can run within an LXC container that is outside the management scope of a container deployment environment (e.g., Docker).

[0022] The nodes can be served by an operating system and / or a virtual machine monitor (VMM) (e.g., hypervisor). The VMM can be used to create and run the nodes on the host 201. Multiple cores can reside within a single LXC container running the VFS, and the VFS can run using a single Linux kernel on a single host 201. Thus, a single host 201 can include multiple front-ends 202, multiple memory controllers 204, multiple back-ends 206, and / or one or more drives 208. The drives 208 can run in kernel space outside the scope of the LXC container.

[0023] SR-IOV PCIe virtual functions can be used to run the networking stack 210 in user space 222. SR-IOV allows for the isolation of PCI Express such that a single physical PCI Express can be shared across a virtual environment and different virtual functions can be provided to different virtual components on a single physical server machine. The I / O stack 210 enables the VFS nodes to bypass the standard TCP / IP stack 220 and communicate directly with the network adapters 218. A portable operating system interface for uniX (POSIX) VFS functionality can be provided to the VFS driver 208 through lockless queues. SR-IOV or full PCIe physical function addresses can also be used to run a non-volatile memory express (NVMe) driver 214 in user space 222, thereby completely bypassing the Linux IO stack. NVMe can be used to access non-volatile storage 216 attached via a PCI Express (PCIe) bus. The non-volatile storage 220 can be flash memory, for example, in the form of a solid state drive (SSD) or storage class memory (SCM) that can appear in the form of a SSD or a memory module (DIMM). Other examples can include storage class memory technologies such as 3D-XPoint.

[0024] By coupling the physical SSD 216 with the SSD agent 214 and networking 210, the SSD can be implemented as a networked device. Optionally, the SSD can be implemented as a network-attached NVMe SSD 242 or 244 by using a network protocol such as NVMe-oF (NVMe over Fabrics). NVMe-oF can allow for the use of redundant network links to access the NVMe device, thereby providing a higher level or resiliency. The network adapters 226, 228, 230, and 232 can include hardware acceleration for connecting to the NVMe SSDs 242 and 244 to convert them to networked NVMe-oF devices without using the server. The NVMe SSDs 242 and 244 can each include two physical ports, and all data can be accessed through either of the ports.

[0025] Each client process / application 212 can run directly on the operating system or can run in a virtual machine and / or container served by the operating system and / or hypervisor. The client processes 212 can read data from storage and / or write data to storage in the course of performing their primary functions. However, the primary functions of the client processes 212 are not related to storage (i.e., the processes are only concerned that their data is reliably stored and is retrievable when needed, and are not concerned with where, when, or how the data is stored). Example applications that give rise to such processes include: email servers, web servers, office productivity applications, customer relationship management (CRM), animated video rendering, genomic computing, chip design, software builds, and enterprise resource planning (ERP).

[0026] The client applications 212 can make system calls to the kernel 224 that communicate with the VFS driver 208. The VFS driver 208 places corresponding requests on a queue of the VFS frontend 202. If there are several VFS frontends, the driver can load balance access to different frontends, ensuring that a single file / directory is always accessed via the same frontend. This can be done by sharding the frontends based on the ID of the file or directory. The VFS frontend 202 provides an interface for routing file system requests to the appropriate VFS backend based on the bucket responsible for the operation. The appropriate VFS backend can be on the same host or can be on another host.

[0027] The VFS backend 206 hosts several buckets, each of which services file system requests that receive and execute tasks, otherwise manage the virtual file system (e.g., load balancing, logging, maintaining metadata, caching, moving data between tiers, removing stale data, correcting corrupt data, etc.).

[0028] The VFS SSD agent 214 handles interactions with the respective storage devices 216. This can include, for example, translating addresses and generating commands that are issued to the storage devices (e.g., over SATA, SAS, PCIe, or other suitable buses). Thus, the VFS SSD agent 214 operates as an intermediary between the storage devices 216 and the VFS backend 206 of the virtual file system. The SSD agent 214 can also communicate with standard network storage devices that support standard protocols such as NVMe-oF (NVMe over Fabrics).

[0029] Figure 3 Another representation of a distributed file system is shown in accordance with an example implementation of the present disclosure. In this example, the VFS frontend 202 is shown as a single instance, but it can be distributed across multiple hosts. The VFS backend 206 is shown as a single instance, but it can be distributed across multiple hosts. The VFS SSD agent 214 is shown as a single instance, but it can be distributed across multiple hosts. Figure 3In this context, element 302 represents the memory resources (e.g., DRAM and / or other short-term memory) and processing resources (e.g., x86 processors, ARM processors, NICs, ASICs, FPGAs, etc.) on which various nodes (compute, storage, and / or VFS) of a virtual file system reside, such as those mentioned above. Figure 2 As described. Element 308 represents one or more physical storage devices 216 that provide long-term storage for a virtual file system.

[0030] like Figure 3 As shown, the physical storage is organized into multiple Distributed Fault Resilient Address Spaces (DFRAS) 318. Each space comprises multiple data blocks 310, and each data block 310 further comprises multiple blocks 312. Organizing blocks 312 into data blocks 310 is convenient only in some implementations and may not be done in all implementations. Each block 312 stores committed data 316 (which may present various states discussed below) and / or metadata 314 describing or referencing the committed data 316.

[0031] Organizing storage 308 into multiple DFRAS enables high-performance parallel commits from many (perhaps all) nodes of the virtual file system (e.g., Figure 1 All nodes 1041 to 104 N 1061 to 106 M and 1201 to 120 J (Parallel commits can be performed simultaneously). In the example implementation, each node in the virtual file system can own one or more corresponding DFRAS from multiple DFRAS, and has exclusive read / commit access to the DFRAS it owns.

[0032] Each bucket has its own DFRAS, and therefore no coordination with any other nodes is required when writing to it. Each bucket can build stripes on many different data blocks across many different SSDs, so each bucket with a DFRAS can select the "data block stripe" to write to based on many parameters, and once a data block is assigned to that bucket, no coordination is required. All buckets can efficiently write to all SSDs without any coordination.

[0033] Each DFRAS owned and accessible only by its owner bucket running on a particular node allows each of the nodes of the VFS to control a portion of the storage 308 without having to coordinate with any other node (except during initialization or after a node failure to maintain the [re]allocation of buckets of DFRASes, e.g., this can be performed asynchronously from the actual reads / commits to the storage 308). Thus, in such implementations, each node can read / commit to the DFRASes of its bucket independently of what the other nodes are doing without needing to reach any consensus in reading and committing to the storage 308. Moreover, in the case of a particular node failure, the fact that a particular node owns multiple buckets allows for a more intelligent and efficient reallocation of its workload to other nodes (rather than having to allocate the entire workload to a single node, which can create “hot spots”). In this regard, in some implementations, the number of buckets can be large relative to the number of nodes in the system, such that any one bucket can be a relatively small load placed on another node. This allows for a fine-grained reallocation of the workload of a failed node according to the capabilities and capacities of other nodes (e.g., a higher percentage of the failed node buckets can be given to nodes with more capabilities and capacities).

[0034] To allow for such operations, metadata mapping each bucket to its current owning node can be maintained, such that reads and commits to the storage 308 can be redirected to the appropriate node.

[0035] Load distribution is possible because the entire file system metadata space (e.g., directories, file attributes, content ranges in files, etc.) can be broken up (e.g., split or fragmented) into small, uniform pieces (e.g., “fragments”). For example, a large system with 30k servers can split the metadata space into 128k or 256k fragments.

[0036] Each such metadata fragment can be maintained in a “bucket.” Each VFS node can be responsible for several buckets. When a bucket serves a metadata fragment on a given backend, that bucket is considered to be “active” or the “leader” of that bucket. Typically, there are many more buckets than VFS nodes. For example, a small system with 6 nodes can have 120 buckets, and a large system with 1,000 nodes can have 8k buckets.

[0037] Each bucket can be active on a small set of nodes, typically the 5 nodes that form the quintuple of that bucket. The cluster configuration keeps all participating nodes up to date on the quintuple assignment for each bucket.

[0038] Each quintuple watches itself. For example, if a cluster has 10k servers and each server has 6 buckets, each server will only need to talk to 30 different servers to maintain the state of its buckets (6 buckets will have 6 quintuples, so 6 = 30). This is a much smaller number than a centralized entity would have to watch all nodes and keep cluster-wide state. Using quintuples allows performance to scale with larger clusters because nodes do not do more work as the cluster size increases. This can cause the disadvantage that small clusters in "dumb" mode can actually generate more communication than physical nodes, but that disadvantage is overcome by sending only a single heartbeat between two servers that have all their shared buckets (this will change to only one bucket as the cluster grows, but if you have a small 5 server cluster, just include all buckets in all messages and each server will only talk to the other 4 servers). Quintuples can use an algorithm similar to the Raft consensus algorithm to decide (i.e., agree).

[0039] Each bucket can have a group of compute nodes that can run that bucket. For example, five VFS nodes can run a bucket. However, at any given moment, only one of the nodes in the group is the controller / leader. In addition, for a large enough cluster, no two buckets share the same group. If there are only 5 or 6 nodes in the cluster, most buckets can share the back end. In a fairly large cluster, there are many different groups of nodes. For example, for 26 nodes, there are more than 64,000 (26 ) possible groups of five nodes (i.e., quintuples).

[0040] All nodes in the group know and agree (i.e., agree) which node is the actual active controller (i.e., leader) of that bucket. A node accessing that bucket can remember ("cache") the last node that was the leader of that bucket in the group (e.g., five) members. If it accesses the bucket leader, the bucket leader performs the requested operation. If it accesses a node that is not the current leader, the node indicates that the leader "redirect" the access. If there is a timeout of the leader node accessing the cache, the contacting node can try a different node of the same quintuple. All nodes in the cluster share a common "configuration" of the cluster that allows the nodes to know which server can run each bucket.

[0041] Each bucket can have a load / usage value that indicates how much the bucket is being used by applications running on the file system. For example, a server node with 11 lightly used buckets can receive another bucket of metadata to run before a server with 9 heavily used buckets, even though the number of used buckets will be unbalanced. The load value can be determined from average response latency, number of parallel running operations, memory consumed, or other metrics.

[0042] Redistribution can occur even when VFS nodes are not failing. If the system identifies that one node is busier than others based on tracked load metrics, the system can move (i.e., “failover”) one of its buckets to another server that is less busy. However, before actually relocating the bucket to a different host, load balancing can be achieved by shifting writes and reads. Since each write can end on a different group of nodes decided by the DFRAS, a node with higher load can not be selected to be in the stripe that is being written to. The system can also choose not to service reads from the high load node. For example, a “degraded mode read” can be performed, where a block in the high load node is reconstructed from other blocks of the same stripe. The degraded mode read is a read performed via the remaining nodes in the same stripe and reconstructing the data via fail-safes. Degraded mode reads can be performed when read latency is too high, as the initiator of the read can assume that the node is down. If the load is high enough to create higher read latency, the cluster can revert to reading that data from other nodes and using degraded mode reads to reconstruct the data needed.

[0043] Each bucket manages its own instance of distributed erasure coding (i.e., DFRAS 518) and does not need to coordinate with other buckets to perform read or write operations. There can be thousands of concurrent distributed erasure coding instances working concurrently, each for a different bucket. This is a component of scaling performance as it effectively allows any large file system to be divided into independent pieces that do not need coordination, thus providing high performance regardless of scale.

[0044] Each bucket handles all file system operations that fall into its shards. For example, directory structures, file attributes, and file data ranges will fall under the jurisdiction of a particular bucket.

[0045] Operations completed from any front end start by finding out what bucket owns the operation. Then, the back end leader and node of that bucket are determined. The determination can be performed by trying the last known leader. If the last known leader is not the current leader, the node can know which node is the current leader. If the last known leader is no longer part of the bucket's quintuple, the back end will let the front end know that it should go back to the configuration used to find the members of the bucket's quintuple. The distribution of operations allows complex operations to be handled by multiple servers rather than by a single computer in a standard system.

[0046] If the size of the cluster is small (e.g., 5) and the quintuple is used, there will be buckets that share the same group. As the cluster size grows, the buckets are reassigned so that no two groups are the same.

[0047] A graphics processing unit (GPU) is a specialized electronic circuit designed to rapidly manipulate and alter memory to accelerate the creation of images in a frame buffer intended for output to a display device. GPUs are used in embedded systems, mobile phones, personal computers, workstations, and game consoles. Modern GPUs are very efficient at manipulating computer graphics and image processing, and their highly parallel structure makes them more effective than general-purpose CPUs for algorithms where processing of large blocks of data is done in parallel. In personal computers, a GPU can exist on a video card, or it can be embedded on the motherboard or in the CPU die in some CPUs.

[0048] The term GPU was popularized by NVIDIA. NVIDIA's GPUs were presented as "single-chip processors with integrated transform, lighting, triangle setup / culling, and rendering engines."

[0049] Figure 4 An example of a distributed file system including a GPU-based server 409 according to an example implementation of the present disclosure is shown. In addition to the GPU-based server 409, the distributed file system includes a computing device (e.g., CPU-based server 401) and a non-volatile system memory 419 (e.g., multiple SSDs 427a, 427b, and 427c).

[0050] The CPU-based server 401 includes a front end 403 and a back end 405. The back end 405 includes at least one bucket 407. The multiple SSDs 427a, 427b, and 427c can be configured as multiple blocks, e.g., block a, block b, and block c.

[0051] Each bucket in the back end can operate to build one or more failure resilient stripes 429 that include a plurality of chunks. For example, for 10 chunks, 8 data chunks can be protected with 2 error protection / correction chunks (i.e., using an 8+2 stripe). Likewise, for 10 failure domains, 6 data chunks can be protected with 4 error protection / correction chunks (i.e., using a 6+4 stripe).

[0052] To illustrate, 3 storage devices and 1 stripe are shown in Figure 4 Bucket 407 can operate to build a failure resilient stripe 429 that includes chunk a, chunk b, and chunk c. Each of the plurality of storage chunks in a particular failure resilient stripe can be located in a different storage device of the plurality of storage devices.

[0053] GPU-based server 409 can be connected to CPU-based server 401 through PCIe interface 417 and can access system memory 419 via system memory bus 423. However, when GPU 409 performs computations, GPU 409 uses dedicated GPU memory 421 via graphics memory bus 425. For example, the dedicated memory of NVIDIA’s GPUs is RAM that is connected directly to the GPU with a high-bandwidth memory interface. Thus, data stored on GPU memory 421 can be transferred from system memory 419 through PCIe interface 417.

[0054] Data transferred from SSDs 427a, 427b, and 427c can first be placed in kernel page cache 431. Then, data can be copied from kernel page cache 431 to application user space memory 433 in system main memory 419. Finally, data can be copied from application user space memory 433 into GPU memory 421. Since many GPU applications do not need to access system main memory, there are two wasted copies. Moreover, these GPU applications are very sensitive to performance, so it is important to transfer data directly to GPU memory.

[0055] Portable Operating System Interface (POSIX) is a series of standards specified by the IEEE Computer Society for maintaining compatibility between operating systems. When writing a standard POSIX application, they assume that files remain open for a long time. The standard POSIX process includes finding an index node (a series of lookup requests), opening a file, accessing the file, and finally closing. The standard POSIX process is useful if files are open for a long time.

[0056] However, GPU-based applications deal with large numbers of small files in GPU memory (e.g., images, voice samples, text snippets, etc.). The POSIX overhead in GPU-based applications is quite high.

[0057] According to one or more embodiments of the present disclosure, when an application uses GPU data, the data can be read directly from a distributed file system that includes both system memory 419 and graphics memory 421. For example, graphics processing device 409 can operate to transfer storage blocks a, b, or c to volatile storage device 421 while bypassing kernel page cache 431. A GPU-based application can be provided to place the contents of small files in the user space memory 433 of the application, thereby bypassing the POSIX protocol and kernel page cache 431 and adjusting efficient key-value access. Further, graphics processing device 409 can operate to access multiple storage blocks in parallel from SSDs 427a, 427b, and 427c via a high bandwidth interface.

[0058] A GPU-based application can provide a string of full file names (e.g., “ / dir1 / dir2 / ... / dirN / file name”) and memory locations. The system will communicate directly with the user space frontend and bypass kernel drivers completely. The system can fetch the contents of the GPU-based application files in parallel and place them in GPU memory 421, thereby eliminating POSIX overhead. Further, by eliminating the need to copy to and from kernel page cache 431, latency can be reduced. The GPU application can copy data to GPU memory 421 through PCIe interface 417 and notify GPU 409 that processing can begin (e.g., by ringing a doorbell). For example, computing device frontend 403 can indicate to the GPU and / or GPU-based server 409 that the data required for a graphics operation is available in volatile storage device 421.

[0059] Once frontend 403 verifies that the data is fully placed in GPU memory 421, frontend 403 can notify the GPU application that the data is there, so the user application can launch the GPU program to act on that data. However, since waking up the GPU client application also takes time and increases latency, the application can store the GPU program in the GPU and the GPU can wait for a doorbell (i.e., an indication to the hardware) that all data is placed in the correct location. Frontend 403 can then ring the doorbell of the GPU to begin processing the data.

[0060] Data can also be written directly into GPU memory 421. Instead of instructing CPU-based server 401 (e.g., on a network interface card) to write incoming packets to system memory 419, data packets can be written to GPU memory 421 using GPU direct and RDMA techniques. With direct GPU memory access, key value access places data directly in GPU memory 421.

[0061] GPU-based server 409 can include a GPU front end 411 and a GPU back end 413. GPU back end 413 can include at least one GPU bucket 415. Buckets 415 on GPU back end 413 can operate to manage data in volatile storage 421. Computing device 401 and graphics processing device 409 can be coupled via network interface 435. GPU front end 411 can determine the location of data required for a graphics operation. This data can be received directly from the computing device over network interface 435. For example, GPU front end 411 can communicate with front end 403, which can determine that data required for a graphics operation is located in a fault-protected stripe 429 directed by bucket 407 in back end 405 of computing device 401. GPU front end 411 can indicate to a graphics processing unit (GPU) when data required for a graphics operation is available in volatile storage 421.

[0062] Figure 5 is a flowchart illustrating an example method for using GPUs in a distributed file system. In block 501, a plurality of fault-protected stripes is constructed using a computing device such that each of the plurality of fault-protected stripes includes a plurality of storage blocks, and each of the plurality of storage blocks is located in a different flash storage device of a plurality of flash storage devices.

[0063] In block 503, a location of data for a graphics operation is determined. If the data for the graphics operation is located in a fault-protected stripe, then in block 507, graphics data in one or more blocks of the fault-resilient stripe is transferred to volatile storage. This data can be transferred in parallel over a high-bandwidth bus that bypasses a kernel page cache. In block 509, the transferred data can be subjected to a graphics operation using a graphics processing device.

[0064] If the data for the graphics operation is not located in a fault-protected stripe, then in block 505, graphics data can be transferred directly from a network to volatile storage via a network interface. Once the data required for the graphics operation is available in volatile storage, the availability is indicated to a GPU.

[0065] Figure 6 An example implementation is shown in which two distributed fault-resilient address spaces reside on a plurality of solid state storage disks.

[0066] Data block 9101,1 Up to 910 D,C It can be organized into multiple data block stripes 9201 to 920. S (S is an integer). In the example implementation, forward error correction (e.g., erasure coding) is used to individually protect each data block stripe 920. s (s is an integer, where 1 ≤ s ≤ S). Therefore, any particular data block stripe 920 can be determined based on the desired data protection level. s Data block 910 d,c The quantity.

[0067] For illustrative purposes, assume each data block has 920 stripes. s Includes N = M + K (where each of N, M, and K is an integer) data blocks 910 d,c Then there are N data blocks 910 d,c M data blocks 910 d,c It can store data bits (usually binary bits or "bits" of the current storage device) and N data blocks 910 d,c K data blocks 910 d,c It can store protection bits (again, typically bits). Then, the virtual file system can store N data blocks from N different fault domains. d,c 920 allocated to each strip s .

[0068] As used herein, a "fault domain" refers to a set of components in which the failure of any single component (loss of power, becoming unresponsive, etc.) could cause all components to fail. For example, if a rack has a single top-of-rack switch, the failure of that switch would reduce the connectivity of all components on that rack (e.g., compute, storage, and / or VFS nodes). Therefore, for the rest of the system, it is equivalent to whether all components on that rack fail together. The virtual file system according to this disclosure may include fewer fault domains than data block 910.

[0069] In an example implementation where the nodes of the virtual file system are connected and powered in a fully redundant manner with only a single storage device 906 per such node, the fault domain can be limited to that single storage device 906. Therefore, in the example implementation, each data block stripe 920 s Including those residing in storage devices 9061 to 906 D Multiple data blocks 910 on each of the N storage devices d,c (Therefore, D is greater than or equal to N). In Figure 6 An example of this implementation is shown in the figure.

[0070] existFigure 6 In this configuration, D=7, N=5, M=4, K=1, and the storage is organized into two DFRAS. These numbers are for illustrative purposes only and are not intended to be limiting. For illustration, three data block stripes 920 of the first DFRAS are arbitrarily invoked. The first data block stripe 9201 consists of data block 910. 1,1 910 2,2 910 3,3 910 4,5 and 910 5,6 Composition; the second data block stripe 9202 is composed of data block 910 3,2 910 4,3 910 5,3 910 6,2 and 910 7,3 Composed of; and the third data block stripe 9203 is composed of data block 910. 1,4 910 2,4 910 3,5 910 5,7 and 910 7,5 composition.

[0071] Despite Figure 6 In the simple example, D=7 and N=5, but in actual implementations, D can be much larger than N (e.g., a multiple of an integer greater than 1, and possibly up to many orders of magnitude), and two values ​​can be chosen such that the probability that any two data block stripes 920 of a single DFRAS reside on the same set of N storage devices 906 (or more generally, on the same set of N fault domains) is below a desired threshold. In this way, any single storage device 906 d A failure of any single fault domain (or more generally, any single fault domain) will result in the loss of any particular stripe 920 (the expected statistics may be determined based on the selected values ​​of D and N, the size of the N storage devices 906, and the arrangement of the fault domains). s The maximum number of data blocks is 910. b,c Furthermore, a double failure will result in the loss of the vast majority of stripes, at most a single data block (910). b,c And only a small number of bands (determined based on the values ​​of D and N) will lose any particular band 920 s The number of two data blocks in the data (e.g., the number of two fault stripes may be exponentially less than the number of one fault stripe).

[0072] For example, if each storage device has 906 d If it's 1TB and each data block is 128MB, then the storage device is 906. dThe failure will result in the loss of one data block 910 out of 7812 (=1TB / 128MB) data block stripes 920 (the expected statistics may be determined based on the selected values ​​of D and N, the size of N storage devices 906, and the arrangement of the failure domain). For each such affected data block stripe 920 s Appropriate forward error correction algorithms and data block striping 920 can be used. s Use the other N-1 data blocks to quickly reconstruct the lost data block 910 d,c Furthermore, because the 7812 affected data block stripes are evenly distributed across all storage devices 9061 to 906... D Above, reconstruct the 7812 lost blocks 910 d,c The expected statistics will be determined based on (possibly based on the selected values ​​of D and N, the size of the N storage devices 906, and the arrangement of the fault domain) from storage devices 9061 to 906. D Each device reads the same amount of data (i.e., the burden of reconstructing lost data is evenly distributed across all storage devices 9061 to 906). D This is to enable very rapid recovery from failures.

[0073] Next, proceed to storage devices 9061 to 906. D In the case of concurrent failures of two storage devices (or more generally, concurrent failures of two fault domains), due to the data block stripes 9201 to 920 in each DFRAS. S In all storage devices 9061 to 906 D The data is evenly distributed across the surface, with only a very small number of data block stripes from 9201 to 920. S Two of its N data blocks will be lost. The virtual file system can operate based on the indicator data block stripes 9201 to 920. S With storage devices 9061 to 906 D The metadata mapping between the two missing data block stripes is used to quickly identify such stripes. Once these two missing data block stripes are identified, the virtual file system can prioritize reconstructing those two missing data block stripes before starting to reconstruct one missing data block stripe. The remaining data block stripes will have only a single missing data block, and for them (the vast majority of affected data block stripes), two storage devices 906 d Concurrent failures with only one storage device 906 d The faults are the same. A similar principle applies to third concurrent faults (in two concurrent fault scenarios, the number of data block stripes with three fault blocks is even less than the number of data block stripes with two fault blocks), etc. In the example implementation, it can be based on data block stripe 920. sthe number of missing data in the data block stripe 920 s . For example, this can be accomplished by controlling the rate at which reads and commits for reconstruction are performed, the rate at which FEC calculations for reconstruction are performed, the rate at which network messages for reconstruction are transmitted, etc.

[0074] Figure 7 A forward error correction scheme that can be used to protect data stored to non-volatile memory of a virtual file system is shown in accordance with an example implementation of the present disclosure. Storage blocks 902 of block stripes 9301-9304 of a DFRAS are shown 1,1 to 902 7,7 . In the protection scheme of Figure 7 , five blocks of each stripe are used to store data digits and two blocks of each stripe are used to store data storage protection digits (i.e., M = 5 and K = 2). In Figure 7 , the protection digits are calculated using the following equations (1) through (9): P1 = D11 D22 D33 D44 D54 (1) P2 = D21 D32 D43 D53 D14 (2) P3 = D31 D42 D52 D13 D24 (3) P4 = D41 D51 D12 D23 D34 (4) Z = D51 D52 D53 D54 (5) Q1 = D11 D12 D13 D14 Z (6) Q2 = D21 D22 D23 D24 Z (7) Q3 = D31 D32 D33 D34 Z (8) Q4 = D41 D42 D43 D44 Z (9) Thus, Figure 7 The four stripes 9301-9304 in are part of a multi-stripe (in this case, four stripe) FEC protected domain, and various combinations of the above equations (1)-(9) can be used to recover the loss of two or fewer blocks in any of the block stripes 9301-9304. For comparison, an example of a single stripe protected domain would be if D11, D22, D33, D44, D54 were protected by P1 alone, and D11, D22, D33, D44, D54 and P1 were all written to stripe 9301 (9301 would be a single stripe FEC protected domain).

[0075] According to example implementations of the present disclosure, a plurality of computing devices are communicatively coupled to each other via a network, and each of the plurality of computing devices includes one or more of a plurality of storage devices. A plurality of failure resilient address spaces are distributed across the plurality of storage devices such that each of the plurality of failure resilient address spaces spans the plurality of storage devices. Each of the plurality of failure resilient address spaces is organized as a plurality of stripes (e.g., Figure 6 and Figure 7 The plurality of stripes 930 in each of the plurality of failure resilient address spaces. Each one or more of the plurality of stripes is part of a respective one of a plurality of forward error correction (FEC) protected domains (e.g., such as the multi-stripe FEC domains in Figure 6 The plurality of stripes can each include a plurality of storage blocks (e.g., the plurality 912). Each block of a particular stripe of the plurality of stripes can reside on a different storage device of the plurality of storage devices. A first portion of the plurality of storage blocks (e.g., the five quantities of stripes 9301-9304 of Figure 7 The two 902 1,2 and 902 1,6 of stripe 9301 of Figure 7 The two 902 1,1 and 902 1,7 of stripe 9301 of

[0076] A plurality of computing devices can operate to order a plurality of stripes. The ordering can be used to select which stripe of the plurality of stripes to use for a next commit operation for one of a plurality of failure resilient address spaces. The ordering can be based on how many protected and / or unprotected storage blocks are in each of the plurality of stripes. For any particular stripe of the plurality of stripes, the ordering can be based on a bitmap stored on a plurality of storage devices having the particular stripe of the plurality of stripes. The ordering can be based on how many blocks of data are currently stored in each of the plurality of stripes. The ordering can be based on read and write overhead for committing to each of the plurality of stripes. At any given time, each of the plurality of failure resilient address spaces can be owned by only one of the plurality of computing devices, and each of the plurality of failure resilient address spaces can be read and written by only its owner. Each of the computing devices can own a plurality of failure resilient address spaces. The plurality of storage devices can be organized into a plurality of failure domains. Each of the plurality of stripes can span the plurality of failure domains. Each of the plurality of failure resilient address spaces can span all of the plurality of failure domains, such that in the event of failure of any particular failure domain of the plurality of failure domains, a workload to reconstruct lost data is distributed in each of the other failure domains of the plurality of failure domains. The plurality of stripes can be distributed across the plurality of failure domains such that in the event of concurrent failure of two failure domains of the plurality of failure domains, the chance that two blocks of any particular stripe of the plurality of stripes reside on the two failure failure domains of the plurality of failure domains is exponentially smaller than the chance that only one block of any particular stripe of the plurality of stripes resides on the two failure failure domains of the plurality of failure domains.

[0077] The plurality of computing devices can operate to reconstruct any of the plurality of stripes having two failed blocks first, and then reconstruct any of the plurality of stripes having only one failed block. The plurality of computing devices can operate to perform reconstruction of the plurality of stripes having two failed blocks at a higher rate than reconstruction of the plurality of stripes having only one failed block (e.g., having a greater percentage of CPU clock cycles dedicated to reconstruction, a greater percentage of network transfer opportunities dedicated to reconstruction, etc.). The plurality of computing devices can operate to determine a rate at which to reconstruct any particular lost block based on how many other blocks of the same stripe of the plurality of stripes have already been lost in the event of failure of one or more failure domains. Wherein the one or more failure domains comprise a plurality of storage devices. Each of the plurality of FEC protection domains can span a number of stripes of the plurality of stripes.

[0078] The plurality of stripes can be organized into a plurality of groups (e.g., as Figure 6 data block stripes 9201-920 S), wherein each of the plurality of groups includes one or more of the plurality of stripes, and the plurality of computing devices are operable to order, for each of the groups, one or more of the plurality of stripes of the group. The plurality of computing devices can be operable to perform a continuous commit operation on a selected group of the plurality of groups until one or more of the plurality of stripes in the group no longer satisfy the determined criteria, and to select a different group of the plurality of groups when the selected group of the plurality of groups no longer satisfies the determined criteria. The criteria can be based on how many chunks are available for new data to be written.

[0079] While the present method and / or system has been described with reference to certain implementations, it will be understood by those skilled in the art that various changes can be made and equivalents can be substituted for elements thereof without departing from the scope of the present method and / or system. In addition, many modifications can be made to adapt a particular situation or material to the teachings of the present disclosure without departing from its central scope. Therefore, it is intended that the present method and / or system not be limited to the particular implementations disclosed as the optimal only, but that the present method and / or system will include all implementations falling within the scope of the appended claims.

[0080] As used herein, the terms “circuits” and “circuitry” refer to physical electronic components (i.e., hardware) and any software and / or firmware (“code”) which can configure the hardware, be executed by the hardware, and / or otherwise be associated with the hardware. As used herein, for example, a particular processor and memory can comprise first “circuitry” when executing a first one or more lines of code and can comprise second “circuitry” when executing a second one or more lines of code. As used herein, “and / or” means any one or more of the items in the list joined by “and / or”. As used herein “x and / or y” means any element of the three-element set {(x), (y), (x, y)}. In other words, “x and / or y” means “one or both of x and y”. In other words, “x, y, and / or z” means any element of the seven-element set {(x), (y), (z), (x, y), (x, z), (y, z), (x, y, z)}. In other words, “x, y, and / or z” means “one or more of x, y and z”. As used herein, the term “exemplary” means serving as a non-limiting example, instance, or illustration. As used herein, the terms “e.g.,” and “for example,” set off lists of one or more non-limiting examples, instances, or illustrations. As used herein, circuitry is “operable to” perform a function whenever the circuitry comprises hardware and code that, when executed by the hardware, permit performance of that function, whether or not the function is actually performed, enabled to be performed, disabled, or un-enabled (e.g., by a user-configurable setting, factory trimming, etc.).

Claims

1. A system comprising: Multiple non-volatile storage devices are organized into multiple distributed fault-resistant address spaces (DFRAS), where each DFRAS is specifically associated with a corresponding bucket in multiple buckets of a distributed file system. The backend is configured to execute a multi-node consensus protocol to assign a leader to each bucket and to build multiple fault-protected stripes across the multiple non-volatile storage devices associated with the bucket for each bucket leader; Graphics processing device, including volatile storage device and GPU front end; and The GPU front end is configured as follows: Receive the filename corresponding to the data required for graphical operations; Obtain the bucket identifier that identifies the bucket leader storing the data from the front end of the distributed file system; Without traversing POSIX metadata and bypassing the kernel page cache, the blocks of the bucket leader's fault-protected stripe are directly transferred to the volatile storage device; and The backend is configured as follows: Maintain the load metric for the multiple fault-protected stripes of the bucket leader; and The plurality of fault protection strips are sorted based on the load metric, and the sorted strips are selected for submission.

2. The system according to claim 1, wherein, Each DFRAS includes logically distinct address regions that cannot be accessed by buckets other than the corresponding bucket.

3. The system according to claim 1, wherein, The multi-node consensus protocol includes a quorum-based leadership election among five groups of nodes.

4. The system according to claim 1, wherein, The fault protection stripe blocks are transferred to the volatile storage device using GPU Direct Memory Access.

5. The system according to claim 1, wherein, The fault protection stripe includes erasure coding segments distributed across multiple storage devices.

6. The system according to claim 1, wherein, The GPU front end sends a signal to the back end via a doorbell mechanism to initiate data acquisition.

7. The system according to claim 1, wherein, The backend is configured to update the stripe sorting based on at least one of the following: access frequency, write queue depth, wait time, and occupancy.

8. The system according to claim 1, wherein, The system is configured to retry transmission from the replication stripe when the bucket leader is overloaded.

9. The system according to claim 1, wherein, The commit operation includes writing the updated block into the sorted stripe while maintaining the erasure coding parity.

10. The system according to claim 1, wherein, Each bucket maintains an exclusive mapping between the file identifier and the corresponding DFRAS region.

11. The system according to claim 1, wherein, The GPU front-end bypass inode lookup, directory traversal, and virtual file system license verification are at least one of the following:

12. A method comprising: A multi-node consensus protocol is executed across multiple buckets in a distributed file system to assign a leader to each bucket; Multiple fault-protection stripes are constructed by the bucket leader across non-volatile storage devices specifically associated with the bucket; The filename of the data that is received and recognized in the GPU front end is used to identify graphics operations. Determine the bucket identifier and stripe of the bucket storing the data without traversing the POSIX metadata; The stripe blocks are directly transferred to GPU-accessible volatile memory, while bypassing the kernel page cache. The bucket leader sorts the multiple fault protection stripes based on load metrics; as well as The updated data is submitted to the strip selected according to the sorting.

13. The method of claim 12, comprising: The sorting of multiple stripes is updated based on the average access wait time.

14. The method according to claim 12, wherein, The transport block includes: Perform GPU direct DMA.

15. The method of claim 12, comprising: When the bucket leader is temporarily unavailable, the transport is redirected to the replication stripe.

16. The method according to claim 12, wherein, Determining the stripe includes: resolving the filename to stripe mapping exclusively maintained by the bucket leader.

17. The method of claim 12, comprising: Detect degradation mode conditions and select alternative stripes based on a threshold for reducing load.

18. A non-transitory machine-readable medium storing instructions that, when executed, cause a machine to perform operations including: The leader of the bucket in the distributed file system is assigned through a multi-node consensus protocol; Multiple fault protection stripes are built across non-volatile storage devices for each bucket leader; Receive filenames at the GPU front end; Determine the stripe storing data identified by the filename without traversing POSIX metadata; The stripe blocks are directly transferred to the GPU memory, while bypassing the kernel page cache; Sort multiple stripes based on load metrics; as well as The updated data is submitted to the strip selected according to the sorting.

19. The non-transitory machine-readable medium according to claim 18, wherein, The instruction causes the machine to perform load balancing selection for the replication stripe during a degradation operation.

20. The non-transitory machine-readable medium according to claim 18, wherein, The instruction causes the machine to update the stripe sorting based on the bucket-specific access history.

Images