Flash memory register with write leveling

Abstract

Flash memory registers with write leveling are disclosed. A plurality of computing devices are communicatively coupled to each other via a network, and each of the plurality of computing devices is operatively coupled to one or more of a plurality of storage devices. A plurality of failure resilient address spaces are distributed across the plurality of storage devices such that each of the plurality of failure resilient address spaces spans the plurality of storage devices. Each computing device is operable to save a secondary register of record changes in memory. When data is read from memory, the latest changes can be applied to the data according to one or more corresponding register blocks. As such, the secondary register can enable the plurality of computing devices to delay and / or consolidate writes to memory.

Description

[0001] This application is a divisional application, with its parent application having application number 201880058207.0, application date of October 4, 2018, and invention title "Flash Register with Write Balancing".

[0002] Cross-referencing

[0003] The entire contents of U.S. Patent Application No. 15 / 243,519, entitled “Distributed Erasure-Encoded Virtual File System,” are incorporated herein by reference. Background Technology

[0004] By comparing this scheme with some aspects of the current methods and systems described in the remainder of the accompanying drawings of this disclosure, the limitations and disadvantages of conventional data storage schemes will become apparent to those skilled in the art. Summary of the Invention

[0005] As set forth more fully as claimed, a method and system are provided for a NAND flash persistent register having write leveling as shown and / or described in at least one of the accompanying drawings. Attached Figure Description

[0006] Figure 1 Various exemplary configurations of a virtual file system according to aspects of this disclosure are shown.

[0007] Figure 2 An exemplary configuration of a virtual file system node according to aspects of this disclosure is shown.

[0008] Figure 3 Another representation of a virtual file system according to an exemplary embodiment of this disclosure is shown.

[0009] Figure 4 An embodiment of a flash register with write balancing according to an exemplary implementation of the present disclosure is shown.

[0010] Figure 5 Exemplary embodiments of the present disclosure are shown. Figure 4 An example of a shadow register associated with a flash register in a memory.

[0011] Figure 6 This is a flowchart illustrating a method for write balancing using flash registers in a load-balanced distributed storage system according to an exemplary embodiment of the present disclosure.

[0012] Figure 7 An exemplary implementation is shown in which two of the distributed fault-tolerant address spaces reside on multiple solid-state storage disks.

[0013] Figure 8 A forward error correction scheme, which can be used to protect data stored in non-volatile memory of a virtual file system, is illustrated according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0014] NAND flash memory devices support a limited number of writes over their lifetime, and the number of concurrent writes supported is lower than the number of concurrent reads supported. When a leaf is updated, the tree data structure requires updating all pointer layers, so traditional tree-based data structures are primarily used in designs. However, when stored on NAND flash memory, traditional tree-based data structures are inefficient because the number of writes to NAND flash memory is limited, making tree data structures unsuitable.

[0015] The data structure described in this disclosure makes converting writes to reads more efficient and cheaper for flash-based memories. This data structure provides write balancing and caching. Typically, reading is faster than writing. For example, some NAND flash-based devices perform ten times more reads than writes in a given time. Therefore, aspects of this disclosure provide a suitable data structure for NAND flash memory capable of efficiently converting write operations to read operations.

[0016] The system in this disclosure is suitable for small clusters and can also scale to many tens of thousands of nodes. Exemplary implementations with non-volatile memory (NVM) are discussed, such as flash memory in the form of solid-state drives (SSDs). The NVM can be divided into 4kB “blocks” and 128MB “sets”. “Extents” can be stored in volatile memory, such as RAM for fast access, also supported by NVM storage. An extent can store pointers to blocks; for example, storing 256 pointers as 1MB of data stored in a block. In other implementations, larger or smaller memory partitions can also be used. In this disclosure, metadata functionality can be efficiently distributed across multiple servers. For example, in the case of a “hotspot” where a large load locks a designated portion of the file system namespace, the load can be distributed across multiple nodes.

[0017] Figure 1 Various exemplary configurations of the Virtual File System (VFS) according to various aspects of this disclosure are shown. Figure 1The diagram shows a local area network (LAN) 102 comprising one or more VFS nodes 120 (represented by integers from 1 to J, j≥1), and optionally including (indicated by dashed lines): one or more dedicated storage nodes 106 (represented by integers from 1 to M, M≥1), one or more compute nodes 104 (represented by integers from 1 to N, N≥1), and / or an edge router connecting LAN 102 to a remote network 118. Optionally, the remote network 118 includes one or more storage services 114 (represented by integers from 1 to K, K≥1), and / or one or more dedicated storage nodes 115 (represented by integers from 1 to L, L≥1).

[0018] 120 per VFS node j (j is an integer, where 1 ≤ j ≤ J) is a network computing device (e.g., a server, personal computer, etc.), which includes processes for running the VFS process and, optionally, client processes (directly on device 104). n On the operating system and / or device 104 n (In one or more virtual machines running in the system).

[0019] Compute node 104 is a network device that can run a VFS frontend without a VFS backend. Compute node 104 can run a VFS frontend by integrating SR-IOV into the NIC and using a full processor kernel. Alternatively, compute node 104 can run a VFS frontend by routing the network via the Linux kernel network heap and using kernel process scheduling, thus eliminating the need for a full kernel. This is useful if the user does not want to distribute a full kernel for VFS, or if the network hardware is incompatible with VFS requirements.

[0020] Figure 2An exemplary configuration of a VFS node according to various aspects of this disclosure is shown. The VFS node includes a VFS frontend 202, a drive 208, a VFS storage controller 204, a VFS backend 206, and a VFS SSD agent 214. As used in this disclosure, a “VFS process” is a process that implements one or more of the following: VFS frontend 202, VFS storage controller 204, VFS backend 206, and VFS SSD agent 214. Thus, in the exemplary embodiment, VFS node resources (e.g., processing and memory resources) can be shared between client processes and VFS processes. VFS processes can be configured to require relatively few resources to minimize the impact on the operation of client applications. VFS frontend 202, VFS storage controller 204, and / or VFS backend 206, and / or VFS SSD agent 214 can run on the processor of host 201 or the processor of network adapter 218. For multi-core processors, different VFS processes can run on different cores and can run different subsets of services. For client process 212, the interface with the virtual file system depends on the specific physical machine on which the VFS process runs. The client process only needs the existence of drive 208 and frontend 202 to provide services to it.

[0021] A VFS node can be implemented as a single-tenant server running directly on an operating system (e.g., bare metal), or as a virtual machine (VM) and / or container (e.g., Linux container (LXC)) within a bare metal server. A VFS can run as a VM environment within an LXC container. Therefore, within a VM, the only thing that can run is the LXC container that includes the VFS. In a typical bare metal environment, user-space applications exist, and the VFS runs within an LXC container. If the server is running other containerized applications, the VFS can run within an LXC container outside the management scope of the container development environment (e.g., Docker).

[0022] An operating system and / or virtual machine monitor (VMM) (e.g., SuperMonitor) can provide services for VFS nodes. VFS nodes can be created and run on host 201 using a VMM. Multiple kernels can reside within a single LXC container running VFS, and VFS can run on a single host 201 using a single Linux kernel. Therefore, a single host 201 can include multiple VFS front-ends 202, multiple VFS storage controllers 204, multiple VFS back-ends 206, and / or one or more VFS drives 208. VFS drives 208 can run in kernel space outside the scope of the LXC container.

[0023] A network heap 210 can be run in user space 222 using Single Root Input / Output Virtualization (SR-IOV) PCIe virtualization functionality. SR-IOV allows for PCI Express isolation, enabling the sharing of a single physical PCI Express within a virtual environment and providing different virtual functions to virtual components on a single physical server. I / O heap 210 allows VFS nodes to bypass the standard TCP / IP heap 220 and communicate directly with network adapter 218. A portable operating system interface for Unix (POSIX) VFS functionality can be provided to VFS drive 208 via lock-free queues. A Non-Volatile Memory Standard (NVMe) drive 214 can also be run in user space 222 using SR-IOV or full PCIe physical function addresses, completely bypassing the Linux I / O heap. NVMe access can be used to access non-volatile storage media 216 attached via the PCI Express (PCIe) bus. For example, non-volatile storage media 216 can be flash memory in the form of a solid-state drive (SSD) or storage-class memory (SCM) in the form of SSDs or storage blocks (DIMMs). Other embodiments may include storage-class memory technologies such as 3D-Xpoint.

[0024] By coupling the physical SSD 216 with the SSD agent 214 and the network 210, the SSD can be implemented as a network device. Alternatively, the SSD can be implemented as a network-attached NVMe SSD 242 or 244 using a network protocol such as NVMe-oF (Fibre Channel-based NVMe). NVMe-oF allows access to the NVMe device using redundant network links, thus providing a higher level of security or resilience. Network adapters 226, 228, 230, and 232 may include hardware accelerators for connecting to the NVMe SSDs 242 and 244 to convert them into network NVMe-oF devices without the use of a server. Both the NVMe SSDs 242 and 244 may include two physical ports, and all data can be accessed through either of these ports.

[0025] Each client process / application 212 may run directly on the operating system or in a virtual machine and / or container served by the operating system and / or SuperMonitor. In performing its primary functions, the client process 212 may read data from and / or write data to storage. However, the primary function of the client process 212 is storage-independent (i.e., the process is only concerned with the reliable storage of its data and its retrieval as needed, and is not concerned with where, when, or how the data is stored). Exemplary applications that generate this process include: email servers, web servers, office productivity applications, customer relationship management (CRM), animated video rendering, genomic computing, chip design, software building, and enterprise resource planning (ERP).

[0026] Client application 212 can make system calls to kernel 224, which communicates with VFS driver 208. VFS driver 208 places the corresponding request in the queue of VFS frontend 202. If several VFS frontends exist, the driver can load balanced access across different frontends, ensuring that a single file / directory is always accessed via the same frontend. This can be done by "sharding" the frontends based on file or directory IDs. VFS frontend 202 provides an interface for routing file system requests to the appropriate VFS backend based on the bucket responsible for this operation. The appropriate VFS backend may reside on the same host or on another host.

[0027] The VFS backend 206 hosts several buckets, each of which serves the file system requests it receives and completes tasks to manage the virtual file system through other means (such as load balancing, logging, storing metadata, caching, moving data between layers, removing expired data, correcting corrupted data, etc.).

[0028] VFS SSD agent 214 handles interactions with the corresponding storage device 216. This may include, for example, translating addresses and generating commands to be sent to the storage device (e.g., via SATA, SAS, PCIe, or other suitable buses). Thus, VFS SSD agent 214 acts as an intermediary between storage device 216 and the VFS backend 206 of the virtual file system. SSD agent 214 is also capable of communicating with standard network storage devices that support standard protocols such as NVMe-oF (Fibre Channel-based NVMe).

[0029] Figure 3 Another representation of a virtual file system according to an exemplary embodiment of this disclosure is shown. Such as the above regarding... Figure 2 Described, in Figure 3In this context, element 302 represents memory resources (e.g., DRAM and / or other short-term memory) and processing resources (e.g., x86 processors, ARM processors, NICs, ASICs, FPGAs, etc.) of the various nodes (compute, storage, and / or VFS) on which the virtual file system is hosted. Element 308 represents one or more physical storage devices 216 that provide long-term storage for the virtual file system.

[0030] like Figure 3 As shown, physical storage is organized into multiple Distributed Failure Resilience Address Spaces (DFRAS) 518. Each DFRAS includes multiple blocks 310, which in turn include multiple blocks 312. In some embodiments, blocks 312 are organized into blocks 310 only for convenience, and this is not feasible in all embodiments. Each block 312 stores commit data 316 (which may take various states as described below) and / or metadata 314 describing or referencing the commit data 316.

[0031] Organizing storage 308 into multiple DFRAS allows for access from multiple—or perhaps all—nodes of the virtual file system (e.g., Figure 1 All nodes 1041–104 in N 1061–106 M and 1201–120 J High-performance parallel commits can be performed by executing concurrent commits. In an exemplary implementation, each node in the virtual file system can own one or more corresponding DFRASs from multiple DFRASs and have exclusive read / commit access to its own DFRAS.

[0032] Each bucket has its own DFRAS, so writing to it does not require collaboration with any other nodes. Each bucket can build stripes across multiple different blocks on multiple different SSDs, thus each bucket with its own DFRAS can choose which "block stripe" to write to simultaneously based on multiple parameters, and once a block is assigned to the bucket, no collaboration is required to perform this operation. All buckets can be efficiently written to all SSDs without any collaboration.

[0033] Each DFRAS owned and accessed solely by its own bucket running on a designated node allows individual nodes in the VFS to control a portion of storage 308 without needing to cooperate with any other nodes (except for the [re]distribution] of buckets that hold DFRAS during initialization or after a node failure, e.g., the [re]distribution] of actual reads / commits to storage 308 can be performed asynchronously). Thus, in this implementation, each node can read / commit DFRAS from its bucket independently of the operations being performed by other nodes, without needing any consensus when reading and committing to storage 308. Furthermore, the fact that a particular node owns multiple buckets in the event of a node failure allows for a more intelligent and efficient reallocation of its workload to other nodes (without distributing the entire workload to a single node, potentially creating "hot spots"). Therefore, in some implementations, the number of buckets may be large relative to the number of nodes in the system, such that any single bucket can be a relatively small load placed on another node. This allows for fine-grained redistribution of the load on a failed node based on the capabilities and capacity of other nodes (e.g., a higher percentage of the failed node bucket can be assigned to nodes with greater capacity and capability).

[0034] To enable this operation, metadata mapping each bucket to its currently owned node can be saved, allowing reads and commits to storage 308 to be redirected to the appropriate node.

[0035] Because the entire file system metadata space (e.g., directories, file attributes, content ranges within files, etc.) can be divided (e.g., split, cut, or sharded) into small, uniform fragments (e.g., "fragments"), load distribution is possible. For example, a large system with 30k servers can have its metadata space divided into 128k or 256k fragments.

[0036] Each of these metadata fragments can be stored in a "bucket". Each VFS node can be responsible for several buckets. When a bucket serves a metadata fragment on a given backend, the bucket is considered "active" or the "leader" of that bucket. Typically, there are many more buckets than VFS nodes. For example, a small system with 6 nodes can have 120 buckets, and a large system with 1000 nodes can have 8k buckets.

[0037] In a small set of nodes, typically five nodes in five groups forming the bucket, each bucket can be activated. Regarding the distribution of the five groups (pneta-groups) for each bucket, the cluster configuration keeps all participating nodes updated.

[0038] Each group of five monitors itself. For example, if the cluster has 10k servers and each server has 6 buckets, then each server only needs to communicate with 30 different servers to maintain the state of its bucket (6 buckets will have 6 groups of five, so 6 * 5 = 30). This is a much smaller number compared to a central entity having to monitor all nodes and maintain the state of the entire cluster. Using groups of five allows for scaling with larger clusters because the nodes don't perform much more work as the cluster size increases. This can have the drawback that, in "dumb" mode, smaller clusters can actually generate more communication than physical nodes exist; however, this drawback is overcome by sending only a single heartbeat between two servers to all their shared buckets (as the cluster grows, this becomes only one bucket; however, if you have a small 5-server cluster, it simply includes all buckets in all messages and each server will only communicate with the other 4). Groups of five can be judged (i.e., reach consensus) using an algorithm similar to the Raft consensus algorithm.

[0039] Each bucket can have a set of compute nodes running on it. For example, five VFS nodes can run one bucket. However, at any given time, only one node in that set is the controller / leader. Furthermore, for sufficiently large clusters, no two buckets share the same grouping. If there are only 5 or 6 nodes in the cluster, most buckets can share a backend, and in quite large clusters, there are multiple distinct groupings of nodes. For example, for 26 nodes, there are more than... There are five possible five-node groups (i.e., five groups).

[0040] All nodes in a group learn which node is the actual active controller (i.e., the leader) of the bucket and reach a consensus on this. A node accessing a bucket can remember ("caches") the last node among the group members (e.g., five) that was the leader of that bucket. If it accesses the bucket leader, the bucket leader performs the required action. If it accesses a node that is not the current leader, the node instructs the leader to "redirect" the access. If the accessed, cached leader node expires, the node can try different nodes within the same five groups. All nodes in the cluster share a common cluster "configuration," allowing nodes to learn which servers can run the various buckets.

[0041] Each bucket can have a load / usage value indicating how much of the bucket weight is used by applications running on the file system. For example, even if there is an imbalance among the multiple buckets used, a server node with 11 lightweight buckets can receive additional metadata buckets to run before a server with 9 heavy-use buckets. The load value can be determined based on average response latency, the number of currently running operations, memory consumption, or other metrics.

[0042] Even when VFS nodes are not faulty, redistribution is possible. If the system identifies a node as busier than others based on tracked load metrics, it can move one of its buckets to a less busy server (i.e., "failed"). However, load balancing can be implemented by shifting writes and reads before the bucket is actually relocated to a different host. Because each write can end in a different group of nodes determined by DFRAS, it is not possible to select a node with a higher load as being in the stripe of data being written. The system can also choose not to serve reads from high-load nodes. For example, a "degraded-mode read" can be performed, where blocks from a high-load node are reconstructed from blocks in the same stripe. A degraded-mode read is a read performed via the remaining nodes in the same stripe and the data is reconstructed via failover protection. Since the initiator of the read can assume that the node is below, a degraded-mode read can be performed when read latency is too high. If the load is high enough to create higher read latency, the cluster can revert to reading data from other nodes and reconstructing the required data using a degraded-mode read.

[0043] Each bucket manages its own Distributed Erasure Coding instance (i.e., DFRAS 518) and does not need to cooperate with other buckets to perform read and write operations. There are potentially thousands of concurrent Distributed Erasure Coding instances working simultaneously, each for a different bucket. Because it effectively allows arbitrarily large file systems to be divided into independent fragments that do not require cooperation, this is a complete part of the scaling performance, thus providing high performance independent of scaling.

[0044] Each bucket handles all file system operations that fall into its fragments. For example, directory structure, file attributes, and file data ranges fall under the jurisdiction of a specific bucket.

[0045] The operation begins by finding which bucket has the operation and can be completed from any frontend. Then, the backend leader and nodes of that bucket are determined. This determination is performed by trying the last known leader. If the last known leader is not the current leader, the node learns which node is the current leader. If the last known leader is no longer part of the bucket's five groups, the backend will inform the frontend that it should restore its configuration to find members in the bucket's five groups. This distribution of operations allows complex operations to be processed across multiple servers, rather than a single computer in a standard system.

[0046] File system metadata is built from directories, which store information about the files stored within them. Each file is represented by an inode block that stores the file's basic data, and also by a range block that stores specific information about what range is stored in the file system. Registers are data structures that store all the inode and range information for the storage system. Registers also allow conversion between different types of information. Using key / value storage, object IDs (i.e., keys) lock data within 4KB blocks. Object IDs can be 32 bits wide, and values ​​can be 4KB blocks plus a small tag.

[0047] For the "Set" command, the register receives the object type ID (e.g., directory ID, inode ID, range ID, etc.) and the new 4k block value. Part of the stored 4k data is the ID of the object pointing to that 4k block (i.e., the "reverse pointer").

[0048] For the "get" command, the register receives the object type ID and a reverse pointer and returns NULL (i.e., the object does not exist) or 4k representing the object.

[0049] Because different object types can obtain the same ID, back pointers are important; however, the same object ID cannot point to different object types. Therefore, knowing the object ID allows concurrent reads on all blocks with the same object ID, and only one block with the correct back pointer is returned. Because these concurrent reads can be performed in parallel, the overall latency remains very low. The number of concurrent buckets supported by each node is based on how much RAM is allocated to the registers.

[0050] Registers are used to point to block data from pointers / keys. Each bucket stores and manages its own registers. The dirty part of a register (i.e., information that has not yet been moved to flash memory) is RAM. Register data is also protected by Distributed Erasure Coding (DEC) in DFRAS. A complete reconstruction of the register (scanning the dirty part) is possible by another node becoming the leader of its bucket.

[0051] Figure 4An embodiment of a write-balanced flash memory register 401 according to an exemplary implementation of this disclosure is shown. In this embodiment, the register includes two items – an L1 (level 1) item and an L2 (level 2) item. Each register can store 512 L1 blocks (2MB in total) in RAM. The correct L1 key (index in the 512 array) is derived by the first 9 bits of the object's register hash ID. Each L1 block can store data for four L2 groups, therefore, a total of 2048 (512x4) L2 groups can exist. Each L2 group is specified by the first 11 bits of the register hash ID. The first 9 bits point to the rightmost L1 block, and then the remaining 2 bits pick up the L2 group information within that L1 block.

[0052] The information for each L2 group is stored within approximately 11kB of the L1 block. Each block also stores a 4-byte ID pointing to its block ID, allowing it to be verified in a lookup table. Therefore, this data structure can store 4092 bytes of data.

[0053] Multiple categorized tuples (key, value) within a 4k block will eventually result in the production of the actual data block, selected via the correct reverse pointer. This implements a very efficient "branching factor" with fewer required updates. Multiple L2 updates will update the same L1 block, and reaching the correct L2 block requires only two reads (reading the L1 block in memory that points to the correct L2 block).

[0054] Reading an object requires at least two hops. The number of hops is based on the branching factor of approximately 400 pointer tables per 4k block. Pointers are 4 bytes long and the block ID stored in the Distributed Erasure Code (DEC) is also 4 bytes long; therefore, each item is 8 bytes.

[0055] The correct L2 block for the actual data can be loaded using 1kB of the L1 block. This 1kB lists the ID range for each L2 block. When the system starts, a single L2 block exists for that range, storing all the ID ranges it spans (i.e., 32 – 11 = 21). When this L2 block becomes full (reaching 4k), it is split, and then the L1 blocks point to the two L2 blocks. When each L2 block becomes full, it splits into two new L2 blocks. The L1 block pointing to the L2 blocks updates the ranges stored in the individual L2 blocks and updates the pointers to the new L2 blocks.

[0056] If there is a lot of random work on the system, there will be many value changes that cause a lot of L2 changes, some L1 changes, and write amplification (i.e., more write operations per block). However, write amplification can be "leveled out" (i.e., reduced) by using shadow registers.

[0057] Figure 5 Exemplary embodiments of the present disclosure are shown. Figure 4 An example of shadow registers associated with flash registers in the NVM. Each L1 and L2 block can be associated with a 4k “shadow” block containing changes to the actual data committed to the NVM.

[0058] for Figure 4 The flash register 401 contains 512 L1 blocks and 2048 L2 blocks, with an associated shadow register comprising 10 MB (2560 x 4k blocks). Shadow bit 403 indicates whether a 4k value block is shadowed. This bit is set when a key (shown as 4 bits for illustrative purposes) has an associated shadow value. Therefore, a search for changes to the 4k blocks is performed only as needed. If a specific bit in shadow bit 403 indicates that a shadow block with a 4k value does not exist, the 4k value is returned without searching the shadow register 501.

[0059] By using shaded blocks, the actual 4k blocks storing values ​​do not need to change as individual values ​​change. When a portion of a 4k value block has to change, the difference (potentially small) is stored in the L1 shaded block that follows that change. Once the L1 shaded block is full (e.g., cumulative 4k changes below that level), the L1 shaded block is destroyed, pushing the change down to the L2 shaded block, and then the L1 shaded block is cleared. When the L2 shaded block is full (e.g., cumulative 4k changes to the value it points to), the change is pushed into the actual value blocks (and the bits indicating that these blocks have shaded data are cleared). For any value setting, because no actual 4k data is written to disk, writes to flash memory are "balanced," but only once over long periods. The system can accumulate a large number of register value settings into a single operation, so the system does not change a large number of 4k blocks, but only writes a few updates to the associated shaded L1 and L2 blocks, thereby minimizing the actual number of writes required.

[0060] The shadowing operation is also performed in reverse, by finding the actual 4k value and checking for any changes it applied using the relevant L1 and L2 shadowing checks. Because such overwrite checks can be quite expensive, a large bitmap of all values ​​can be stored in memory (e.g., RAM), and it's possible to mark whether a key stores an overwrite change. Therefore, a search for changes can be performed only when needed. Only the bitmap is stored in memory (e.g., RAM), but it can be easily reconstructed from an on-disk data structure in the event of a failure. Because this data structure stores the state of multiple operations, it allows for multiple writes to accumulate into a smaller number of on-disk writes.

[0061] Figure 6This is a flowchart illustrating an exemplary method for write balancing using flash registers in a load-balanced distributed storage system. Each physical SSD comprises data blocks (e.g., 128MB blocks), and each block comprises regions (e.g., 4kB regions). This physical memory is mapped to multiple logical ranges (e.g., pointers are mapped to 256 regions). This mapping is maintained in a VFS, which acts as registers. The VFS backend includes buckets that manage their associated registers.

[0062] In box 601, the VFS frontend receives a write request. In box 603, it determines which VFS backend owns the bucket associated with the write request. This VFS backend may reside on the same server as the VFS frontend receiving the write request. Alternatively, the VFS backend may reside on a different server. In box 605, the secondary registers in the bucket are accessed based on a function (e.g., a hash) of the inode ID and offset. In box 607, changes made to the flash registers (e.g., in an SSD) via writes are shadowed in the primary registers. In box 609, if the number of changes exceeds the capacity of the primary register shadow, these changes are pushed to the secondary registers. In box 611, if the number of changes pushed to the secondary registers exceeds the capacity of the secondary register shadow, the primary and secondary registers are updated and written to the flash registers (e.g., in an SSD) along with the actual 4k value associated with the key stored in the full L2 shadow page. If the number of changes pushed to the secondary register does not exceed the capacity of the secondary register shadow, the bucket can only continue to process write requests and metadata changes, thus making a small number of updates to the shadow page and not requiring a full 4k page to be written for each operation.

[0063] Figure 7 An exemplary implementation is shown in which two of the distributed fault-tolerant address spaces are hosted on multiple solid-state storage disks. Block 510 can be... 1,1 -510 D,C Organized into multiple block strips 5201–520 S (S is an integer). In an exemplary implementation, forward error correction (e.g., erasure coding) is used individually for each block stripe 520. s (s is an integer, where 1 ≤ s ≤ S) is protected. Therefore, any specific block stripe 520 can be determined based on the required level of data protection. s Block 510 in d,c The quantity.

[0064] For illustrative purposes, assume each block strip is 520. s Includes N = M + K (where each of N, M, and K is an integer) blocks 510 d,c Then there are N blocks 510d,c The M blocks can store data numbers (usually binary numbers or "bits" of the current storage device) and N blocks 510 d,c The K blocks in the array can store protection numbers (again, typically bits). Therefore, for each stripe 520 s A virtual file system can distribute N blocks from N different fault domains. d,c .

[0065] As used herein, a "fault domain" refers to a group of components in which the failure of any single component (power loss, unresponsiveness, etc.) would cause all components to fail. For example, if a rack has a single top-of-rack switch, a failure of that switch would disrupt connectivity to all components on that rack (e.g., compute, storage, and / or VFS nodes). Thus, the rest of the system is equivalent to the failure of all components on that rack together. The virtual file system according to this disclosure may include fewer fault domains than block 510.

[0066] In an exemplary embodiment where the virtual file system nodes are connected and powered in a fully redundant manner by only a single storage device 506 for each node, the fault domain can be only that single storage device 506. Thus, in the exemplary embodiment, each block stripe 520 s Includes storage in N storage devices 5061–506 D (Therefore, D is greater than or equal to N) multiple blocks 510 on each storage device d,c . Figure 7 An embodiment of this implementation is shown in the figure.

[0067] exist Figure 7 In this configuration, D=7, N=5, M=4, K=1, and the memory is organized into two DFRAS. These quantities are for illustration only and are not intended to be limiting. For illustration, three block stripes 520 in the first DFRAS are arbitrarily selected. The first block stripe 5201 consists of block 510. 1,1 510 2,2 510 3,3 510 4,5 and 510 5,6 Composition; the second block strip 5202 is composed of block 510 3,2 510 4,3 510 5,3 510 6,2 and 510 7,3 Composed of; and the third block strip 5203 is composed of block 510 1,4 510 2,4 510 3,5 5105,7 and 510 7,5 constitute.

[0068] Despite Figure 7 In a simple example, D = 7 and N = 5; however, in practical implementations, D can be much larger than N (e.g., a multiple of an integer greater than 1 and possibly many orders of magnitude higher), and both values ​​can be chosen such that the probability of any two block stripes 520 of a single DFRAS residing on the same set of N storage devices 506 (or more generally, the same set of N fault domains) is below a desired threshold. Similarly, any single storage device 506 d (Or more generally,) a failure in any single fault domain will result in (possibly based on the required statistics: the chosen values ​​of D and N, the size of the N storage devices 506, and the arrangement of the fault domains) the loss of at most any particular stripe 520. s One of the blocks 510 bc Furthermore, a double failure will cause most stripes to lose at most a single block (510). b,c Furthermore, only a small number of strips (judged based on the values ​​of D and N) will lose two blocks in any given strip 520 (for example, the number of two faulty strips can be exponentially less than the number of one faulty strip).

[0069] For example, if each storage device has 506 d If it's 1TB and each block is 128MB, then the storage device is 506. d The failure will result in (based on the required statistics, possibly determined by the following factors: the selected values ​​of D and N, the size of N storage devices 506, and the arrangement of the failure domain) 7812 (=1TB / 128MB) block stripes 520 losing one block 510. For each affected block stripe 520 s It can use appropriate forward error correction algorithms and chunked stripes 520 s The other N-1 blocks in the missing block 510 d,c A rapid reconfiguration was performed. Furthermore, because the affected 7812 block stripes were evenly distributed across all storage devices 5061–506... D Therefore, for the 7812 missing modules 510 d,c The refactoring will involve (possibly based on the following statistics: the selection values ​​of D and N, the size of the N storage devices 506, and the arrangement of the fault domains) refactoring from each storage device 5061–506. D Reading small amounts of data (i.e., the burden of reconstructing lost data is evenly distributed across all storage devices 5061–506) D Above this, to provide very fast fault recovery.

[0070] Next, proceed to the two storage devices 5061–506. D In the event of concurrent failures (or more generally, concurrent failures in two fault domains), due to the block stripes 5201–520 of each DFRAS... S Evenly distributed across all storage devices 5061–506 D Above that, so only a very small number of block stripes 5201–520 S Two of its N chunks are lost. The virtual file system can operate based on the instruction chunk stripe 5201–520. S With storage device 5061–506 D The metadata mapping between the two loss chunk stripes allows for rapid identification of the double-loss chunk stripe. Once identified, the virtual file system can prioritize the reconstruction of these double-loss chunk stripes before initiating the reconstruction of the single-loss chunk stripe. The remaining chunk stripes will have only a single loss chunk, and for this (the majority of affected chunk stripes), two storage devices 506 d A concurrent failure occurred on only one storage device 506. d The same fault occurs. The same principle applies to third concurrent faults (in two concurrent fault scenarios, the number of block stripes with three fault blocks is even less than the number with two fault blocks), etc. In an exemplary implementation, based on block stripe 520... s The number of lost items can be controlled by executing block stripes 520. s The rate of reconstruction. For example, this can be implemented by controlling the rate of reads and commits for performing reconstruction, the rate of FEC calculations for performing reconstruction, and the rate of communication of network messages for reconstruction.

[0071] Figure 8 A forward error correction scheme for protecting data stored in non-volatile memory of a virtual file system, according to an exemplary embodiment of this disclosure, is illustrated. Storage block 902 in block stripes 5301–5304 of DFRAS is shown. 1,1 -902 7,7 .exist Figure 8 In the protection scheme, five blocks in each strip are used to store digital copyright data, and two blocks in each strip are used to protect the data storage of the digital rights (i.e., M=5 and K=2). Figure 8 In the process, the protected number is calculated using the following equations (1)–(9):

[0072]

[0073]

[0074]

[0075]

[0076]

[0077]

[0078]

[0079]

[0080]

[0081] thus, Figure 8 The four stripes 5301–5304 constitute part of a multi-strip (in this case, four stripes) FEC protection domain and can recover any two or fewer blocks lost in any block stripes 5301–5304 using various combinations of equations (1)–(9) above. For comparison, the single-strip protection domain embodiment would be as if D11, D22, D33, D44, and D54 were only protected by P1 and D11, D22, D33, D44, D54, and P1 were all written to stripe 5301 (5301 would be a single-strip FEC protection domain).

[0082] According to an exemplary embodiment of this disclosure, multiple computing devices are communicatively coupled to each other via a network, and each of the multiple computing devices includes one or more storage devices among a plurality of storage devices. Multiple failover address spaces are distributed across the multiple storage devices such that each failover address space spans multiple storage devices. Each failover address space is organized into multiple stripes (e.g., as shown in the example embodiments). Figure 7 and 8 Multiple 530s in the multiple stripes). Each or more stripes in the multiple stripes constitute multiple forward error correction (FEC) protection domains (e.g., such as Figure 8 A portion of a corresponding protection domain within a multi-strip FEC domain. Each stripe in the multiple stripes may include multiple storage blocks (e.g., multiple 512s). Each block of a particular stripe in the multiple stripes may reside on different storage devices in the multiple storage devices. The first portion of the multiple storage blocks (e.g., by...) Figure 8 902 of strip 5301 in 1,2 -902 1,6 The five quantities constituted can be used to store data numbers, and the second part of multiple storage blocks (e.g., Figure 8 The two 902 stripes of 5301 in the middle 1,1 and 9021,7 The number of fault-resilience address spaces (FSAL) can be used to store protection numbers calculated at least in part based on data numbers. Multiple computing devices can operate to rank multiple stripes. Ranking can be used to select which stripe among the multiple stripes to use in the next commit operation to one of the multiple fault-resilience address spaces. Ranking can be based on how many protected and / or unprotected storage blocks are in each of the multiple stripes. For any particular stripe among the multiple stripes, ranking can be based on a bitmap stored on multiple storage devices having that particular stripe. Ranking can be based on how many blocks of currently stored data are in each of the multiple stripes. Ranking can be based on the read and write overhead of committing to each of the multiple stripes. At any given time, each fault-resilience address space can be owned by only one of the multiple computing devices, and each of the multiple fault-resilience address spaces can be read and written only by its owner. Each computing device can own multiple fault-resilience address spaces. Multiple storage devices can be organized into multiple fault domains. Each stripe among the multiple stripes can span multiple fault domains. Each fault resilience address space can span all fault domains in multiple fault domains, such that if any one of the fault domains fails, the workload for reconstructing lost data is distributed across the other fault domains in the multiple fault domains. Multiple stripes can be distributed across multiple fault domains, such that in the event of concurrent failures in two fault domains, the change in two blocks of any one specific stripe registered in the two fault domains is exponentially less than the change in only one block of any one specific stripe registered in the two fault domains. Multiple computing devices can operate to first reconstruct any stripe among the multiple stripes with two fault blocks, and then reconstruct any stripe among the multiple stripes with only one fault block. Multiple computing devices can operate to perform the reconstruction of the multiple stripes with two fault blocks at a higher rate than the reconstruction rate of the multiple stripes with only one fault block (e.g., a larger percentage of the CPU clock duty cycle dedicated to reconstruction, a larger percentage of the network transfer opportunity dedicated to reconstruction, etc.). In the event of a failure in one or more fault domains, multiple computing devices can operate to determine the rate of reconstruction for any particular lost block based on the number of other blocks lost in the same stripe across multiple stripes. Here, one or more fault domains comprise multiple storage devices. Each of the multiple FEC protection domains can span multiple stripes across multiple stripes. Multiple stripes can be organized into multiple groups (e.g., Figure 7 Block strips 5201–520 in SIn this context, each of the multiple groups comprises one or more stripes from a plurality of stripes, and for each group, the multiple computing devices are operable to rank one or more stripes from the plurality of stripes in the group. The multiple computing devices can operate to: perform successive commit operations on selected groups from the multiple groups until one or more stripes from the plurality of stripes in that group no longer meet the judgment criteria, and once a selected group from the multiple groups no longer meets the judgment criteria, select a different group from the multiple groups. The criteria can be based on how many blocks are available for the new data to be written.

[0083] Although the method and / or system have been described with reference to specific embodiments, those skilled in the art will understand that various changes and equivalents can be made without departing from the scope of the method and / or system. Furthermore, various modifications can be made to adapt to the specific circumstances or teaching materials of this disclosure without departing from its scope. Therefore, the method and / or system is not intended to be limited to the specific embodiments disclosed, but rather to include all embodiments falling within the scope of the appended claims.

[0084] As used herein, the terms “circuit” and “line” refer to physical electronic components (i.e., hardware) and any software and / or firmware (“code”) that is executed by the hardware and can configure the hardware. As used herein, for example, a specific processor and memory may include a first “line” when the first line or more of the code is executed, and a specific processor and memory may include a second “line” when the second line or more of the code is executed. As used herein, “and / or” refers to any one or more items in a list connected by “and / or”. For example, “x and / or y” refers to any element in the three-element set {(x),(y),(x,y)}. In other words, “x and / or y” refers to “one or both of x and y”. As another embodiment, “x, y, and / or z” refers to any element in the seven-element set {(x),(y),(z),(x,y),(x,z),(y,z),(x,y,z)}. In other words, “x, y, and / or z” refers to “one or more of x, y, and z”. As used herein, the term "exemplary" refers to a non-limiting embodiment, instance, or example. As used herein, the terms "e.g." and "for example" illustrate a list of one or more non-limiting embodiments, instances, or examples. As used herein, a line is "capable of operating" to perform a function, regardless of whether the function's performance is disabled or not enabled (e.g., via user configuration settings, factory maintenance, etc.), provided that the line includes the hardware and code necessary to perform the function (if necessary).

Claims

1. A system for controlling memory access, comprising: A computing device capable of distributing data across multiple storage devices; as well as Registers that can be accessed by the computing device based on an inode ID and an offset, wherein: The computing device is operable to access a first level of the registers to shadow changes made to the plurality of storage devices. If the number of changes made to the plurality of storage devices exceeds the capacity of the first level of the register, the changes are pushed to the second level of the register; and If the number of changes pushed to the second level exceeds the capacity of the second level of the register, the plurality of storage devices are updated according to the changes recorded in the first level and the second level of the register.

2. The system according to claim 1, wherein, The computing device is one of several computing devices in the Virtual File System (VFS).

3. The system according to claim 2, wherein, The metadata buckets are stored on the VFS backend on the computing device.

4. The system according to claim 3, wherein, The bucket containing the metadata is one of multiple buckets on the VFS backend.

5. The system according to claim 4, wherein, Each of the multiple buckets manages a unique register.

6. The system according to claim 4, wherein, The VFS backend is operable to modify data read from one of the plurality of storage devices based on the changes recorded in the register.

7. The system according to claim 1, wherein, The register is protected by erasing the encoding.

8. The system according to claim 1, wherein, The plurality of storage devices include non-volatile flash memory.

9. The system according to claim 1, wherein, The register is protected by verification based on one or more reverse pointers, wherein each of the one or more reverse pointers includes a block ID of an object pointing to a register block.

10. The system according to claim 1, wherein, The computing device is operable to locate memory blocks based on a pointer table stored in a register block.

Images