Flash memory registry with on-disk hash

Abstract

The invention discloses a flash memory registry with on-disk hash. A system includes a registry of a plurality of memory blocks in which: the registry is adaptively resized, the registry including one or more registry blocks, each registry block including a split level and one or more keys, each of the one or more keys including a same number of bits, an index in each of the one or more keys is indicated by a specified number of bits, and the specified number is a split level.

Description

[0001] This application is a divisional application of the patent application with application number 201880086385.4 and invention name “Flash Registry with On-Disk Hash”, which entered the national stage on July 13, 2020 and has an international application date of October 5, 2018 and international application number PCT / IB2018 / 001226. The entire contents of the application are incorporated herein by reference.

[0002] Priority claim

[0003] This application claims priority to U.S. Provisional Patent Application No. 62 / 585,054, filed on November 13, 2017, entitled “FLASH REGISTRY WITH ON-DISK HASH” and U.S. Patent Application No. 16 / 121,500, filed on September 4, 2018, entitled “FLASH REGISTRY WITH ON-DISK HASH”. Background Art

[0004] By comparing such an approach with some aspects of the present method and system set forth in the remainder of this disclosure with reference to the accompanying figures, the limitations and shortcomings of conventional approaches to data storage will become apparent to those skilled in the art.

[0005] CROSS-REFERENCE TO RELATED APPLICATIONS

[0006] The entirety of U.S. Patent Application No. 15 / 243,519, entitled "Distributed Erasure Coded Virtual File System," is incorporated herein by reference in its entirety. Summary of the invention

[0007] Methods and systems for a flash registry with on-disk hashing are provided, substantially as illustrated by and / or described in connection with at least one of the accompanying drawings and more fully set forth in the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 Various example configurations of a virtual file system according to aspects of the present disclosure are shown.

[0009] Figure 2 An example configuration of a virtual file system node according to aspects of the present disclosure is shown.

[0010] Figure 3 Another representation of a virtual file system according to an example implementation of the present disclosure is shown.

[0011] Figure 4 An example of using on-disk hashing to split blocks of a flash registry is shown according to an example implementation of the present disclosure.

[0012] Figure 5An example of using on-disk hashing to split buckets of a flash registry is shown according to an example implementation of the present disclosure.

[0013] Figure 6 An example of using on-disk hashing to merge blocks of a flash registry is shown according to an example implementation of the present disclosure.

[0014] Figure 7 is a flow chart illustrating a method for adding blocks of a flash registry using on-disk hashing according to an example implementation of the present disclosure.

[0015] Figure 8 is a flow chart illustrating a method for accessing blocks of a flash registry utilizing an on-disk hash according to an example implementation of the present disclosure.

[0016] Fig. 9 An example implementation is shown in which two distributed failure-resilient address spaces reside on multiple solid-state storage disks.

[0017] Fig.10 A forward error correction scheme according to an example implementation of the present disclosure is shown, which may be used to protect data stored to a non-volatile memory of a virtual file system. DETAILED DESCRIPTION

[0018] The system in the present disclosure is suitable for small clusters and can also be expanded to many tens of thousands of nodes. Example embodiments of non-volatile memory (NVM) (e.g., flash memory in the form of a solid-state drive (SSD)) are discussed. NVM can be divided into 4kB "blocks" and 128MB "chunks". "Extents" can also be stored in volatile memory, such as RAM for fast access, or can be backed up by NVM storage. The extension area can store pointers to blocks, for example, 256 pointers to 1MB of data stored in the block. In other embodiments, larger or smaller memory partitions can also be used. The metadata functions in the present disclosure can be effectively distributed on many servers. For example, in the case of a "hot spot" that points to a specific part of the file system namespace, this load can be distributed on multiple nodes.

[0019] Figure 1 Various example configurations of a virtual file system (VFS) according to aspects of the present disclosure are shown. Figure 1Shown in the figure is a local area network (LAN) 102, which includes one or more VFS nodes 120 (indexed by integers from 1 to J, j ≥ 1), and optionally includes (shown in dashed lines): one or more dedicated storage nodes 106 (indexed by integers from 1 to M, M ≥ 1); one or more compute nodes 104 (indexed by integers from 1 to N, N ≥ 1); and / or an edge router connecting the LAN 102 to a remote network 118. The remote network 118 optionally includes one or more storage services 114 (indexed by integers from 1 to K, K ≥ 1); and / or one or more dedicated storage nodes 115 (indexed by integers from 1 to L, for L ≥ 1).

[0020] 120 per VFS node j (j is an integer, where 1≤j≤J) is a network computing device (e.g., a server, a personal computer, etc.) that includes circuitry for running a VFS process and an optional client process (or directly on device 104). n operating system and / or on the device 104 n in one or more virtual machines running in a .

[0021] Compute node 104 is a networking device that can run a VFS front end without a VFS back end. Compute node 104 can run the VFS front end by putting SR-IOV into the NIC and using a full processor core. Alternatively, compute node 104 can run the VFS front end by routing the networking via the Linux kernel networking stack and using kernel process scheduling, thus not having the requirement of a full kernel. This is useful if the user does not want to allocate a full core for VFS, or the networking hardware is not compatible with the VFS requirements.

[0022] Figure 2An example configuration of a VFS node according to aspects of the present disclosure is shown. The VFS node includes a VFS front end 202 and a driver 208, a VFS memory controller 204, a VFS back end 206, and a VFS SSD agent 214. As used in the present disclosure, a "VFS process" is a process that implements one or more of the following: a VFS front end 202, a VFS memory controller 204, a VFS back end 206, and a VFS SSD agent 214. Therefore, in an example implementation, resources (e.g., processing and memory resources) of a VFS node can be shared between client processes and VFS processes. The processes of the VFS can be configured to require a relatively small amount of resources to minimize the impact on the performance of client applications. The VFS front end 202, the VFS memory controller 204, and / or the VFS back end 206, and / or the VFS SSD agent 214 can run on a processor of the host 201 or on a processor of a network adapter 218. For a multi-core processor, different VFS processes can run on different cores and can run different subsets of services. From the perspective of the client process 212, the interface with the virtual file system is independent of the specific physical machine running the VFS process. The client process only needs the driver 208 and the front end 202 to exist and can provide services for them.

[0023] A VFS node can be implemented as a single tenant server (e.g., bare metal) running directly on an operating system, or as a virtual machine (VM) and / or container (e.g., Linux container (LXC)) in a bare metal server. VFS can run as a VM environment in an LXC container. Therefore, inside the VM, only an LXC container containing the VFS can run. In a classic bare metal environment, there are user space applications and the VFS runs in an LXC container. If the server is running other containerized applications, the VFS may run in an LXC container outside the management scope of the container deployment environment (e.g., Docker).

[0024] The VFS node can be served by an operating system and / or a virtual machine monitor (VMM) (e.g., a hypervisor). The VMM can be used to create and run a VFS node on a host 201. Multiple kernels can reside in a single LXC container running a VFS, and the VFS can run on a single host 201 using a single Linux kernel. Therefore, a single host 201 may include multiple VFS front ends 202, multiple VFS memory controllers 204, multiple VFS back ends 206, and / or one or more VFS drivers 208. The VFS driver 208 can run in kernel space outside the scope of the LXC container.

[0025] A single root input / output virtualization (SR-IOV) PCIe virtual function can be used to run a network stack 210 in user space 222. SR-IOV allows isolation of PCI Express so that a single physical PCI Express can be shared on a virtual environment, and different virtual functions can be provided to different virtual components on a single physical server computer. The I / O stack 210 enables the VFS node to bypass the standard TCP / IP stack 220 and communicate directly with the network adapter 218. A portable operating system interface for uniX (POSIX) VFS functions can be provided to the VFS driver 208 through a lock-free queue. SR-IOV or a full PCIe physical function address can also be used to run a non-volatile memory Express (NVMe) driver 214 in user space 222, completely bypassing the Linux IO stack. NVMe can be used to access non-volatile storage media 216 connected via a PCI Express (PCIe) bus. The non-volatile storage medium 216 may be, for example, flash memory in the form of a solid state drive (SSD) or a storage class memory (SCM) in the form of a solid state drive (SSD) or a memory module (DIMM). Other examples may include storage class memory technologies such as 3D-XPoint.

[0026] By coupling the physical SSD 216 with the SSD agent 214 and the networking 210, the SSD can be implemented as a networked device. Alternatively, the SSD can be implemented as a network-attached NVMe SSD 242 or 244 by using a network protocol such as NVMe-oF (NVMe over fabric). NVMe-oF can allow access to NVMe devices using redundant network links, thereby providing a higher level or resilience. Network adapters 226, 228, 230, and 232 can include hardware acceleration for connecting to NVMe SSDs 242 and 244 to convert them into networked NVMe-oF devices without the use of a server. NVMe SSDs 242 and 244 can each include two physical ports, and all data can be accessed through any one of these ports.

[0027] Each client process / application 212 can run directly on the operating system, or can run in a virtual machine and / or container served by the operating system and / or hypervisor. The client process 212 can read data from the memory and / or write data to the memory in the process of performing its main function. However, the main function of the client process 212 is independent of storage (that is, the process only focuses on its data being reliably stored and retrievable when needed, without paying attention to the storage location, time or method of the data). Example applications that cause this process include: email servers, web servers, office efficiency applications, customer relationship management (CRM), animation video rendering, genome computing, chip design, software construction, and enterprise resource planning (ERP).

[0028] The client application 212 can make a system call to the kernel 224 that communicates with the VFS driver 208. The VFS driver 208 places the corresponding request on the queue of the VFS front end 202. If there are several VFS front ends, the driver may load-balance access to different front ends to ensure that a single file / directory is always accessed through the same front end. This can be done by "sharding" the front end based on the ID of the file or directory. The VFS front end 202 provides an interface for routing file system requests to the appropriate VFS back end based on the storage bucket responsible for the operation. The appropriate VFS back end can be on the same host or on another host.

[0029] The VFS backend 206 hosts multiple buckets, each of which services file system requests it receives and performs tasks to otherwise manage the virtual file system (e.g., load balancing, logging, maintaining metadata, caching, moving data between tiers, deleting obsolete data, correcting corrupted data, etc.).

[0030] The VFS SSD agent 214 handles the interaction with the corresponding storage device 216. This may include, for example, translating addresses, and generating commands issued to the storage device (e.g., on a SATA, SAS, PCIe, or other suitable bus). Thus, the VFS SSD agent 214 acts as an intermediary between the VFS backend 206 of the virtual file system and the storage device 216. The SSD agent 214 may also communicate with standard network storage devices that support standard protocols, such as NVMe-oF (NVMe over Fabric).

[0031] Figure 3 Another representation of a virtual file system according to an example implementation of the present disclosure is shown. Figure 3, element 302 represents the memory resources (e.g., DRAM and / or other short-term memory) and processing (e.g., x86 processor, ARM processor, NIC, ASIC) resources of the respective nodes (compute, storage and / or VFS) on which the virtual file system resides, for example, as described above with respect to Figure 2 Element 308 represents one or more physical storage devices 216 that provide long-term storage for the virtual file system.

[0032] like Figure 3 As shown, physical storage is organized into multiple distributed failure resilience address spaces (DFRAS) 518. Each distributed failure resilience address space includes multiple groups of blocks 310, which in turn include multiple blocks 312. The organization of blocks 312 into groups of blocks 310 is only a convenience in some implementations and may not be performed in all implementations. Each block 312 stores committed data 316 (which can take various states discussed below) and / or metadata 314 that describes or references the committed data 316.

[0033] Organizing storage device 308 into multiple DFRAS enables high performance parallel commits (e.g., Figure 1 All nodes 104 1 –104 N , 106 1 –106 M and 120 1 –120 J Concurrent commits may be performed in parallel.) In one example implementation, each node of the virtual file system may own one or more of the multiple DFRASs and have exclusive read / commit access rights to the DFRASs it owns.

[0034] Each bucket has its own DFRAS and therefore does not need to coordinate with any other node when writing to it. Each bucket may be striped across many different chunks on many different SSDs, so each bucket and its DFRAS can choose the current "chunk stripe" to write based on many parameters, and once a chunk is assigned to that bucket, it can be written without coordination. All buckets can effectively write to all SSDs without coordination.

[0035] Each DFRAS is owned and accessed only by its owner bucket running on a particular node, thereby allowing each node of the VFS to control a portion of the storage device 308 without having to coordinate with any other node (except during initialization or during [reallocation] of buckets holding DFRAS after a node failure, which may be performed asynchronously with the actual read / commit operations of the storage device 308, for example). Thus, in such an implementation, each node may read / commit the DFRAS to its bucket independently of what other nodes are doing, without requiring any consensus to be reached when reading and committing to the storage device 308. Furthermore, in the event of a failure of a particular node, the fact that the particular node owns multiple buckets allows for a smarter and more efficient reallocation of its workload to other nodes (rather than assigning the entire workload to a single node, potentially creating a "hotspot"). In this regard, in some implementations, the number of buckets may be large relative to the number of nodes in the system, such that any one bucket may be a relatively small load placed on another node. This allows for fine-grained redistribution of the load of a failed node based on the power and capacity of other nodes (e.g., nodes with more power and capacity may receive a higher percentage of the failed node's bucket).

[0036] To allow this operation, metadata may be maintained that maps each bucket to its currently owning node so that reads and commits to storage device 308 may be redirected to the appropriate node.

[0037] Load distribution is possible because the entire file system metadata space (e.g., directories, file attributes, content ranges within files, etc.) can be broken down (e.g., chopped or fragmented) into uniform small pieces (e.g., "shards"). For example, a large system with 30k servers can divide the metadata space into 128k or 256k fragments.

[0038] Each such metadata shard can be stored in a "bucket". Each VFS node may be responsible for several buckets. When a bucket is serving metadata shards on a given backend, the bucket is considered "active" or "leader" for that bucket. Typically, there are more buckets than VFS nodes. For example, a small system with 6 nodes can have 120 buckets, while a large system with 1,000 nodes can have 8k buckets.

[0039] Each bucket can be active on a small group of nodes, typically 5 nodes that make up the quintuple for that bucket. The cluster configuration keeps all participating nodes up to date with the quintuple assignment for each bucket.

[0040] Each quintuple monitors itself. For example, if there are 10k servers in the cluster, and each server has 6 buckets, each server will only need to communicate with 30 different servers to maintain the state of its bucket (6 buckets will have 6 quintuples, so 6*5=30). This is much less than a centralized entity having to monitor all nodes and maintain cluster-wide state. Using quintuples allows performance to scale with larger clusters, because nodes don't do more work when the cluster size increases. This can have the disadvantage that in "dumb" mode a small cluster can actually generate more communication than physical nodes, but this can be overcome by only sending a single heartbeat between two servers because they share all buckets (as the cluster grows, this will change to just one bucket, but if you have a small cluster of 5 servers, it will just include all buckets in all messages, and each server will only communicate with 4 other servers). Quintuples can make decisions (i.e. reach consensus) using an algorithm similar to the Raft consensus algorithm.

[0041] Each bucket may have a set of compute nodes that can run it. For example, five VFS nodes can run a bucket. However, at any given moment, only one node in the group is the controller / leader. Furthermore, for a large enough cluster, no two buckets share the same group. If there are only 5 or 6 nodes in the cluster, most buckets can share a backend. In a fairly large cluster, there may be many different node groups. For example, in the case of 26 nodes, there are more than 64,000 possible five-node groups (i.e., quintuples).

[0042] All nodes in the group know and agree (i.e., reach consensus) which node is the actual active controller (i.e., leader) for that bucket. A node accessing a bucket may remember ("cache") the last node that was the leader for that bucket among the (e.g., five) members in the group. If it accesses the bucket leader, the bucket leader performs the requested operation. If the node it accesses is not the current leader, the node instructs the leader to "redirect" the access. If access to the cached leader node times out, the contacting node can try to use another node for the same five-tuple. All nodes in the cluster share a common "configuration" of the cluster, which allows the node to know which server can run each bucket.

[0043] Each bucket can have a load / utilization value that indicates how heavily the bucket is being used by applications running on the file system. For example, a server node with 11 buckets with low utilization can receive another metadata bucket to run before a server with 9 buckets with high utilization, even if the number of buckets used is unbalanced. The load value can be determined based on average response latency, number of operations running concurrently, memory consumption, or other metrics.

[0044] Redistribution may occur even if a VFS node is not faulty. If the system identifies that one node is busier than the others based on tracked load metrics, the system may move (i.e., "failover") one of its buckets to another, less busy server. However, before actually relocating the bucket to a different host, load balancing may be achieved by shifting writes and reads. Since each write may end up on a different set of nodes as determined by DFRAS, a node with a higher load may not be chosen in the stripe to which the data is written. The system may also choose not to serve reads from a highly loaded node. For example, a "degraded mode read" may be performed, in which blocks in a highly loaded node are reconstructed from other blocks in the same stripe. A degraded mode read is a read performed by the remaining nodes in the same stripe, and the data is reconstructed with failover. A degraded mode read may be performed when read latency is too high, because the initiator of the read may think that the node is down. If the load is high enough to create higher read latency, the cluster may revert to reading that data from other nodes and using degraded mode reads to reconstruct the required data.

[0045] Each bucket manages its own distributed erasure coding instance (i.e., DFRAS 518) and does not need to coordinate with other buckets to perform read or write operations. There may be thousands of concurrent distributed erasure coding instances working at the same time, each for a different bucket. This is integral to scaling performance because it effectively divides any large file system into independent parts that do not require coordination, thereby providing high performance regardless of the scale.

[0046] Each bucket handles all file system operations that belong to its shard. For example, directory structures, file attributes, and file data ranges will fall under the jurisdiction of a specific bucket.

[0047] An operation done from any frontend starts by finding out which bucket owns the operation. The backend leader and node for that bucket is then determined. This determination can be performed by trying the most recently known leader. If the most recently known leader is not the current leader, then the node probably knows which node is the current leader. If the most recently known leader is no longer part of the bucket's 5-tuple, then that backend will let the frontend know that it should go back into the configuration to find the members of the bucket's 5-tuple. The distribution of operations allows complex operations to be handled by multiple servers instead of a single computer in a standard system.

[0048] On-disk hashing (ODH) can manage hash tables implemented on 4k registry pages. These hash tables may have keys and values ​​of various sizes (e.g. <key, value> pairs, key-value entries), supporting many other higher-level data structures that operate within (and across) buckets.

[0049] The disclosed system can support multiple hash tables of the same kind (with different object IDs) stored on the same registry or different kinds of hash tables stored on the same registry. The hash can be based on a cryptographic hash function and provide a unique signature for each hash type and instance. The registry key for ODH can be based on the type of hash, the ID of the hash instance, and the block index. Because the registry (including, for example, the block index) is stored in RAM, it is efficient to look for a 4k block and find out if it exists.

[0050] ODH is an extensible (foldable) hash implemented efficiently on 4k blocks stored in the register. Therefore, the number of 4k blocks used is proportional to the amount of data stored on the ODH.

[0051] ODH has a dynamic structure based on key and value types and stores hash information as a sorted <key, value> array, which allows for efficient binary searches. Sorting is done based on the cryptographic hash signature of the key rather than the key itself to ensure uniform distribution of the sorting criteria across all bits. Because hash sets can consume more energy than what is consumed in a single 4k block, a mechanism for expanding and collapsing hash sets is needed.

[0052] Figure 4 An example of using ODH to split blocks of a flash registry according to an example implementation of the present disclosure is shown. A table of 400 entries and 10 bytes per entry (e.g., two bytes for the key and 8 bytes for the value) can be implemented in each 4kB ODH registry block. For illustration purposes, Figure 4The registry blocks in are shown as having up to four <key, value> entries. In addition to the local hash table of <key, value> pairs, each ODH registry block may also include header information indicating the index and split level. The split level may be stored within the block with the "key" (or hash of the block).

[0053] The elements of each ODH block are sorted by key and can be serialized one after another.

[0054] The ODH registry blocks are stored on DFRAS. Set / get operations are implemented by ODH, which is a few layers "above" DFRAS. ODH assumes there is a running registry that provides the service of storing the ODH 4k data based on the index of each such page. This also achieves write leveling when the ODH pages are updated.

[0055] As long as the number of entries fits in a block, the code can find the ODH by searching a single block. Using the type and instance IDs, the ODH can enumerate the <key, value> pairs in the block until it finds the correct pair.

[0056] When a block fills up, it increases the split level and moves some entries into new blocks based on the cryptographic hash. Entries are moved to blocks by the least significant bit (LSB) of the hash of the key. LSBs that match the index are kept, while LSBs that match the new block (increased by "1") are moved to the new block. Because the sorting and movement are based on the hash of the key, and such hashing produces a uniform distribution of hash values, about half of the entries will remain in the original block, and half of the entries will be migrated to the newly created block.

[0057] In the absence of a hash index at split level 0 <index=NIL, split=1>, an ODH can be created using one registry block 401A. The LSBs of the keys in the first four ODH entries are 0010, 1010, 1110, and 1111, and the corresponding values ​​are value_x0, value_x1, value_x2, and value_x3, respectively.

[0058] When a block is split, new blocks are added and existing blocks do not change their addresses. This maintains an "anchor point" for finding data. After the split, the anchor block is the block whose index starts with "0" because when a leading "0" is added to a number, the value of that number does not change, so the address of the block does not actually change.

[0059] exist Figure 4In FIG. 4 , when the registry block 401A is split, the new block 402A is added with the header information <index=1b, split=1>. The <key, value> pairs ending with "1" in the existing block 401A are moved to the new block 402A. The <key, value> pairs ending with "0" in the existing block 401A are retained in the hash table, and the header information is changed from <index=NIL, split=0> to <index=0b, split=1>. For the purpose of illustration, after the block 401A is split, although the physical memory address of the block is not changed, it is designated as block 401B.

[0060] After block 401A is split, ODH registry block 401B receives a <key, value> pair (0000, value_y0). Because registry block 401B has reached its capacity, registry block 401B is split and a new block 403A is added with header information <index=10b, split=2>. The <key, value> pairs ending with "10" in the existing block 401B are moved to the new block 403A. The <key, value> pairs ending with "00" in the existing block 401A remain in the hash table, and the header information is changed from <index=0b, split=1> to <index=00b, split=2>. For illustration purposes, after block 401B is split, it is designated as block 401C.

[0061] After splitting from block 401A, ODH registry block 402A receives <key, value> pairs (0001, value_y1), (0101, value_y2), and (1011, value_y3). Because registry block 402A has reached its capacity, registry block 402A is split and header information <index=11b, split=2> is added for new block 404A. <key, value> pairs ending with "11" in existing block 402A are moved to new block 404A. <key, value> pairs ending with "01" in existing block 401A are retained in the hash table, and the header information is changed from <index=1b, split=1> to <index=01b, split=2>. For illustrative purposes, after block 402A is split, it is designated as block 402B.

[0062] ODH registry block 403A receives a <key, value> pair (0110, value_z3) after being split from block 401B. Because registry block 403A has reached its capacity, registry block 403A is split and a new block 405A is added. The <key, value> pair in the existing block 403a ending with "110" is moved to a new block 405A with header information <index=110b, split=3>. The <key, value> pair ending with "010" in the existing block 403A is retained in the hash table, and the header information is changed from <index=10b, split=2> to <index=010b, split=3>. For the purpose of illustration, after block 403A is split, it is designated as block 403B.

[0063] After block 401B is split, ODH registry block 401C receives <key, value> pairs (0100, value_z0) and (1000, value_z1). Since registry block 401C has not yet reached its capacity, no split is required.

[0064] Figure 4 The diagram illustrates the ODH in a single bucket being processed by a single backend (and a single CPU core). Figure 5 An example of using on-disk hashing to split buckets of a flash registry according to an example implementation of the present disclosure is shown. To support very large hash sets, hash set processing can be supported on several processes (and several buckets). In addition, the buckets themselves can be split and / or merged to achieve load balancing. For example, when a new server is added to a cluster, existing buckets may be split and redistributed across all servers.

[0065] In About Figure 4 After the split, the ODH includes block 401C, block 403B, block 405A, block 402B, and block 404A. All five of these ODH blocks can be located on a single storage bucket 500A. For the purpose of illustration, storage bucket 500A and storage bucket 500B are the same logical storage bucket. Storage bucket 500B represents storage bucket 500A after the storage bucket split.

[0066] When the bucket splitting condition has been determined for the ODH, the ODH of bucket 500A may be split into buckets 500B and 501A, wherein bucket 501A is a new bucket.

[0067] The powerful property of this bucket split is that only half of the 4k block must be moved to the new bucket, while the remaining half of the block remains. This is true for splitting an ODH as well as splitting an entire bucket. Figure 4If the bucket of all ODHs in the local domain is split, each ODH will leave half of the <key, value> entries in the original bucket and relocate half of the <key, value> entries to the new bucket.

[0068] The split can be determined based on the LSB of the key in each block. For illustrative purposes, after the bucket is split, the corresponding blocks are designated with new version numbers (e.g., 401C becomes 401D), but the physical memory addresses of these blocks may remain unchanged. In addition, the split level (corresponding to the index width) is reduced.

[0069] The LSB of the keys in chunks 401C, 403B, and 405A is "0" (corresponding to bucket 500). These chunks are relabeled as chunks 401D, 403C, and 405B, respectively, and remain in the same buckets (i.e., bucket 500A is relabeled as bucket 500B) after the ODH on bucket 500A is split.

[0070] The LSB of the keys in chunks 402B and 404A is "1" (corresponding to new bucket 501). These chunks are relabeled as chunks 402C and 404B, respectively, and moved to bucket 501A (bucket "1") after bucket 500A is split.

[0071] When the ODH currently residing on buckets 500B and 501A is further split, the two LSBs of the key will be considered. The two LSBs of the key in block 401D are "00" (corresponding to bucket 500). Block 401D is relabeled as block 401E and remains in the same bucket (i.e., bucket 500B is relabeled as bucket 500C) after the ODH on bucket 500B (bucket "0") is split.

[0072] The two LSBs of the keys in blocks 403C and 405B are "10" (corresponding to new bucket 502). Blocks 403C and 405B are relabeled as blocks 403D and 405C and moved to bucket 502A (bucket "10") after the ODH on bucket 500B (bucket "0") is split.

[0073] In block 402C, the two LSBs of the key are "01" (corresponding to bucket 501). Block 402C is relabeled as block 402D and remains in the same bucket (i.e., bucket 501A is relabeled as bucket 501B) after the ODH on bucket 501A (bucket "1") is split.

[0074] The two LSBs of the key in block 404B are "11" (corresponding to new bucket 503). Block 404B is relabeled as block 404C and moved to bucket 503A (bucket "11") after the ODH on bucket 501A (bucket "1") is split.

[0075] Sibling blocks (i.e., two blocks at the same split level and from the same parent block) can be merged if the combined number of entries will fit into a small enough portion of the block (e.g., less than 2k of a 4k block). Figure 6 An example of using on-disk hashing to merge blocks of a flash registry is shown according to an example implementation of the present disclosure.

[0076] In About Figure 4 After the split described, the ODH includes block 401C, block 403B, block 405A, block 402B, and block 404A. Block 403B and block 405A are at the same split level (i.e., split level 3) and come from the same parent block. For example, if two <key, value> entries are deleted from block 405A, the combined number of entries (i.e., 2) will fit into a sufficiently small portion of the block. Therefore, the split level and corresponding index width of block 403B can be reduced to form block 601 (which can be located at the same address as 403B).

[0077] Likewise, block 402B and block 404A are at the same split level (i.e., split level 2) and are from the same parent block. For example, if the <key, value> entry is deleted from block 402B, and two <key, value> entries are deleted from block 404A, then the combined number of entries (i.e., 2) will fit into a sufficiently small portion of the block. Therefore, the split level and corresponding index width of block 402B can be reduced to form block 602 (which can be located at the same address as 402B).

[0078] Figure 7 is a flow chart illustrating an example method for adding a block of a flash registry using an on-disk hash according to an example implementation of the invention.At block 701, the ODH receives a <key, value> pair to be added to the registry.

[0079] At box 703, the corresponding storage bucket and block are located according to one or more LSBs of the key. The number of LSBs can be determined according to the current maximum split level. If one or more LSBs do not match the index, the number of LSBs can be reduced by one. The search for a matching index continues on the block with the second maximum split level, and so on. For example, if index = "11" is not found (assuming the maximum split level is 2), the search for index = "1" continues when split level = 1.

[0080] If adding additional <key, value> entries to the appropriate table exceeds or approaches the block capacity (at 705), a block split may be required. If the new block can be added at 707, a block split may occur at 711. If the new block cannot be added at 707, an IO error may be returned at 709. Reasons for not being able to add a new block include memory capacity and / or processor load.

[0081] If it is determined at 705 that the block capacity is sufficient or a block split is performed at 711, then the newly received <key, value> pair is added at 713. At 715, if DEC indicates that it is necessary, then the following can be done: Figure 5 Bucket split shown.

[0082] Figure 8 is a flow chart illustrating a method for accessing a block of a flash registry using an ODH according to an example implementation of the present disclosure. At block 801 , a key hash is calculated.

[0083] At box 803, the corresponding storage bucket and block are searched according to one or more LSBs of the key. The number of LSBs can be determined according to the current maximum split level. If one or more LSBs do not match the index, the number of LSBs can be reduced by one. The search for a matching index continues on the block with the second maximum split level, and so on. For example, if index = "11" is not found (assuming the maximum split level is 2), the search for index = "1" continues when the split level = 1. If the entry corresponding to the key is not found in 803, an IO error may be returned at 805.

[0084] If an entry corresponding to the key is found at 803, then the <key, value> entry in the hash table is accessed at 807. If the accessed <key, value> entry is deleted at 809, then it is possible to merge the block. If at 811, the total number of <key, value> entries in the accessed table and its sibling tables fits in a single block, then at 813, the table and its sibling tables are merged into the anchor block.

[0085] Fig. 9 An example implementation is shown in which two distributed failure resilient address spaces reside on multiple solid-state storage disks. Block 510 1,1 To 510 D,C Can be organized into multiple chunk stripes 520 1 -520 S (S is an integer). In an example implementation, forward error correction (eg, erasure coding) is used to protect each chunk stripe 520 separately. s (s is an integer where 1≤s≤S). Thus, any particular chunk stripe 520 may be determined based on the desired level of data protection. s Block 510 d,c The number of

[0086] For the purpose of illustration, assume that each chunk stripe 520 s The number of blocks 510 is N=M+K (where each of N, M and K is an integer). d,c , then N blocks 510 d,c The M of them can store data codes (usually binary digits or "bits" for current storage devices) and the N blocks 510 d,c The K of them can store protection codes (again usually bits). The virtual file system can then send each stripe 520 s Allocate N chunks 510 from N different fault domains d,c .

[0087] As used herein, a "fault domain" refers to a group of components where a failure in any one component (a component loses power, becomes unresponsive, etc.) may result in the failure of all components. For example, if a rack has a single top-of-rack switch, a failure of that switch will disconnect all components (e.g., compute, storage, and / or VFS nodes) on that rack. Thus, to the rest of the system, this is equivalent to all components on that rack failing together. A virtual file system according to the present disclosure may include fewer fault domains than chunk 510.

[0088] In an example implementation where the nodes of the virtual file system are connected and powered with only a single storage device 506 in a fully redundant manner for each such node, the failure domain may be only that single storage device 506. Thus, in an example implementation, each chunk stripe 520 s Including located in the storage device 506 1 -506 D A plurality of blocks 510 on each of N d,c (Thus, D is greater than or equal to N). An example of such an implementation is Figure 7 Displayed in.

[0089] exist Fig. 9 , D=7, N=5, M=4, K=1, and the storage devices are organized into two DFRAS. These numbers are for illustration only and are not intended to be limiting. The three block stripes 520 of the first DFRAS are arbitrarily called out for illustration. The first block stripe 520 1 By block 510 1,1 , 510 2,2 , 510 3,3 , 510 4,5 and 510 5,6 Composition: Second group of blocks strip 520 2 By block 510 3,2 , 510 4,3 , 5105,3 , 510 6,2 and 510 7,3 Composition: The third group of strips 520 3 By block 510 1,4 , 510 2,4 , 510 3,5 , 510 5,7 and 510 7,5 composition.

[0090] Although in actual implementation Fig. 9 In the simple example of , D=7 and N=5, but D may be much larger than N (e.g., greater than an integer multiple of 1, and possibly up to multiple orders of magnitude), and both values ​​may be selected so that the probability that any two chunk stripes 520 of a single DFRAS reside on the same set of N storage devices 506 (or more generally, on the same set of N failure domains) is below a desired threshold. In this way, any single storage device 506 d The failure of any particular stripe 520 (or more generally, any single failure domain) will result (expected statistics may be determined based on the selected values ​​of D and N, the size of the N storage devices 506, and the arrangement of the failure domains) in s At most one block 510 b,c Furthermore, a double failure will cause the majority of stripes to lose at most a single block 510 b,c , and only a small number of stripes (determined based on the values ​​of D and N) will be removed from any particular strip 520 s two chunks are lost in a single failure (e.g., the number of stripes with two failures may be exponentially smaller than the number of stripes with one failure).

[0091] For example, if each storage device 506 d is 1 TB, and each block is 128 MB, then storage device 506 d The failure of will result in (expected statistics may be determined based on the following: the selected values ​​of D and N, the size of the N storage devices 506 and the arrangement of the failure domains) 7812 (=1TB / 128MB) chunk stripes 520 losing one chunk 510. For each such affected chunk stripe 520 s , appropriate forward error correction algorithms and block striping 520 can be used s The other N-1 chunks are used to quickly rebuild the lost chunk 510 d,c In addition, since the affected 7812 chunk stripes are evenly distributed across all storage devices 506 1 -506 D Therefore, the lost 7812 chunks 510 are reconstructed d,cwill involve (the desired statistics may be determined based on the selected values ​​of D and N, the size of the N storage devices 506, and the arrangement of the failure domains) 1 -506 D Reading the same amount of data (ie, the burden of rebuilding the lost data) is evenly distributed across all storage devices 506 1 -506 D to recover very quickly from failures).

[0092] Next, turn to the two storage devices 506 1 –506 D In the case of concurrent failures (or more generally, concurrent failures of two fault domains), since each DFRAS block stripe 520 1 -520 S On all storage devices 506 1 -506 D Evenly distributed on the 1 -520 S Two of the N chunks will be lost. The virtual file system is operable to quickly identify such a twice lost chunk stripe based on metadata indicating that the chunk stripe 520 1 -520 S and storage device 506 1 -506 D Once such twice lost chunk stripes are identified, the virtual file system can prioritize rebuilding those twice lost chunk stripes before starting to rebuild the once lost chunk stripes. The remaining chunk stripes will have only a single lost chunk, and for them (the vast majority of the affected chunk stripes), the two storage devices 506 d Concurrent failure with only one storage device 506 d A similar principle applies to three concurrent failures (in a two concurrent failure scenario, the number of chunk stripes with three failed blocks will be much less than the number with two failed blocks), and so on. In an example implementation, chunk stripes 520 may be used to store the chunks. s The number of losses in the control execution block stripe 520 s This can be achieved by, for example, controlling the rate at which reads and commits for reconstruction are performed, the rate at which FEC calculations for reconstruction are performed, the rate at which network messages for reconstruction are transmitted, etc.

[0093] Fig.10 FIG. 5 shows a forward error correction scheme according to an example implementation of the present disclosure, which can be used to protect data stored in a non-volatile memory of a virtual file system. FIG. 5 shows a block stripe 530 of a DFRAS1 –530 4 Storage block 902 1,1 –902 7,7 .exist Fig.10 In the protection scheme of , five blocks of each stripe are used to store data codes, and two blocks of each stripe are used to store data of protection codes (i.e., M=5 and K=2). Fig.10 The protection code is calculated using the following formulas (1)-(9):

[0094]

[0095] therefore, Fig.10 Four strips in 530 1 -530 4 is part of a multi-strip (in this case, four stripes) FEC protection domain, and any block stripe 530 1 -530 4 The loss of any two or fewer blocks in is recovered by using various combinations of equations (1)-(9) above. For comparison, an example of a single-strip protection domain is: if D1 is protected only by P1 1 , D2 2 , D3 3 , D4 4 , D5 4 , and D1 1 , D2 2 , D3 3 , D4 4 , D5 4 and P1 writes all stripes 530 1 (530 1 is a single stripe FEC protection domain).

[0096] According to an example implementation of the present disclosure, a plurality of computing devices are communicatively coupled to each other via a network, and each of the plurality of computing devices includes one or more of a plurality of storage devices. A plurality of failure resilient address spaces are distributed over the plurality of storage devices, such that each of the plurality of failure resilient address spaces spans across the plurality of storage devices. Each of the plurality of failure resilient address spaces is organized into a plurality of stripes (e.g., Fig. 9 and Fig.10 Each or more of the plurality of stripes is a plurality of forward error correction (FEC) protection domains (e.g., such as Fig.10 Each of the plurality of stripes may include a plurality of storage blocks (e.g., a plurality of 512). Each block of a particular stripe in the plurality of stripes may be located on a different storage device in the plurality of storage devices. A first portion of the plurality of storage blocks (e.g., Fig.10Strip 530 1 of 902 12 –902 16 The number of 5 composed of the plurality of storage blocks) can be used to store the data code, and the second part of the plurality of storage blocks (for example, Fig.10 Strip 530 1 Two 902 1,1 and 902 1,7 The number of) can be used to store protection codes calculated at least in part based on data codes. Multiple computing devices can be operable to sort multiple stripes. The sorting can be used to select which of the multiple stripes is used for the next commit operation of one of the multiple fault-resilient address spaces. The sorting can be based on how many protected and / or unprotected storage blocks there are in each of the multiple stripes. For any of the multiple stripes, the sorting can be based on a bitmap stored on multiple storage devices having a specific one of the multiple stripes. The sorting can be based on the number of blocks currently storing data in each of the multiple stripes. The sorting can be based on the read and write overhead for committing to each of the multiple stripes. Each fault-resilient address space can be owned by only one of the multiple computing devices at any given time, and each of the multiple fault-resilient address spaces can only be read and written by its owner. Each computing device can own multiple fault-resilient address spaces. Multiple storage devices can be organized into multiple fault domains. Each of the multiple stripes can span multiple fault domains. Each failure resilient address space may span all of the multiple failure domains such that when any particular failure domain in the multiple failure domains fails, the workload for rebuilding the lost data is distributed between each other failure domain in the multiple failure domains. The plurality of stripes may be distributed across the multiple failure domains such that in the event of a simultaneous failure of two of the multiple failure domains, the probability of two blocks in any particular stripe in the multiple stripes on the two failure domains in the multiple failure domains being present is less than the probability of only one block in any particular stripe in the multiple stripes on the two failure domains in the multiple failure domains being present. The plurality of computing devices may be used to first rebuild any of the multiple stripes with two failed blocks and then rebuild any of the multiple stripes with only one failed block. The plurality of computing devices may be used to perform the reconstruction of the multiple stripes with two failed blocks at a higher rate (e.g., a greater percentage of CPU clock cycles dedicated to reconstruction, a greater percentage of network transmission opportunities dedicated to reconstruction, etc.) than the reconstruction rate of the multiple stripes with only one failed block. The plurality of computing devices may be used to determine the rate at which any particular lost block is rebuilt based on how many other blocks of the same stripe in the multiple stripes are lost in the event of a failure of one or more failure domains. Wherein, one or more of the plurality of fault domains include a plurality of storage devices. Each of the plurality of FEC protection domains may span a plurality of stripes of the plurality of stripes. The plurality of stripes may be organized into a plurality of groups (e.g., Fig. 9 Chunk stripe 520 1 -520 S ), wherein each of the plurality of groups includes one or more of the plurality of stripes, and the plurality of computing devices are operable to sort, for each group, one or more of the plurality of stripes of the group. The plurality of computing devices are operable to: perform continuous commit operations on a selected one of the plurality of groups until one or more of the plurality of stripes of the group no longer satisfies a determined criterion, and select another one of the plurality of groups when the selected one of the plurality of groups no longer satisfies the determined criterion. The criterion may be based on how many blocks are available for new data to be written.

[0097] Although the present method and / or system has been described with reference to certain implementations, it will be appreciated by those skilled in the art that various changes may be made and equivalents may be substituted without departing from the scope of the present method and / or system. In addition, many modifications may be made to adapt a particular situation or material to the teachings of the present invention without departing from the scope of the invention. Therefore, it is intended that the present method and / or system is not limited to the particular implementation disclosed, but rather that the present method and / or system will include all implementations that fall within the scope of the appended claims.

[0098] As used herein, the terms "circuit" and "circuitry" refer to physical electronic components (i.e., hardware) and any software and / or firmware ("code") that can configure hardware, be executed by hardware, and / or be associated with hardware. As used herein, for example, a specific processor and memory may include a first "circuit" when executing a first line or more lines of code, and may include a second "circuit" when executing a second line or more lines of code. As used herein, "and / or" refers to any one or more items connected by "and / or" in a list. As an example, "x and / or y" represents any element in a three-element set {(x), (y), (x, y)}. In other words, "x and / or y" refers to "one or both of x and y". As another example, "x, y and / or z" represents a seven-element set {(x), (y), (z), (x, y), (x, z), (y, z), (x, y, z)}. In other words, "x, y and / or z" refers to "one or more of x, y and z". As used herein, the term "exemplary" means serving as a non-limiting example, instance, or illustration. As used herein, the terms "for example" and "for example" lead to a list of one or more non-limiting examples, instances, or illustrations. As utilized herein, a circuit is "operable" to perform a function so long as the circuit contains the necessary hardware and code (if necessary) required to perform that function, regardless of whether the performance of that function is disabled (e.g., by a user-configurable setting, a factory adjustment, etc.).

Claims

1. A system, include: A registry of multiple storage blocks, where: The registry is adaptively sized. The registration table includes one or more registration blocks. Each registry block consists of a split level and one or more keys. Each of the one or more keys comprises the same number of bits, an index into each of the one or more keys is indicated by a specified number of bits, and The specified number is the split level.

2. The system according to claim 1, in, The plurality of memory blocks include non-volatile memory.

3. The system according to claim 1, in, The plurality of storage blocks include flash memory.

4. The system according to claim 1, in, The size of the registration table is adaptively adjusted as one or more storage blocks are written to.

5. The system according to claim 1, in, The size of the register is adaptively adjusted as one or more storage blocks are read.

6. The system according to claim 1, in, Each of the one or more keys in each of the one or more registry blocks is associated with one of the one or more key-value entries.

7. The system according to claim 6, in, The key-value entries are added to a registry block of the one or more registry blocks until the number of the one or more key-value entries exceeds a predetermined capacity, at which time the registry block is split and a new registry block is added to the one or more registry blocks.

8. The system according to claim 6, in, The one or more key-value entries are deleted from a registry block of the one or more registry blocks until the number of the one or more key-value entries is equal to or below a predetermined level, at which time the registry block is merged with another registry block of the one or more registry blocks.

9. The system according to claim 1, in, The storage network includes a fail-over address space distributed across multiple storage devices.

10. The system according to claim 8, in, Each registry block is identified by an index and a split level.

11. A method, include: Maintaining a registry of one or more storage blocks; as well as Adaptively adjust the size of the registry, wherein The registration table includes one or more registration blocks. Each registry block consists of a split level and one or more keys. Each of the one or more keys comprises the same number of bits, an index into each of the one or more keys is indicated by a specified number of bits, and The specified number is the split level.

12. The method according to claim 11, in, The plurality of memory blocks include non-volatile memory.

13. The method according to claim 11, in, The plurality of storage blocks include flash memory.

14. The method according to claim 11, in, The size of the registration table is adaptively adjusted as one or more storage blocks are written to.

15. The method according to claim 11, in, The size of the registration table is adaptively adjusted as the one or more storage blocks are read.

16. The method according to claim 11, in, Each of the one or more keys in each of the one or more registry blocks is associated with one of the one or more key-value entries.

17. The method according to claim 16, in, The method comprises: adding the one or more key-value entries to a registry block of the one or more registry blocks; and If the number of the one or more key-value entries exceeds a predetermined capacity, the registry block is split and a new registry block is added to the one or more registry blocks.

18. The method according to claim 16, in, The method comprises: Deleting the one or more key-value entries from a registry block of the one or more registry blocks; and If the number of the one or more key-value entries is equal to or below a predetermined level, the registry block is merged with another registry block of the one or more registry blocks.

19. The method according to claim 11, in, The storage network includes a failure-resilient address space distributed across multiple storage devices.

20. The method according to claim 18, in, Each registry block is identified by an index and a split level.

Images