Systems, methods, and devices for improving local object storage

Abstract

The present invention is based on and greatly extends the concept of block devices to incorporate additional features and functionality required to implement high performance and high value storage subsystems. This Extended Block Device (EBD) normal form eliminates the restrictive concept imposed by the block device and is replaced with more flexible options for use by the application layer, providing single IO access to any file.

Description

[0001] Cross Reference to Related Applications

[0002] This international application claims priority to U.S. Provisional Patent Application Serial No. 63 / 529,611, filed July 28, 2023, entitled “SYSTEM, METHOD AND APPARATUS FOR IMPROVED LOCAL OBJECT STORAGE,” the disclosure of which is incorporated herein by reference. TECHNICAL FIELD

[0003] This application generally relates to techniques that increase the flexibility of block storage devices by eliminating restrictive concepts that limit the existing block device paradigm, methods that better allocate logical block addresses by providing variable size blocks, and methods that improve atomic updates. BACKGROUND

[0004] A block device is a computer component, such as a disk drive and other mass storage devices such as flash and RAM-based disks. Traditionally, for block storage devices, the application using the storage accesses the device using a “block number.” The device driver then converts the block number to a physical address on the device. This conversion process usually involves linearly mapping the block number to the corresponding location on the block storage device.

[0005] However, in considering object storage performance, the Applicant here assumes and describes a file system that implements “single IO” access to any file, approaches theoretical update performance, is compatible with Shingled Magnetic Recording (SMR) and zoned media, and has outstanding data integrity.

[0006] The file system part of the invention is inspired by databases from the 1980s and how to implement its coarse structure on top of a “Fast Block Device” or FBD, for which the Applicant first applied for a patent in 2008, and on which the significant improvements of the invention are built.

[0007] The Applicant’s research results prove the effectiveness of the “Extended Block Device” or EBD and how it can optimize file systems, databases, and other storage applications.

[0008] Most storage subsystems are built on top of block devices, which provide the basic functionality for storage solutions but lack many of the features and functions required by storage solutions. What is needed is greater flexibility in this regard and a new paradigm of operation. SUMMARY

[0009] The present invention is a new paradigm that extends the concept of the above block device to encompass additional features and functionality required to implement high performance and high value storage subsystems. This "Extended Block Device" (EBD) eliminates the restrictive concept imposed on block devices and replaces it with more flexible capabilities and functionality for the application layer to utilize, providing single IO access to any file, re-imagining the entire operational paradigm. BRIEF DESCRIPTION OF DRAWINGS

[0010] While the specification concludes with claims particularly pointing out and distinctly claiming that which is regarded as the present invention, it is believed that the present invention will be better understood from the following description in conjunction with the drawings, in which like reference numerals denote similar elements and components throughout the figures.

[0011] The various drawings set forth herein provide representative views of various aspects and features of the extended block device article, system, technique, device and method employing the principles of the present invention in illustrative configurations, wherein:

[0012] Figure 1 A simple conventional block storage and access paradigm of the prior art employing uniform size byte blocks is generally shown;

[0013] Figure 2 A first improvement to the simplified prior art as shown in Figure 1 shows individual variable length byte blocks instead of fixed blocks, where these blocks hold different amounts of data for each logical block address (different block byte lengths);

[0014] Figure 3 Another feature of the present invention, the sparsity of logical block address allocation, is generally shown, where Figure 2 Variable size blocks, also shown in

[0015] Figure 4 Another aspect of the present invention is shown that can be implemented through this paradigm, where Figure 2 and Figure 3 Contiguous or scattered groups of variable size blocks, as shown in DETAILED DESCRIPTION

[0016] The following detailed description is provided to enable those skilled in the art to make and use the invention. Specific terminology is set forth for purposes of explanation to provide a thorough understanding of the invention. However, those skilled in the art will understand that these specific details are not necessary for practicing the invention. Descriptions of specific applications are provided only as representative examples. Various modifications to preferred embodiments will be apparent to those skilled in the art, and the general principles defined herein can be applied to other embodiments and applications without departing from the spirit and scope of the invention.

[0017] This invention is not intended to be limited to the embodiments shown, but should be accorded the broadest possible scope consistent with the principles and features disclosed herein.

[0018] As described above, the aforementioned prior art, particularly the fast block device and related method according to the applicant's prior invention, is a device mapping layer with a purpose entirely different from standard block devices. Instead of simply linearly translating logical block numbers into physical device addresses, the fast block device and related method dynamically remaps data to optimize data access and update patterns. This dynamic remapping can be used with a variety of storage devices to achieve significant performance improvements over linearly mapped devices, as well as other benefits for certain specialized types of hardware. For example, when the fast block device concept is applied to flash memory, the speed of random writes to that device can be increased by nearly two orders of magnitude.

[0019] These early and ongoing efforts in the field of fast block devices are illustrated and explained by way of example in the applicant’s various patents, such as U.S. Patent Nos. 8,380,944, 8,812,778, 9,092,325, 9,535,830, 10,248,359, 10,817,185, 10,860,255, 11,455,099, 11,687,445 and 12,045,162 (collectively, the “Applicant’s Prior Art”), the disclosures of which are incorporated herein by reference.

[0020] Now for reference Figure 1 The diagram illustrates the paradigm of the prior art described above, generally denoted by reference numeral 100, where each block is of the same size, generally denoted by reference numeral 110. In this paradigm, each logical block address uses storage space regardless of whether it actually contains content or has ever been written to. The improvement of this invention solves this inefficiency.

[0021] This invention begins by extending the concept of block devices proposed above and in other disclosures to include the additional features and functionalities required to implement high-performance and high-value storage subsystems. As previously stated, the “Extended Block Device” (EBD) technology and methods proposed herein first eliminate the restrictive concepts imposed by the block device paradigm and replace them with more flexible application-layer options.

[0022] To illustrate the various components involved in different embodiments of the improved method of the present invention, the following topics are discussed.

[0023] Sparse LBA Logical block addresses (LBAs) typically have a 1:1 relationship with disk blocks, such as... Figure 1 As shown. However, EBD first allows for any number of these LBAs, which in turn makes it easier to assign and delete LBAs.

[0024] Tracking LBA allocation The allocation and release of LBAs are now handled by EBD. Applications no longer need to manage allocation bitmaps or use other LBA "available list" methods.

[0025] Variable size "chunks" LBA references a variable-size byte array, not a fixed-size block. For example... Figure 1 As shown, the flexibility offered by this new paradigm is unparalleled in existing technologies, and allows applications to directly map structures of different sizes to a single LBA, thereby significantly reducing the overhead and complexity of implementing solutions.

[0026] For example, now refer to Figure 2 This improved variable-length block paradigm is illustrated, generally denoted by reference numeral 200. As shown, the blocks described above come in various sizes, from small blocks, generally denoted by reference numeral 211, that may be a few hundred bytes, to medium-sized blocks, generally denoted by reference numeral 212, that may be several thousand bytes, generally denoted by reference numeral 213. As discussed, the variableness of the block size preferably depends on powers of 2.

[0027] Now for reference Figure 3 The sparse allocation process of the present invention is illustrated, generally denoted by reference numeral 300. In this embodiment, variable-sized blocks can occupy each logical block address, as shown in blocks 311, 312, and 313, or can be left blank for future use, generally denoted by reference numeral 320. These blanks 320 do not consume storage space. Having a “sparse” allocation strategy makes the design of many applications easier because they can use contiguous LBA allocation ranges without having to use them immediately or forever.

[0028] Atomic updateUpdates are no longer limited to the "single-block write" model of block devices, as described in detail in the applicant's prior art. Instead, multiple LBA assignments and multiple LBA updates can now be combined into a formal "atomic update" transaction, which is therefore guaranteed to arrive at the media intact. These updates can also be appended to new data and existing data in use, and they can be effectively merged to create an update engine that is efficient and convenient for applications that need to maintain data integrity in the event of system failure.

[0029] refer to Figure 4 The use of atomic writes within the paradigm of this invention is illustrated, generally indicated by reference numeral 400. Multiple groups of blocks, such as those indicated by reference numerals 430A, 430B, and 430C, can be updated together as a single atomic transaction. As mentioned above, a blank 420 exists. Atomic transactions guarantee that either all involved blocks are stored intact on the medium, or none are stored on the medium. Atomic updates are one of the most difficult parts of building a storage engine and typically involve multiple copies of the same data. This invention implements atomic updates as part of a single write stream of packaged blocks, thereby achieving storage efficiency, update consistency, all with a simple and easy-to-use interface. This simplicity contrasts sharply with the complexity of handling atomic writes in previous paradigms.

[0030] As stated above, the concept of this application is an important extension to the earlier fast block device paradigm, such as Figure 1 As illustrated in the applicant's prior art, this case relates to the creation of an "Extended Block Device" (EBD) and the creation of objects using that EBD to optimize the file system.

[0031] For example, in one embodiment of the invention, the block mapping table may optionally be stored in "virtual" memory. The table itself is preferably divided into multiple fixed-size regions, and each region preferably contains a portion of the mapping table. Padding bytes may be appended to the end of each fixed-size region if it can be evenly divided by the size of the mapping table elements. The block mapping table virtual memory device is itself an FBD device and optionally resides on the same medium as the primary FBD device. This improved technique allows the FBD to operate with significantly lower dedicated memory requirements.

[0032] Therefore, the number of blocks and the maximum logical block address, or LBA, in this invention can be set to arbitrarily large numbers. This implements sparse LBA, which is known to be useful for application design.

[0033] The mapping blocks in this invention are enlarged to allow for the representation of more information beyond the addresses of stored data, as is the case in the art. This additional information will be described in more detail below.

[0034] The mapping block can now store whether an LBA is available or has been allocated. As mentioned above, it should be understood that an LBA can be allocated, but it cannot yet store any information.

[0035] The present invention includes an interface that is created to allow applications to allocate and release LBAs, as described above, whether allocated individually or in blocks of binary size, with an upper limit of at least approximately 32,768 consecutive LBAs.

[0036] As previously stated, in the embodiments of the present invention described herein, each LBA can reference a variable-size byte array, as described above. Figures 2-4 This differs from the standard definition of a "block device," where all blocks are the same size, as described above. Figure 1 The example illustrates this. Here, each LBA can reference data ranging in size from empty to approximately 16 megabytes. Of course, it should be understood that this limitation may vary depending on the implementation and will change as technology evolves. Because variable-sized blocks are stored contiguously on the medium, this allows for “single IO” access to data structures of any size with efficient space utilization.

[0037] Variable-size blocks are ideal for applications that naturally have variable-size structures. This includes "hash sets," where hash logic creates a natural distribution of size.

[0038] According to embodiments of the present invention, all updates to the mapping table and related blocks are performed as formal atomic updates, such as... Figure 4 The example illustrates this. Although atomic updates are part of the original FBD structural paradigm, they are only exposed at the application layer. The atomic updates according to the invention now include multiple LBAs and multiple mapping blocks, such as the block groups 430A, 430B, and 430C described above. Atomic updates can be appended to and merged into available atomic update buffer limits.

[0039] All updates are maintained using the FBD structure and engine as FBD linear updates.

[0040] This invention supports secondary writes. The LBA can reference secondary writes made before the primary atomic update. These updates are preferably stored in the FBD map and are entirely managed by the FBD atomic update engine. Secondary updates are written to "secondary regions," either in another region of the current medium or on separate media. Secondary writes are entirely part of the FBD atomic update transaction engine.

[0041] POSIX (Portable Operating System Interface) is a set of standard operating system interfaces based on Unix operating systems. In another embodiment, the present invention preferably employs a POSIX file system using EBD features. EBD allows for a "direct access" file system that provides "single I / O" file access using a direct hashing algorithm. This type of algorithm is generally difficult to implement, but the EBD feature set of the present invention allows for direct mapping of hash block structures to EBD variable-size blocks. EBD atomic updates allow for hash block splitting / merging without regard to data integrity semantics.

[0042] Key / value or KV databases are similar to the POSIX file system mentioned earlier, but it is also conceivable to use a different application programming interface (API). Similarly, object storage applications similar to the POSIX file system are also envisioned, but using object get / place APIs.

[0043] Regarding other embodiments of the present invention, the following is another description of the techniques employed, as well as historical background for illustrating these concepts.

[0044] As background technology, most storage systems are built using "disk drives," which provide applications with a block device abstraction, as described in detail in conjunction with the applicant's prior art. This block device is a collection of data blocks of the same size, each addressable using an LBA (Logical Block Address), such as... Figure 1 As shown. These blocks themselves may be 512 bytes long, or 4096 bytes long, or some other size, but the operational concept here is that they are all the same size.

[0045] It all starts with the actual disk drive. Physical sectors are mapped to LBAs. These sectors can be retrieved or updated in any order, without any restrictions. Each LBA has a physical location that stores one byte.

[0046] Over time, the definition of "actual disk" began to expand. First, the concept of a "bad block table" was created to hide media defects from applications. A small subset of sectors would be "remapped" by the disk controller to different unused portions of the disk drive.

[0047] Then came “restricted write media.” This is a type of storage media that allows random reading but has update rules that prohibit random writing. These are widely used today along with NAND flash-based SSDs (Solid State Drives) and SMR (Shingled Magnetic Recording)-based hard drives. These devices expose the actual underlying media, and some models of SSDs and HDDs (Hard Disk Drives) do just that. For SSDs, these are called “partition drives.” For SMR HDDs, these are called “host-managed SMR drives.” Most of these drives don’t expose the limitations of the media but instead use a drive-level controller to overlay the apparent ability to write randomly, even if the media doesn’t. For flash SSDs, this layer is called the FTL (Flash Translation Layer). This layer maps logical block addresses to flash media locations. This mapping is not static; the mapping table changes with each new update. FTLs are complex to design, especially in the area of ​​handling unexpected downtime events without data corruption. SMR hard drives typically have a layer similar to the FTL, but the internal mechanisms are often very different and the FTL name is not used.

[0048] Everything is still in blocks: in all these existing technologies, the concept of LBAs pointing to fixed-size blocks of bytes still exists.

[0049] FTL can break this paradigm: the internal logic of FTL, at least the software-based FTL of the subject of this invention, breaks this paradigm in a new way. While existing applications expect the block device paradigm, another set of abstractions more suited to specific storage scenarios can be constructed. As previously mentioned, this new paradigm is called "extended block device".

[0050] As illustrated in the various embodiments described above and herein, the extended block device differs from the prior art (including the applicant's prior art) in at least four important ways in terms of the content provided to this application.

[0051] First, the size of the block is variable, such as Figure 2 As shown. These are more "binary large object blobs" than blocks. For example, if an application writes 174 bytes to the LBA, future reads will read 174 bytes. Implementation limitations exist, but they are wide enough to treat many storage structures directly as single blocks without the inefficiencies and complexities of splitting blocks into sub-parts or chaining multiple blocks together. Current implementation limitations are a lower limit of 16 bytes and an upper limit of approximately 4 megabytes. Special cases of zero-byte blocks also exist. As discussed, the principles of this invention are not limited by the details of this implementation.

[0052] Secondly, LBA is sparse, such as Figure 3As shown, the device has a number of LBAs far exceeding its capacity. This is convenient for applications because they can assign a large range of consecutively numbered LBAs. Consequently, applications can build very large structures without needing to match a large extent table. Combined with large blocks, the size of the extent table in the file system can be reduced by approximately one million times.

[0053] Third, block allocations are large, definable, and traceable. An application can allocate 1 to approximately 32,768 consecutive LBAs in a single call. Allocations must be powers of 2, and the returned LBAs will always be on power-of-2 boundaries.

[0054] Fourth, updates are flexible and atomic, such as... Figure 4 As shown. Technically, extended block devices support Large, Mergeable, Scatter Gather, Non-Contiguous, Allocation Aware, and Atomic Updates.

[0055] For large-scale updates, which can contain many megabytes, the current implementation guarantees that at least four LBAs of the maximum size can be stored as part of a single atomic update.

[0056] Regarding merging, updates can be merged before they are committed. If three LBAs are updated in one transaction, and a second transaction includes three LBAs, two of which overlap with queued updates, a new update transaction will be constructed, which includes the four LBAs as a combination of transactions. This allows updates to efficiently utilize device bandwidth while still maintaining update consistency for truly atomic updates.

[0057] Regarding distributed collection, the update engine allows applications to represent blocks as long lists of memory addresses without needing to pre-set contiguous memory, which was required by earlier paradigms in the prior art.

[0058] Regarding non-contiguous groups, LBAs can be in any order and do not need to be consecutive groups.

[0059] Regarding allocation sensing, LBA allocation and release are part of the atomic update structure.

[0060] The importance of the atomic update engine described in this article cannot be underestimated. A large portion of the logic and overhead of a database or file system is dedicated to data integrity in the event of a crash. This is why journaling, copy-on-write, and other techniques exist. The extended block device of this invention directly builds complex structures in situ, ensuring consistency across the media, without any of these steps.

[0061] For example, the "new object creation" operation according to the present invention involves: one or more assignments of LBAs, updating of one or more existing control LBAs, and updating of one or more new LBAs.

[0062] Since all of these can be performed in a single atomic update, these operations either all enter the media or none of them do. Logs are no longer needed, and applications no longer need to worry about the disordered update order to maintain data integrity—a serious problem in existing technologies.

[0063] Regarding extended block device implementations, these "devices" are implemented in software on traditional block devices and / or raw flash or SMR media. The underlying device sees the workload associated with FTL. Data is densely packed, creating excellent storage performance and utilization. For media with limited write persistence, such as NAND flash, update wear is minimized.

[0064] Regarding the file system prototype, a prototype file system has been built using these techniques. This file system is optimized for fast file creation and retrieval. It is not used for in-place block updates within files.

[0065] In this exemplary prototype, the file system behaves more like a key / value database or object store, exhibiting a file system with directories, etc. The performance profile of this file system is essentially "one I / O" to retrieve any file and linear writes to create files. Overall, it is very close to the theoretical efficiency limit, although the file system interface is generally considered less convenient.

[0066] Benchmark tests comparing the file system envisioned in this invention with a key / value (KV) database show that the file system outperforms in most operations, especially for very large datasets (which would be very large directories). For a single Serial Advanced Technology Attached (SATA) SSD, the update rate of a single thread writing files directly to a single directory exceeds 250,000 updates per second. These architectures support high parallelism, creating outstanding performance in mixed workloads involving millions of directories and billions of files.

[0067] By further describing the background and basis of the invention, a variable block size data storage solution, the scalability of the invention relative to existing block mapping technologies is discussed below in conjunction with another embodiment. As discussed, the block mapping layer is significantly and exemplary extended here to support new features that are not adequately supported in the prior art.

[0068] As previously stated, the present invention comprises three main features: variable-size data “blocks”, tracking block allocation and releasing in a binary range of 1 to 32K, and having large, mergeable atomic updates, all of which form a layer called “VBSFBD” for variable-size fast block devices.

[0069] Combining these three characteristics, key / value storage technology can be directly mapped to media with the following data structure. This structure can be used to represent key / value storage "buckets" or directories in a file system. As mentioned above, this storage structure is inspired by the FILE / ITEM storage structure, which first appeared in a class of databases commonly known as "Pick databases" (named after its inventor, Richard Pick), dating back to the 1970s. This database is known for using hash lookup techniques to perform "single IO" access to any data record.

[0070] The technique described in this article retains the advantages of this original database design, but allows for automatic scaling of storage as items are added and removed in a more flexible environment, while still maintaining the "single IO" performance target. Other aspects of the original database design have also been extended to allow for large "blob" storage exceeding 1 exabyte or 1 EB. Specific features of this embodiment will be discussed below.

[0071] Header 0 This is a small block containing counter fields. Due to the nature of the directory structure, this block is updated every time a key-value pair or file (hereinafter referred to as an item) is added, modified, or deleted. Because this block is small, the VBSFDB write overhead is minimal, and this block can easily become part of an atomic VBSFDB update. Preferred fields in this block include: the number of items, the number of active groups, the number of assigned groups, permissions, and timestamps. Of course, it should be understood that alternative or additional fields can be considered.

[0072] Header 1 This is a list of grouped extension regions. It contains a list of LBNs (Logical Block Numbers) that hold multiple groups of items. This list is used for contiguous allocation, starting with a single LBN for a single group and growing to 32K or more LBNs with each allocation. This allocation scheme guarantees minimum utilization efficiency while keeping the length of the extension region list low enough that the list itself can be stored in a single VBSFBD "block". This allocation scheme also guarantees low update frequency when this extension region list is large, reducing update overhead even for KV / directory sets containing trillions of items.

[0073] The preferred fields in this block include: the number of extended extents and an array of extended extent LBNs. Furthermore, each extended extent is preferably allocated starting from 1, 2, 4, ..., 32K. This binary approach guarantees minimum allocation efficiency, keeping lookups and extended extent table size fast. Of course, it should be understood that alternative or additional fields can be considered.

[0074] Packet This is the core of the KV / DIR structure. It contains a list of linearly packed items. Preferred fields in this block include a header containing the number of items in the group, and an array of the length of each item. It should be understood that this technique is particularly good for fast grouped searches.

[0075] The DIR layout becomes a collection of VBSFBD "blocks" allocated in binary increments. The directory itself consists of two blocks allocated together (Hdr0 and Hdr1). As groups are added, they are allocated in binary increments, except that the first two allocations are single blocks.

[0076] ITEM The header of a single item is packaged into a group. The group number is calculated using a hash function. If the item is small, it can be stored within a "group". Control information for larger items is stored in "groups", and its contents are stored in one or more VBSFBD "blocks" using an extended area table. The extended area table for an item matches the data layout of Header1. When the extended area table is small, it is stored in the group; when it becomes large, it is stored outside the group. This, combined with the large maximum block size of VBSFBD, allows very large (>1EB) "blobs" to be stored using only a single extended area table. This structure is optimized for the storage and retrieval of large items, guaranteeing linearity of 4MB or higher for large "blobs" without the possibility of fragmentation.

[0077] The preferred fields in this item include: the item hash / Len (8 bytes) for fast matching during group lookups; and the item key (filename), which is validated if the hash / Len matches. Because the lengths are already matched, memory comparison is used instead of string comparison operations. Therefore, memory comparison operations are much faster. The preferred fields in this item also include: permissions; and a timestamp.

[0078] For small items, the contents can be included. For medium-sized items, a small extended area table can be included. For large items, any number of LBAs of an external extended area table can be included.

[0079] ITEM lookupThis is accomplished using a hashing method. Using hashing for key-value lookups is a common technique. Hash lookups have two limitations. First, the number of hash buckets (called groups) needs to be predicted; second, the size of the hash buckets can vary.

[0080] The first problem with bucket count was alleviated by using the grouping "split-merge" functionality, which first appeared in databases in the mid-1980s. This solution is believed to have originated from a university paper, but the original author and date are unknown (although the paper predates 1985). This technique has been used in some commercial database products, including Prime Information and OpenQM.

[0081] The complexity of this prior art lies in its deterioration of the distribution of group sizes and its complex update process, which makes it difficult to protect data integrity. The LVSFBD of this invention eliminates both of these problems. The variability of group size is entirely within the constraints of variable block size. The mergeable atomic updates in LVSFBD make updates a single atomic operation, encompassing all allocations, control blocks, and multiple groups, all used for group splitting and merging, without requiring additional logic, as long as the entire update is committed in a single transaction. This becomes negligible for the LVSFBD techniques described herein in various embodiments.

[0082] Regarding device performance, the VFSFBD layer implements a linear write stream. This allows the atomic update engine to merge blocks of different sizes, each using only the number of bytes of its power space and bandwidth. Split / merge operations are tuned to occur only every X operations (set to 32 in tests, but this is adjustable), keeping split / merge overhead below 5%.

[0083] Regarding the split / merge implementation, group splitting / merging is a separate function. For splitting, a single group is split into two groups. During the operation, these groups are locked. The source group is read from the medium before locking, and the lock is released once the update is scheduled as an atomic write. This keeps the lock local and for a short duration.

[0084] Split and merge operations also apply to pre-fetch I / O operations. Tests show that continuous KV creation operations exceeding 2 million per second for a single file are easily achievable. Aggregation operations on multiple files scale well because there is no cross-locking between different files.

[0085] The merge operation is similar, except that two packets are pre-read from the medium before setting the lock.

[0086] ITEM contentThe payload of an item can range from a few bytes to several gigabytes. This design is optimized for "KV-style" or "object-style" access, where the entire item is created at once (or at least in large chunks) and retrieved in one go. The item extension area is stored in a list of large, variable-size chunks, 4MB at a time. This eliminates any impact of LBN fragmentation.

[0087] The following is an example of using extended block devices in a file system "directory" design.

[0088] First, the directory structure is the basic unit of lookup in a file system. A directory contains a list of file descriptors. It is crucial that directories remain efficient in terms of space usage and the amount of I / O required to retrieve or update entries. A particularly complex part of file system design is ensuring that directory updates can be completed correctly even in the event of a system crash, without leaving corrupted blocks on the media.

[0089] This design uses a hash directory. This means the directory has multiple assigned "groups," each storing a set of file descriptors. One advantage of the group lookup method is that it allows direct access to named files with a single computation plus one I / O operation. The disadvantage of hash lookup is that the underlying storage becomes complex due to the variable size of the groups.

[0090] The directory begins with two variable-size blocks. One is used to store a count of the directory and other simple data such as update timestamps and permissions. The second is used to store a table of LBAs that group the directories.

[0091] As shown below, the first block is quite small. The second block starts small but grows as the directory group count increases.

[0092]

[0093] The above list of block extension regions is a linear array of LBA numbers. Each number represents the starting LBA of a block. Since the LBAs are allocated in powers of 2, these numbers represent a continuously growing sequence of LBAs, which can be represented with a relatively small amount of storage. For example:

[0094]

[0095] This power-of-two allocation increment is highly efficient. A 4MB block can hold 512,000 entries. These entries can represent 1,600,000,000 (16 billion) groups, which is a file containing over 250 billion files in a single directory. Larger directories can be handled using an extended group extension list. This creates a scenario where group LBAs can be located with very few operations in a very small in-memory lookup table.

[0096] As files grow (or shrink), the table must grow or shrink accordingly, but for small directories, the block is also small, so the overhead of updating is low. As the directory file count grows, the update frequency slows down as the scope of LBA becomes larger.

[0097] The worst-case scenario for updating is when creating files that need to be split into groups, which also requires assigning new LBAs to the groups. This operation involves the following elements.

[0098]

[0099] Considering the specific size and numbers, if the 16384 packets are split into 16385 packets, hdr0 will be very small (approximately 200 bytes). The packet extension block has 17 entries, so its length is only 136 bytes. The size of these two packets is variable, but roughly in the range of 2K to 4K. The 16K "LBA allocation" operation does not actually involve blocks, but needs to be recorded in the atomic update, and approximately 16 bytes are recorded. Therefore, the total amount of writing to the medium for this very complex operation is less than 5K, which will be linear due to the working principle of the block conversion engine.

[0100] For traditional structures, this would require non-linear writes to multiple blocks and logs. Recovery from a crash would necessitate cleanup of the structure. EBD's atomic update feature enables file system "applications" to perform this update in a single, guaranteed safe step, ultimately creating a small, linear I / O operation. Even better, the atomic update engine can combine a large number of such updates into a single I / O operation, bringing file I / O update operations close to the linear speed of the media.

[0101] Unless otherwise specified, the use of the article "a" or "an" to modify a noun in this text can be understood to include one or more of the nouns being modified.

[0102] While the systems and methods described herein have been shown and described with reference to the illustrated embodiments, those skilled in the art will recognize or be able to determine many equivalents of the embodiments described herein using only conventional experimentation. These equivalents are included within the scope of this disclosure and the appended claims.

[0103] Therefore, the systems and methods described herein are not limited to the embodiments described herein, but may include practices other than those described, and should be interpreted broadly to the extent permitted by applicable law.

[0104] Finally, the systems and methods described in the various embodiments are not necessarily limited to these specific embodiments, but are defined by the appended claims, their literal counterparts, and by all equivalent counterparts in accordance with the principles of equivalence determination and equivalence.

Claims

1. A method for improving object storage performance, comprising: Write the new data linearly to at least one variable-size data block. Each of the variable-size data blocks is tracked by a corresponding logical block address. The method thus provides single I / O access performance for any object stored in this way.

2. The method of claim 1, wherein at least two of the variable-size data blocks are consecutive.

3. The method of claim 1, wherein at least two of the variable-size data blocks are non-contiguous.

4. The method of claim 1, wherein at least two of the variable-size data blocks are of different sizes.

5. The method of claim 1, wherein logical block addresses can be allocated or released in consecutive logical block addresses that are powers of 2.

6. The method of claim 5, wherein in the linear write step, allocation and release operations are performed on about 1 to about 32768 logical block addresses.

7. The method according to claim 1, further comprising: Perform an atomic update on the at least one variable-size data block.

8. The method of claim 7, wherein the atomic update is performed on the grouping of the variable-size data blocks.

9. The method of claim 8, wherein the grouping of the variable-size data blocks is contiguous.

10. The method of claim 8, wherein at least two of the variable-size data blocks within the group are non-contiguous.

11. The method of claim 8, wherein the plurality of header fields associated with the group include subfields selected from the group consisting of: the number of file items, the number of active groups, the number of assigned groups, permissions, timestamps, the number of extensions, an array list of extension logical block addresses, the number of items in the group, an array list of item lengths within the group, and combinations thereof.

12. A storage device, comprising: Multiple variable-size data blocks, each of which contains new data written linearly; and Multiple logical block addresses, each of which tracks a corresponding variable-size data block stored thereon. This provides a single I / O access performance for any object stored in this way.

13. The storage device of claim 12, wherein at least two of the variable-size data blocks are contiguous on the storage device.

14. The method of claim 12, wherein at least two of the variable-size data blocks are not contiguous on the storage device.

15. The method of claim 12, wherein at least two of the variable-size data blocks are of different sizes.

16. The method of claim 12, wherein logical block addresses can be allocated or released in power-of-2 consecutive logical block addresses.

17. The method of claim 16, wherein in the linear write step, allocation and release operations are performed on about 1 to about 32768 logical block addresses.

18. The method of claim 12, further comprising: Atomic update, wherein the atomic update is performed on the at least one variable-size data block.

19. The method of claim 18, wherein the atomic update is performed on the variable-sized data blocks of a group stored on the storage device.

20. The method of claim 19, wherein the variable-sized data blocks of the group are contiguous.

21. The method of claim 19, wherein at least two of the variable-size data blocks within the group are non-contiguous.

22. The method of claim 19, wherein the plurality of header fields associated with the group include subfields selected from the group consisting of: the number of file items, the number of active groups, the number of assigned groups, permissions, timestamps, the number of extensions, an array list of extension logical block addresses, the number of items in the group, an array list of item lengths within the group, and combinations thereof.

23. A storage system, comprising: At least one storage device; Multiple variable-size data blocks are stored within the at least one storage device, each of the variable-size data blocks having new data linearly written into it; and Within each of the at least one storage device, there are multiple logical block addresses, each logical block address tracking a corresponding variable-size data block stored thereon. This provides a single I / O access performance for any object stored in this way.

Images