Key value repository and file system

Abstract

The embodiment of the invention relates to a key value repository and a file system. A key value repository is integrated with a file system to provide improved operation. The key value repository may include a log engine, a hash engine, a ranking engine, and a garbage collection manager. Features of the key value repository may be configured to reduce the number of I / O operations involving the file system, thereby improving read efficiency, reducing write latency, and reducing write amplification problems inherent in the combined key value repository and file system.

Description

Technical Field

[0001] The present disclosure is directed to a key-value store and a file system, and in particular to an integrated key-value store and file system combination for providing cloud computing services. Background Art

[0002] Typical key-value storage libraries (such as existing database systems) use log-structured merge trees. These trees provide fast data ingestion, but there may be tradeoffs in write amplification, resource contention, relatively poor read performance, and computational overhead. Summary of the invention

[0003] The present disclosure is directed to a key-value store and a file system, and in particular to an integrated key-value store and file system combination for providing cloud computing services.

[0004] By implementing a key-value separation mechanism that reduces write amplification and resource contention, performance during data write operations may be improved, especially for cloud services that process data containing large values ​​(eg, tens of KB).

[0005] Additionally, the key-value store and file system can include specialized engines to meet specific data access patterns. The first engine provides hash table-like read performance, thereby enhancing overall read efficiency. The second engine is designed to process data with temporal locality (such as pre-write logs or raft logs), thereby achieving high-performance data ingestion.

[0006] In addition, the extended tail latency during data writes can be addressed by optimizing the journal compression of the underlying file system. This optimization helps reduce the latency of write operations, thereby improving overall system performance. Improvements in overall system performance can include reduced CPU and storage consumption, increased throughput, and reduced latency.

[0007] In an embodiment, a system including a key-value system, a file system, and a collaboration layer. The key-value system includes a log engine configured to provide parallel writing of multiple log files, and a hash engine configured to process point queries by storing index information in a compact index. The key-value system also includes a sorting engine configured to process range scans by using a partitioned log structure merge (LSM) tree. The file system includes a log-structured append-only write model. The collaboration layer is configured to facilitate collaboration between the key-value system and the file system.

[0008] In an embodiment, the file system is configured to update data based on a key-value system but not update metadata associated with the data.

[0009] In an embodiment, the key-value system is configured to provide data to a file system to make the data persistent within the file system.

[0010] In an embodiment, the hash engine is configured to retain the compact index in the cache memory. In an embodiment, the index information is compressed. In an embodiment, the index information retained in the cache memory is a portion of the index information selected by removing some of the index information according to a least recently used strategy.

[0011] In an embodiment, the journal engine is configured to merge multiple user operations into a single I / O operation to be performed with the file system. In an embodiment, the merging of multiple user operations includes placing the multiple user operations into a queue and flushing the queue to the file system when a predetermined condition occurs.

[0012] In an embodiment, the sorting engine is configured to split data from a first LSM into a plurality of shards, each of the plurality of shards comprising a second LSM tree having fewer layers than the first LSM tree. In an embodiment, the sorting engine comprises a job scheduler configured to prioritize flush operations and level-zero to level-one compression operations. In an embodiment, the sorting engine comprises a shard manager configured to direct the splitting of data such that each of the second LSM trees has a number of layers within a predetermined range. In an embodiment, the sorting engine comprises a timestamp manager configured to cause level-zero or level-one applications to have timestamp values ​​that strictly increase over time for the same key.

[0013] In an embodiment, the key-value system also includes a garbage collection module configured to merge multiple layers of the LSM tree into a last layer of the LSM tree. In an embodiment, the collaboration layer is configured to coordinate garbage collection operations across the key-value system and the file system. In an embodiment, coordinating garbage collection operations includes: the file system exposing region usage information to the key-value system, and the key-value system using the region usage information to determine files to be compressed or garbage collected. In an embodiment, the garbage collection module is configured to determine the size of garbage data within the blob file based on an SST file associated with the blob file. In an embodiment, the SST file contains redundant keys at the end of the SST file.

[0014] In an embodiment, the file system includes a superblock, a journal, and data components, the superblock component including general file system metadata, and the journal including checkpoints of a log of file system operations.

[0015] In an embodiment, the key-value system includes a scheduler configured to schedule background tasks so that each input / output (I / O) request issued by an upper layer application has consistent I / O amplification.

[0016] In an embodiment, the system is configured to handle immediate files and common files, wherein the file system allocates storage space for common files within a predefined range. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A schematic diagram of a key-value storage library and a file system according to an embodiment is shown.

[0018] Figure 2 Modules of a key-value store and a file system according to an embodiment are shown.

[0019] Figure 3 A key-value store and file system that handles multiple I / O streams according to an embodiment is shown.

[0020] Figure 4 A design of a log engine for shared logs according to an embodiment is shown.

[0021] Figure 5 Data management for a shared log according to an embodiment is shown.

[0022] Figure 6 Metadata management according to an embodiment is shown.

[0023] Figure 7 Persisting metadata on disk according to an embodiment is shown.

[0024] Figure 8 A hashing and sorting engine according to an embodiment is shown.

[0025] Fig. 9 A hash engine according to an embodiment is shown.

[0026] Fig.10 A ranking engine according to an embodiment is shown.

[0027] Fig.11 A garbage collection module according to an embodiment is shown.

[0028] Fig. 12A A garbage collection process according to an embodiment is shown.

[0029] Fig. 12B A garbage collection process according to an embodiment is shown.

[0030] Fig.13 A model of a garbage collected storage device according to an embodiment is shown.

[0031] Fig.14 A disk layout of a file system according to an embodiment is shown.

[0032] Fig.15A method of flushing file metadata changes and log records while the disk is running is shown.

[0033] Fig.16 An I / O flow path of a file system according to an embodiment is shown.

[0034] Fig.17 Fault redundancy in a file system according to an embodiment is shown.

[0035] Fig.18 A synchronous I / O operation in a file system according to an embodiment is shown.

[0036] Specific implementation method

[0037] The present disclosure relates to a key-value store and a file system, and in particular to an integrated key-value store and file system combination for providing cloud computing services.

[0038] In the following detailed description, reference is made to the accompanying drawings that form a part of the description. In the accompanying drawings, similar symbols generally represent similar components unless the context dictates otherwise. In addition, unless otherwise stated, the description of the subsequent drawings may refer to the features in any previous drawings to provide a clearer context and substantive explanation of the current example embodiments. However, the example embodiments described in the detailed description, drawings, and claims are not intended to be limiting. Other embodiments may be utilized, and other changes may be made without departing from the spirit or scope of the subject matter presented herein. It will be readily understood that the various aspects of the present disclosure as generally described and narrated herein and shown in the accompanying drawings may be arranged, replaced, combined, separated, and designed in a variety of different configurations, all of which are expressly contemplated herein.

[0039] In addition, various parts of the present disclosure may be described in terms of functional block components and various processing steps. It should be understood that such functional blocks may be implemented by any number of firmware, software, and / or hardware components configured to perform the specified functions.

[0040] Figure 1 A key-value store and file system according to an embodiment is shown. The key-value store and file system 100 includes a key-value system 102. The key-value system 102 includes a log engine 104, a hash engine 106, a sort engine 108, and a garbage collection manager 110. The key-value store and file system 100 also includes a collaboration layer 112 and a file system 114. The key-value store and file system 100 can interact with a kernel space 116, which includes one or more disks 118. The key-value store and file system 100 can interact with an application 120.

[0041] The key-value and file system 100 can be used for storage in cloud applications, for example, to provide data persistence required by cloud services. The key-value system 102 is configured to provide a key-value repository, for example, as part of a storage backend for a cloud service. Non-limiting examples of cloud services that use the key-value system 102 include shopping, social media, metadata management, etc. The file system 114 can be a dedicated user-level append-only file system that is configured to provide storage specifically to facilitate the operation of the key-value system 102.

[0042] The log engine 104 is configured to allow multiple log files to be written simultaneously, thereby reducing the number of compression and garbage collection operations. The logs written by the log engine 104 can be configured so that strong ordering is not required for processing logs. The log engine 104 is configured to improve throughput performance issues in log writing and increase recovery speed by reducing the synchronous write overhead of the log from multiple input / output (I / O) operations to a single I / O operation, using lock-free queues to aggregate writes to control latency and improve throughput, and / or providing asynchronous interfaces to enhance the thread model. In the case where the key-value system 102 and the file system 114 are integrated and collaborated, the log engine 104 can be used to store a pre-written log (WAL) with a predefined structure, which has a defined actual file size. The defined file size for the WAL can in turn result in fewer I / O operations being required, thereby improving performance while mitigating potential trade-offs related to data consistency. The operation of the log engine 104 is Figure 4-Figure 7 are further detailed in and described below.

[0043] The hash engine 106 is configured to process point queries within the key-value system 102. In particular, the hash engine 106 is configured to reduce tail latency in point queries. The hash engine 106 includes a separation of data and index components, and maintaining the index in a cache memory, for example, by compressing the index and / or caching a portion of the data. The portion of data may be selected using, for example, a least recently used strategy. The operation of the hash engine 106 is Figure 8 and Fig. 9 are further detailed in and described below.

[0044] The sorting engine 108 is configured to perform range scan operations while reducing write amplification factors and / or read / write latency associated with such operations. The sorting engine 108 is configured to use partitioned log structure merge (LSM) trees. The sorting engine 108 can also perform classification and task scheduling of I / O streams. The operations of the sorting engine 108 are Figure 8 and Fig.10 are further detailed in and described below.

[0045] The garbage collection manager 110 is configured to perform garbage collection and / or compaction operations in the key-value store and file system 100. The garbage collection manager 110 may be configured to reduce unnecessary data movement during garbage collection and / or compaction operations in the key-value store and file system 100. The garbage collection manager 110 may perform garbage collection and / or compaction operations based on knowledge of application-side data deletions, such as page expiration. The garbage collection and compaction performed by the garbage collection manager 110 may be configured to arrange data to support other modules, such as the sorting engine 108. The garbage collection manager 110 may coordinate the preservation of data during garbage collection and / or compaction operations. The operation of the garbage collection manager 110 is Figure 11-13 are further detailed in and described below.

[0046] The collaboration layer 112 is configured to facilitate collaboration between the key-value system 102 and the file system 114. Based on the collaboration between the key-value system 102 and the file system 114, the collaboration layer 112 can also facilitate efficient compression and / or garbage collection operations in the key-value system 102. The collaboration can reduce write amplification problems caused by compression and / or garbage collection operations. In an embodiment, the collaboration layer 112 can expose region usage information from the key-value system 102 to the file system 114.

[0047] The file system 114 can be configured to split data out of the log and use log-structured append-only writes as the write model. In an embodiment, the file system can also provide pre-allocated data space, where synchronous writes occur only for data persistence, and in an embodiment, metadata persistence is not required. In an embodiment, data persistence and global log persistence for different files can be performed separately. These aspects of the file system can allow the file system to avoid some metadata persistence operations, such as operations caused by a single data write persistence operation.

[0048] The file system 114 can be configured to support general files and instant files. Both general files and instant files can be written sequentially, and can be read sequentially or randomly. General files can be optimized for continuous low latency in sequential or random reads. General files can be used to write data in batches, without flushing the data (such as SST files) to the disk after each write. Storage space is allocated in large units, and a non-limiting example of the unit size is 1MB. Large allocation units can reduce the metadata size of general files so that the metadata of all general files can be stored in the memory during normal file system operations. By storing metadata in the memory, no further I / O is required to access the metadata for the read operation of the general file regardless of the read offset. This can reduce the read tail latency of the general file. Instant files can be optimized for fast, incremental synchronous writes, while having good sequential and random read performance near the tail. Instant files can be used to write data that needs to be frequently flushed to the disk to achieve instant durability, such as pre-written log files of key-value systems. For instant files, the data and metadata written separately each time can be bundled together. Bundled data and metadata can be written to a journal file shared by all instant files. Bundling data and writing it to a journal file can improve the speed of incremental writes and sync operations. This approach is structured to support sequential reads, but there may be tradeoffs in terms of random reads. Since instant files are expected to be mostly read sequentially, and random reads are mainly concentrated near the tail, the most recently written data for each instant file that is actively being written can be cached to improve read performance.

[0049] The file system 114 may include a user space I / O scheduler to assign I / O priorities to different I / O types. The I / O scheduler marks the foreground I / O as a high priority, while the background I / O will be marked as a low priority. In addition, the key value system 102 may include a scheduler for scheduling its background tasks to ensure that each I / O issued by the upper layer application has a consistent I / O amplification. By jointly designing the I / O scheduling in the key value system 102 and the file system 114, since both the I / O amplification and the I / O latency are consistent, the tail latency can remain stable and low. In addition, reading common files from the file system does not require the I / O of metadata, and using a large space for common files can ensure that most read operations only require a single I / O.

[0050] Kernel space 116 may contain disks 118. Disks 118 may include one or more storage media, such as solid-state drives (SSDs). In an embodiment, at least some of disks 118 are zoned storage (ZNS) SSDs.

[0051] The application 120 is any suitable application that utilizes the key-value and file system 100, for example, online shopping, social media, metadata management applications, etc. The application 120 may interface with the key-value and file system 100 through any suitable application programming interface (API). In an embodiment, the API may be specific to a particular type of file, for example, whether the nature of the file is a general file or an instant file is determined by the API that receives the file.

[0052] Figure 2 Modules of a key-value store and file system according to an embodiment are shown. The key-value store and file system 200 includes a log engine 202, a hash engine 204, a sort engine 206, a garbage collection manager 208, and a file system 210.

[0053] The log engine 202 is configured to allow multiple log files to be written simultaneously, thereby reducing the number of compression and garbage collection operations. The logs written by the log engine 202 can be configured so that no strong ordering is required to process the logs. The log engine 202 is configured to improve the throughput performance issues in log writing and improve the recovery speed by reducing the synchronous write overhead of the log from multiple input / output (I / O) operations to a single I / O operation, using lock-free queues to aggregate writes to control latency, and improve throughput, and / or providing asynchronous interfaces to enhance the thread model. The operation of the log engine 202 is Figure 4-Figure 7 are further detailed in and described below.

[0054] The hash engine 204 is configured to process point queries within the key-value store and file storage system 200. In particular, the hash engine 204 is configured to reduce tail latency in point queries. The hash engine 204 includes a separation of data and index components, and maintaining the index in a cache memory, for example, by compressing the index and / or caching partial data. The partial data may be selected using, for example, a least recently used strategy. The operation of the hash engine 204 is Figure 8 and Fig. 9 are further detailed in and described below.

[0055] The sorting engine 206 is configured to perform range scan operations while reducing write amplification factors and / or read / write latency associated with such operations. The sorting engine 206 is configured to use partitioned log structure merge (LSM) trees. I / O flow classification and task scheduling can also be performed by the sorting engine 206. The operations of the sorting engine 206 are as follows: Figure 8 and Fig.10 are further detailed in and described below.

[0056] The garbage collection manager 208 is an active garbage collection manager for the key-value store and file system 200. The garbage collection manager 208 can be configured to reduce unnecessary data movement during garbage collection and / or compaction operations in the key-value store and file system 200. The garbage collection manager 208 can perform garbage collection and / or compaction operations based on knowledge of application-side data deletions (such as page expiration). The garbage collection and compaction performed by the garbage collection manager 208 can be configured to arrange data to support other modules (such as the sorting engine 206). The garbage collection manager 208 can coordinate the preservation of data during garbage collection and / or compaction operations. The operation of the garbage collection manager 208 is Figure 11-13 are further detailed in and described below.

[0057] The file system 210 can be an append-only file system for a key-value storage library. The file system 210 can be configured to split data out of the log and use log-structured append-only writes as the write model. In an embodiment, the file system 210 can also provide pre-allocated data space, where synchronous writes occur only for the persistence of data, and in an embodiment, there is no need to make metadata persistent. In an embodiment, data persistence and global log persistence for different files can be performed separately. These aspects of the file system can allow the file system to avoid some metadata persistence operations, such as those caused by a single data write persistence operation.

[0058] Figure 3 A key-value store and file system that handles multiple I / O streams according to an embodiment is shown. Figure 3 Shown in Figure 1 and Figure 2 Utilization of the modules shown in and further described herein. Figure 3 In the embodiment shown in , the key-value store interfaced with the file system and the page server of the MySQL-compatible OTLP database. The interface with the page server of the MySQL-compatible OTLP database is a non-limiting example of a suitable application for interfacing with the key-value store and the file system according to the embodiment. Figure 3 As shown in , the database page server uses a key-value store and a file system to store each of the base page, MVPage, incremental log, and redo log. The key-value and file system can use a log engine to index and persist the redo log from the database page server. These operations can have a read and write amplification of 1 each. MVPage can be processed using a hash engine. Incremental logs and metadata can be processed using a sorting engine respectively.

[0059] Figure 4The design of a log engine for a shared log according to an embodiment is shown. The log engine is configured to store log data in upper-layer software, for example, in database software. The log engine may provide two interface classes, LogFile and LogFileFactory. The LogFile class is configured to act as a file descriptor, and the LogFileFactory class is configured to act as a file system. The log engine is configured to reduce random writes on the SSD. Reducing random writes to the SSD can reduce tail latency and improve the efficiency of managing file metadata updates in the memory. Operations can be sorted using a unique sequence number to ensure the order on the disk. High reliability is achieved by cyclic redundancy check protection of the log file. The redundancy check protection provided by the log engine can achieve error recovery of the upper-layer software. The log engine can be configured to aggregate log data and implement a lock-free queue. Queuing in the log engine can reduce user thread blocking. In addition, the log engine can save the metadata in the memory to the underlying file system through a background thread when it reaches a certain size, thereby efficiently using the data memory usage. In an embodiment, efficient data memory usage can be represented by data memory usage of about 0.1% or less.

[0060] Figure 4 The log engine shown in FIG. 1 may include a user interface manager configured to receive user requests (such as Figure 4 The log engine may further include an internal layer configured to manage metadata, shared logs, and background tasks. Figure 4 The journaling engine shown in also includes a background layer that is configured to interface with the file system, for example, by writing the journals to a physical journal in order to make the journals persistent. Figure 4 The log engine shown in is configured to create a shared log, which can merge multiple user operations. Non-limiting examples of such user operations include appending, creating, deleting, pruning, and truncating operations. These merged operations can be processed with a single I / O request to the file system, thereby allowing the number of overall I / O requests to be reduced. In particular, the received operation is changed from the queue state to the input queue of the internal layer. The internal layer is configured to exchange the input queue with the output queue under specific conditions (such as, when the background thread is available), and now the output queue can flush all previous contents of the input queue to the file system, thereby making the various requests in the current input queue persist in the file system through a single I / O operation.

[0061] In an embodiment, multiple shared log instances may be used to manage different forms of data, such as user data, metadata, and other files. User data may be log data stored by the log engine. Metadata for improving read operations may also be included in the shared log instance within the log engine. Metadata management may include one or both of an in-memory metadata storage module and a persistent metadata storage on a storage device memory using a standard mapping. The persistent metadata storage may take the form of a format such as a log structured merge (LSM) class. Metadata management within the log engine is described below and is described in detail in the following. Figure 6 is shown in .

[0062] Figure 5 FIG. 2 shows data management of a shared log according to an embodiment. Figure 5 As shown in , the client requests three separate "append" operations. The logging engine receives these operations as it enqueues them. From the enqueuer, these operations are as described above about Figure 4 The received user operations are packaged into log records and wait for the background thread to flush all requests to the disk, making the log updates durable as shown in DataLog, where the corresponding records each include the corresponding request attachments.

[0063] Figure 6 Metadata management according to an embodiment is shown. In particular, Figure 6 The append, create, delete, and prune requests from the client and the operations in memory and disk in response to the requests are shown. Figure 4 and 5 As shown in , metadata management can be performed by the logging engine. Metadata management can include managing the file system's namespace and file mappings, handling storage metadata for each log file, and implementing a mechanism for metadata persistence by writing to disk. The namespace can maintain a mapping of file names to file identifiers (fids), and the file mapping can track fids to persistent metadata instances (such as the above description of Figure 4 Each LSM meta-instance can maintain a mapping of log sequence numbers (LSNs) to log record locations in the underlying file system (e.g., lsn:<fid,offset,lsn> ). An LSM meta-instance can include both level 0 (L0, directly indexed) and level 1 (L1, indirect indexed) portions in memory.

[0064] For the append operation, the corresponding entry is added to the metadata of the corresponding log file. In an embodiment, it is suggested that the size of each such entry is about 24B, thereby accounting for about 3% of the total data of the 1k record size. As the log data grows, the memory occupied by the metadata will grow significantly, for example, for a 1T disk, the metadata is close to 30GB. Therefore, it becomes infeasible to store all log file metadata in the memory. On the contrary, the log engine can regularly write the log file metadata to the disk to make the log file metadata lasting. Making the log file metadata lasting can be an access mode involving frequent writes and infrequent reads, and the possibility of reading the most recently written data is high. In order to deal with this problem, the log engine can maintain the latest metadata entries in the memory, write the older entries to the disk, and use the indirect mapping table in the memory to point to the metadata entries on the disk.

[0065] Figure 7 It is shown that metadata is made persistent on disk according to an embodiment. Making metadata persistent may include log file creation and / or deletion. Making metadata persistent may include modifications to namespace and file mappings, pruning log files, requiring actions to be taken on LSM metadata to achieve efficient metadata management, and / or a "L0Dump" operation, which involves making the L0 map persistent by writing the L0 map to disk and saving the on-disk location of the L0 map in L1.

[0066] Figure 8 A hash and sort engine according to an embodiment is shown. The combination engine 800 includes a hash engine 802 and a sort engine 804. The hash engine 802 includes hash engine shards 806, each shard 806 including a memory repository 808 and a disk repository 810. The sort engine 804 includes sort engine shards 812, each shard 812 including a memory repository 814 and a disk repository 816. The combination engine 800 also includes a write-ahead log (WAL) 818, a memory storage (Memstore) manager 820, and a shard manager 822. The combination engine 800 may also include a background worker 824 to run background tasks of the system including the combination engine 800, such as compression and / or garbage collection.

[0067] The combination engine 800 includes a hash engine 802 for processing point queries and a sorting engine 804 for performing range queries.

[0068] The hash engine 802 is configured to process point queries. In an embodiment, the hash engine 802 is configured to separate index and data components, and is configured to store the index component in a cache memory. The index component can be stored in the cache memory by compression (such as CritBitTrie compression and / or caching partial data (such as data selected using a least recently used (LRU) strategy to identify the data most likely to be queried)) so that the index hit rate in the cache memory can be maximized. The hash engine 802 can also be configured so that, in the event of a cache miss (where the index data associated with the point query is not in the cache memory), the minimum number of disk I / O requests is required to respond to the query. In an embodiment, when a cache miss occurs, the hash engine 802 may only require one disk I / O request to respond to the query.

[0069] The hash engine 802 can be divided into hash engine shards 806, each hash engine shard including a memory repository 808 and a disk repository 810. The memory repository 808 is a cache memory configured to store index data, such as compressed index data or index data selected using an LRU strategy so as to fit within the memory repository 808. The disk repository 810 is a persistent storage device configured to contain blob files and blob file indexes. When the cache hits, the blob file can be referenced, for example, based on the index data from the memory repository 808 to respond to the point query. When the cache misses, the blob file index can be referenced to obtain the required index information by accessing the disk repository 810 with the minimum number of I / O operations.

[0070] The sorting engine 804 is configured to respond to range queries. The sorting engine 804 is configured to reduce read and write latency and write amplification associated with responding to range queries. The sorting engine 804 may include classifying I / O tasks and scheduling them accordingly to reduce read and write latency. The sorting engine 804 may also include splitting the LSM tree of the database to reduce the number of layers of such a tree, thereby reducing write amplification. In an embodiment, splitting the LSM tree may be performed by the shard manager 822. In addition, the sorting engine itself may include sorting engine shards 812. Each sorting engine shard 812 may include a corresponding memory repository 814 and a disk repository 816 for a corresponding partition of the LSM tree and its data. More details on examples of the sorting engine 804 are provided in Fig.10 is provided in and described below.

[0071] A write-ahead log (WAL) 818 may provide a log where data write operations may be performed in a secure storage device before permanent changes are made to the database by the data write operation. The WAL 818 may provide greater security and durability, thereby improving recovery capabilities in the event of a crash.

[0072] The memory storage manager 820 is a memory management module for one or both of the hash engine 802 and the sort engine 812. The memory storage manager 820 may include any suitable hardware, software, and combination thereof for performing the memory management described herein. The memory storage manager 820 may include control of the cache of the hash engine 802,

[0073] The shard manager 822 controls the sharding of the hash engine 802 and the sorting engine 804 into corresponding shards 806, 812. The shard manager 822 can be configured to organize the corresponding shards 806, 812 to provide data isolation. Data isolation can reduce unnecessary replication during garbage collection. In addition, the shard manager 822 can organize the shards 806, 812 so that the shards have their own index management. By partitioning the shards 806, 812 to simplify index management, the shard manager can thereby reduce index overhead and reduce the computing resources required to accommodate index overhead. The shard manager 822 can also control the sharding of the hash engine 802 and / or the sorting engine 804 to control resource requirements in order to support multi-tenant operations. The shard manager 822 can also perform segmentation of the LSM tree of the database to reduce write amplification for the sorting engine 804.

[0074] Fig. 9 The index structure of a hash engine shard according to an embodiment is shown. The cache memory 900 contains a first level compact index 902 and one or more second level compact indexes 904. The storage memory 906 contains blob files 908, each blob file having a blob file digest 910.

[0075] The cache memory 900 is a cache memory separated from the storage device memory 906. The cache memory 900 may include one or more memory devices suitable for cache operations, such as dynamic random access memory (DRAM), persistent memory (PMEM), one or more SSDs (such as NVMe SSD), etc. The cache memory is configured to store a compact index. In an embodiment, the compact index can be accessed without accessing the storage device memory 906. The compact index may contain a key value obtained by separating the key value from the data component in the blob file 908. In an embodiment, the key value separation mechanism utilizes the LSM tree structure, thereby providing advantages such as block-level disk indexes and compressed data blocks to reduce index overhead. In an embodiment, the compact index can identify data in response to a point query without accessing the storage device memory 906. The compact index stored in the cache memory 900 may include a first-level compact index 902 and a second-level compact index 904.

[0076] The first level compact index 902 maps keys to blob file numbers (blob_file_no) of blob files 908. The first level compact index can be stored in any suitable file, for example, residing in an SST file. Each SST file can have an associated SST file summary, which can contain the first level compact index 902. The first level compact index 902 can be organized according to an LSM tree structure, thereby receiving the results of key-value separation in the form of an LSM tree and storing them accordingly.

[0077] The second level compact index 904 maps keys to block offsets (block_offset) within the blob file 908. Each blob file 908 may have a corresponding blob file digest 910 that contains offsets referenced by the corresponding second level compact index 904. For the second level compact index 904, a blob file digest 910 may be generated for each corresponding blob file 908 to store the offsets corresponding to each second level compact index 904.

[0078] The storage device memory 906 is a memory configured to store data of the database, such as a blob file 908. The storage device memory 906 can be any suitable storage device memory, such as one or more PMEMs, SSDs, hard disk drives (HDDs), combinations thereof, etc. The blob file 908 is a binary large object, such as a chunk of binary data, for example, a binary data encoding large object, such as video, audio, images, and combinations thereof, as non-limiting examples. The blob file may include a blob file digest 910, which contains data referenced by the second level compact index, so that the blob file digest 910 can identify a valid data block according to an offset provided in the second level compact index 904.

[0079] Fig.10 A sorting engine according to an embodiment is shown. The sorting engine 1000 is configured to organize and manage data to support range scans and reduce write amplification and latency of such processes. The sorting engine 1000 includes sharding 1002, a job scheduler 1004, I / O classification 1006, co-designed compression and garbage collection 1008, compression 1010, and garbage collection 1012. The sorting engine 1000 may also include a memory storage manager 1014, a sharding manager 1016, fault tolerance 1018, an asynchronous API 1020, a multi-tenant module 1022, and a timestamp manager 1024. An input / output (I / O) layer 1026 may communicate with a storage device 1028.

[0080] The sorting engine 1000 is configured to generate shards 1002 to split data from a large LSM tree structure into smaller LSM tree structures within the shards 1002. Sharding the LSM tree structure reduces the number of layers of each tree, thereby reducing write amplification. Splitting the LSM tree into shards 1002 can be performed by a shard manager 1016.

[0081] Job scheduler 1004 refers to such a component or module, which is programmed, designed, or otherwise configured to collaborate to ensure stable latency of individual I / O operations to and from storage devices.Job scheduler 1004 can be configured to reduce or avoid adverse conditions (such as write pauses or write stops) to enhance or even optimize data retrieval and reduce tail latency.Job scheduler 1004 can be configured to distinguish different background tasks based on their priorities and allocate different I / O bandwidth resources to them.In an embodiment, background tasks can be classified into three different priority types.In this embodiment, job scheduler 1004 can give priority to ensuring fast flushing, which means that flushing operations have the highest priority.Fast flushing clears enough space in the write buffer of the memory, thereby accommodating foreground write requests in a timely manner.Flushing speed directly affects the write tail latency.In this embodiment, L0 to L1 compression tasks can have a second priority.If L0 to L1 compression speed is slow, it directly increases the number of read I / O operations, thereby extending the read tail latency. In this embodiment, the priority of L1 to L(N) compression can be set to the lowest priority. These compression tasks are mainly used to maintain the form of the LSM tree and will not have a significant impact on the read and write latency in the short term.

[0082] Prioritizing the flushing process can reduce write stalls or write stops, because the main cause of these problems is that the write buffer is full. Therefore, maintaining the flushing speed can reduce write stalls and write stops.

[0083] In the sorting engine 1000, read I / O amplification may be affected by the number of L0 files. An increase in the number of L0 files may occur due to a slow L0 to L1 compression, which may be caused by, for example, queuing delays of the L0 to L1 compression while a higher level compression is ongoing and / or a concurrent higher level compression taking up the I / O bandwidth of the L0 to L1 compression. Therefore, the job scheduler 1004 may prioritize the L0 to L1 compression over other compressions.

[0084] I / O classification 1006 can implement priority classification and scheduling methods, including collecting performance indicators (such as request latency and disk I / O throughput) to analyze patterns of different operations. By identifying patterns such as sequential or random access, appropriate priorities can be assigned to I / O operations, such as three levels of "high", "medium" and "low". In an embodiment, an adaptive mechanism can continuously adjust priorities based on real-time observations and historical impacts of priority assignments. Machine learning techniques can improve prediction accuracy and priority adjustment. The thread pool allocation system can dynamically assign threads to different priorities, thereby ensuring timely execution of high priority tasks. In an embodiment, manual user overrides can also be provided to handle exceptions.

[0085] Additionally, the prioritization functionality can utilize requests and collect performance metrics, and can assign appropriate priorities to requests based on the patterns identified. The actual implementation of collecting performance metrics and identifying patterns includes measuring and analyzing various metrics to make informed prioritization decisions. Additionally, to enhance pattern detection and priority assignment, machine learning techniques can be employed, including clustering, anomaly detection, time series analysis, decision trees, neural networks, gradient boosting, principal component analysis, reinforcement learning, and self-organizing maps. Together, these methods provide a versatile tool for effective adaptive behavior identification and priority assignment.

[0086] Co-designed compression and garbage collection 1008 can cooperate with the file system to reduce end-to-end write amplification by performing compression and garbage collection in coordination with garbage collection in the file system. Co-designed compression and garbage collection 1008 can operate, for example, based on region usage information from the file system.

[0087] Compression 1010 is configured to perform compression tasks within sorting engine 1000. Garbage collection 1012 is configured to perform garbage collection tasks within sorting engine 1000. Compression 1010 and garbage collection 1012 may be controlled to operate based on outputs of job scheduler 1004, I / O classification 1006, and / or co-designed compression and garbage collection 1008.

[0088] The memory storage manager 1014 is a module for memory management. The memory storage manager 110 may include any suitable hardware, software, and combination thereof for performing memory management, for example, managing memory when the slices 1002 are generated according to the slice manager 1016 .

[0089] The shard manager 1016 is a component or module that is programmed, designed, or otherwise configured to automatically or manually split data in a large LSM tree structure into smaller LSM tree structures (such as shards 1002). The sharding performed by the shard manager 1016 thereby reduces the total number of layers in the LSM tree, thereby reducing write amplification. When the data within the LSM tree structure reaches a certain threshold, the shard manager 1016 can split the LSM tree into shards 1002, or split a large shard 1002 into smaller separate shards 1002. Conversely, when the data within the data structure is reduced due to deletion and its data volume is below a specific threshold, the shard manager can, for example, merge adjacent shards to form a single larger shard when the corresponding data volume of the shards to be merged is below a predetermined threshold.

[0090] The shard manager 1016 can be configured to ensure the atomicity of the split operation so that the split operation succeeds completely or fails completely without splitting any data in the corresponding shard when other data fails. Then, any failed operation is rolled back to maintain data integrity. The shard manager 1016 can be configured to operate so as to reduce the impact on front-end I / O operations, for example by keeping the total number of LSM levels within a fixed range. As described above, the split LSM advantageously reduces read-write amplification relative to conventional LSMs because each split LSM itself is an independent LSM tree, so only smaller data volumes with fewer management layers need to be managed, thereby reducing pre-read and write files. The split LSM provided by the shard manager 1016 can also allow more aggressive hierarchical compression to reduce WAF, promote tenant isolation and adaptive policies, making it easier to implement adaptive policies (such as, different compression policies for various splits) and control tenant isolation, thereby achieving better trade-offs for workloads with locality characteristics and reducing write amplification without compromising read performance.

[0091] Fault tolerance 1018 is a component or module that is programmed, designed, or otherwise configured to provide sector-level fault tolerance so that damage to a single sector within a file does not affect data consistency and visibility. In order to prevent damage to individual sectors on the disk from causing specific files or data to be unreadable, thereby causing the entire database data of the upper-layer distributed system to need to be rebuilt, data redundancy blocks for key file data can be generated to ensure that the data of the file can be correctly restored even if several consecutive sectors within the file are damaged. Alternatively or additionally, the file system can provide redundant protection for metadata to prevent the unavailability of metadata from causing the entire file system to be unreadable.

[0092] The asynchronous API 1020 is a component or module that is programmed, designed, or otherwise configured to manage I / O operations to and from the storage device so that I / O waits do not block upper-layer threads from performing other tasks. The asynchronous API 1020 can collaborate with the underlying file system to alleviate the blocking of I / O operations, thereby improving parallelism, latency, and response time by collaboratively selecting executors for executing asynchronous tasks and asynchronous task callbacks.

[0093] The multi-tenant module 1022 is a component or module that is programmed, designed, or otherwise configured to provide shard-level resource restrictions and isolation, because the upper layer application (e.g., L0 or L1) may have different resource usage restrictions for different shards. Non-limiting examples of such resources include I / O bandwidth, memory size, and number of threads (e.g., the number of asynchronous I / O execution threads / compression threads), because the upper layer application is configured to have resource usage caps for various engines and shards. In addition, the multi-tenant manager 140 is programmed, designed, or otherwise configured to provide regular monitoring and result statistics on the use of each resource type, thereby allowing the upper layer to dynamically adjust the quota value of each resource type on different shards based on its resource monitoring status, thereby promoting multi-tenant functionality. The multi-tenant module 1022 can be designed or otherwise configured to monitor the real-time resource usage status of each shard, such as allowing the upper layer application to dynamically adjust the quota value of each shard accordingly, thereby maximizing resource utilization.

[0094] The timestamp manager 1024 is a component or module that is programmed, designed, or otherwise configured to automatically or manually timestamp each key-value pair. These timestamps can be used to implement multi-version concurrency control (MVCC) related features. The timestamp manager 1024 can be configured so that the upper layer application (e.g., L0 or L1) has a timestamp value that strictly increases over time for the same key, thereby ensuring the expected behavior during the read process. The user timestamp can be combined with the user key and stored as a unified key in the sorting engine 1000. During encoding, the timestamp can be used to ensure that the internal keys in the sorting engine 1000 are sorted first. The functions of the timestamp manager 1024 can be supported by read, write, and delete functions. For example, timestamps can be associated with various operations. Garbage collection and / or compression processes can remove expired timestamp data to promote data integrity, thereby triggering active compression when capacity limits are reached, thereby reclaiming storage space. The timestamp manager 1024 as described and illustrated herein significantly enhances robustness and efficiency, provides comprehensive support for timestamp management, ensures data integrity, and optimizes storage utilization.

[0095] Fig.11A garbage collection module according to an embodiment is shown. Fig.11 The garbage collection module is configured to reduce unnecessary data movement during garbage collection and compaction. In particular, the garbage collection module can provide periodic compaction to clean up expired pages of the application. Periodic garbage collection can include merging multiple layers of the same page from the LSM tree into a final layer, and the upper layers can perform corresponding functions during the compaction process. This can reduce the amount of data processed during garbage collection. By basing the garbage rate on the entry level rather than the space level, garbage collection can also be scheduled more accurately and more precisely. This can be achieved by selecting the blob file with the highest garbage space for garbage collection. By performing this calculation at the entry level, it can be ensured that the blob file with the highest garbage space is selected, thereby achieving more efficient GC operation. The garbage collection strategy can be further improved, such as making the strategy consider a wider range of scenarios, thereby improving concurrency during high watermarks and improving space recovery rates.

[0096] In an embodiment, garbage collection can be configured to reduce associated read amplification by periodic compression to reduce unnecessary data movement of application data that has expired but not identified by the KV side page during garbage collection. The amount of application garbage data within the SST can be determined by providing an API that allows the application to handle the specific decision-making process for determining the amount of application garbage data. For example, the amount of garbage data can be calculated using table attribute information from the data. At a specific point in time, if the recyclable log sequence number (LSN) of the segment exceeds a threshold, it indicates that all data within the SST segment is considered garbage. By comparing the segment IDs and aggregating the results, the exact size of the garbage data in the entire SST can be determined. It is important to note that the size of the garbage data mentioned here does not refer to the size of the garbage data within the SST itself, but to the size within the corresponding Blob, which accurately reflects the total amount of garbage data. This can be data that is subsequently subject to garbage collection operations.

[0097] Fig. 12A and Fig. 12B A garbage collection process according to an embodiment is shown. Adaptive garbage collection may include, for example, determining capacity usage and / or available capacity by engaging with a file system. This may allow the size of all files in the current directory of the instance to be scanned so that we can understand capacity usage. The adaptive garbage collection process may include adjusting garbage collection thresholds, bandwidth, and concurrency based on available capacity. For example, based on capacity conditions, the garbage ratio threshold and the number of GC threads may be adjusted according to a pre-designed strategy. Fig. 12B Shows Fig. 12A A garbage collection process in which there are multiple garbage collection threads that are assigned to resources and operated based on the respective utilization levels of these resources compared to a threshold.

[0098] Fig.13 A model of a storage device for garbage collection according to an embodiment is shown. Fig.13 In the model of the storage device, the complete redundant key set 1302 is stored at the end of the SST file 1300. The presence of the redundant key set 1302 eliminates the need to scan the entire SST file 1300 during the reverse check of garbage collection, which requires checking the validity of all keys, otherwise the entire SST needs to be scanned from beginning to end. During garbage collection, only the redundant key set 1302 at the end of the SST file 1300 needs to be read. For invalid keys, the corresponding values ​​can be skipped, and for valid keys, the corresponding values ​​can be retrieved. Skipping invalid keys can reduce read amplification by enabling only valid values ​​to be retrieved. Fig.13 In SST, each key-value pair occupies a dedicated data block, and the key information stored in the index block is a complete representation of the key in the data block. Therefore, the index block contains comprehensive key information for the SST.

[0099] A file system according to an embodiment may be configured to split data out of a log and use append-only writes of a log structure as a write model. In an embodiment, the file system may also provide pre-allocated data space, where synchronous writes occur only for data persistence, and in an embodiment, metadata persistence is not required. In an embodiment, data persistence and global log persistence for different files may be performed separately. These aspects of the file system may allow the file system to avoid some metadata persistence operations, such as those caused by a single data write persistence operation.

[0100] The file system according to the embodiment supports general files and instant files. Both general files and instant files can be written sequentially, and can be read sequentially or randomly. General files can be optimized for continuous low latency in sequential or random reads. General files can be used to write data in batches, and there is no need to flush data (such as SST files) to the disk after each write. Storage space is allocated in large units, and a non-limiting example of unit size is 1MB. Large allocation units can reduce the metadata size of general files so that the metadata of all general files can be stored in the memory during normal file system operations. By storing metadata in the memory, no matter how the read offset is, the read operation of the general file does not require further I / O to access the metadata. This can reduce the read tail latency of the general file. Instant files can be optimized for fast, incremental synchronous writes, while having good sequential and random read performance near the tail. Instant files can be used to write data that needs to be frequently flushed to the disk to achieve instant durability, such as pre-written log files of key-value systems. For instant files, the data and metadata written separately each time can be bundled together. The bundled data and metadata can be written to a journal file shared by all instant files. Bundling data and writing it to a journal file can speed up incremental writes and sync operations. This approach is structured to support sequential reads, but there may be tradeoffs in terms of random reads. Since instant files are expected to be mostly read sequentially, with random reads concentrated near the tail, the most recently written data for each instant file that is actively being written can be cached to improve read performance.

[0101] Fig.14 A disk layout of a file system according to an embodiment is shown. Fig.14 Can be used as any file system described herein (such as those described above and in Figure 1 and Figure 214 and 210). File system 1400 includes three different types of data: super block 1402, journal 1404, and data 1406. Super block 1402 includes necessary global file system metadata, including journal index node (inode) content, universal unique identifier (UUID) data, version information, block size, and other related parameters. In an embodiment, it is strategically located at the second 4KB of the storage device, where the strategic location of super block 1402 is selected to improve accessibility and coordination with other data components. Journal 1404 is data that captures the latest checkpoint from the operation log of the file system. Journal 1404 can be effectively replayed during the installation process of the file system to improve the recovery process of file metadata in the memory. Finally, data 1406 is dedicated to storing the actual content of each file in the file system. In an embodiment, each data block is organized as a continuous range of logical block addresses (LBAs).

[0102] Fig.15A method of flushing file metadata changes and log records while the disk is running is shown. Method 1500 includes updating data in memory 1502, synchronizing data to a device 1504, and synchronizing metadata to a device 1506. Updating the data in memory at 1502 includes appending data to an internal write buffer of the file using an operation. When appending data, the system checks whether the length of the write buffer has reached a specified threshold, a non-limiting example of such a threshold is 512KB. If the threshold is reached, the data is flushed to the storage device. By synchronizing the data to the device at 1504, the data can be flushed to the storage device. Synchronizing the data to the device at 1504 can include performing a flush operation, wherein the data in the write buffer is written to the storage device. In addition, synchronizing the data to the device at 1504 can include marking the file and adding the file to a global marked file list to indicate that the file has pending changes. In addition, the metadata can be synchronized to the device at 1506. Synchronizing the metadata to the device at 1506 can be performed by calling a function for ensuring data persistence. This function can force the internal write buffer of the file to be written to the storage device to ensure that the data is stored safely. The function can be an "fsync()" function. In an embodiment, all the marked records marked during the synchronization of data to the device at 1504 can be synchronized to the diary file on the device after the last synchronization of log data. This step helps to maintain the durability and consistency of file metadata changes in various computing environments. In addition, when the metadata is synchronized to the device at 1506, the transaction record for the marked file can be updated. The transaction record can be formatted as described in the previous section and then flushed to the diary file. This process can ensure the durability and consistency of file metadata changes, thereby enhancing the reliability and performance of the key-value storage library and the file system in various computing environments.

[0103] Fig.16The I / O flow path of the file system according to the embodiment is shown. The file system may include a user space I / O scheduler that assigns I / O priorities to different I / O types. The I / O scheduler may mark the foreground I / O as a high priority, while the background I / O may be marked as a low priority. In addition, the key value system includes a scheduler to schedule its background tasks to ensure that each I / O issued by the upper layer application has a consistent I / O amplification. Through this common design of I / O scheduling in the key value system and the file system, since both the I / O amplification and the I / O latency are consistent, the tail latency can be kept stable and low. In addition, reading common files from the file system does not require the I / O of metadata 1606, and the controlled size of the common files can ensure that most reads require a single I / O operation. The file system can use the "fdatasync" operation to update data without updating metadata. Unless a new controlled size chunk needs to be created for the common file, user data can be updated without synchronizing the file size. This optimization reduces unnecessary file metadata synchronization, thereby improving performance, enhancing data consistency, and reducing the overall write amplification factor in the log engine by 1 without affecting the synchronization task. The file system can use specialized file types to support efficient synchronous I / O. The file system can store the data of the log engine in a specialized file type on the disk, allowing compression tasks to be performed. The file system can coordinate garbage collection with the key-value system, for example, by exposing area usage information to the key-value system, which can use the information to decide which file to run compression or garbage collection on. The key-value system can be configured to set the priority of the file system's I / O scheduler. For example, the key-value system can mark the foreground I / O as a high priority, while the background I / O will be marked as a low priority for the file system. In addition, the key-value system can schedule its background tasks so that each I / O issued by the upper-layer application has a consistent I / O amplification. Through this joint design of the key-value system and the file system, since both I / O amplification and I / O latency are consistent, the tail latency can remain stable and low. Furthermore, reading the generic file 1600 does not require I / O of metadata, and the defined sizes (such as 1MB each) ensure that most reads require only a single I / O.

[0104] Fig.17 Fault redundancy in a file system according to an embodiment is shown. Fig.17 The interaction between the data storage system, the file system, and the key-value store is shown. Fig.17As shown in , the file system is configured to provide sector-level redundancy for file system metadata. The structure of the file system allows data to be recovered from the data storage system within one or two sector failures. In an embodiment, the key-value system is configured to provide data to the file system to implement sector-level redundancy for file data of the key-value storage library. Sector-level redundancy can reduce the impact radius of the failure, thereby reducing its impact.

[0105] Fig.18 A synchronous I / O operation in a file system according to an embodiment is shown. Although the use of ordinary file synchronization can reduce latency, the operation includes two write operations, one for updating metadata and the other for updating data. The file system and key-value system according to the embodiment can use file data synchronization to update data without updating metadata. By reducing unnecessary file metadata synchronization, the use of file data synchronization operations can improve performance, enhance data consistency, and reduce the overall write amplification factor (WAF) by 1, while successfully performing necessary synchronization tasks. The key-value system can save data from the logging engine in a file type specifically designed for synchronous I / O efficiency. The use of these specialized files can also authorize key-value storage repositories for non-blocking compression tasks, such as Fig.18 The file system and key-value system can also collaborate to reduce write amplification, for example by exposing region usage information to the key-value system through the file system. The key-value system can use the region usage information to make decisions about compaction and / or garbage collection operations.

[0106] All aspects:

[0107] Aspect 1. A system comprising a key-value system, a file system, and a collaboration layer, wherein:

[0108] The key-value system includes:

[0109] A logging engine that is configured to provide parallel writing of multiple log files,

[0110] a hash engine configured to process point queries by storing index information in a compact index, and

[0111] A sorting engine configured to handle range scans by using a partitioned log structured merge (LSM) tree;

[0112] The file system includes a log-structured append-only write model; and

[0113] The collaboration layer is configured to facilitate collaboration between the key-value system and the file system.

[0114] Aspect 2. The system according to aspect 1, wherein the file system is configured to update data based on the key-value system but not update metadata associated with the data.

[0115] Aspect 3. A system according to any one of aspects 1-2, wherein the key-value system is configured to provide data to the file system so that the data is persistent within the file system.

[0116] Aspect 4. A system according to any one of Aspects 1-3, wherein the hash engine is configured to retain the compact index in a cache memory.

[0117] Aspect 5. The system according to Aspect 4, wherein the index information is compressed.

[0118] Aspect 6. The system according to any one of Aspects 4-5, wherein the index information retained in the cache memory is partial index information selected by omitting some of the index information according to a least recently used policy.

[0119] Aspect 7. A system according to any one of Aspects 1-6, wherein the journaling engine is configured to merge multiple user operations into a single I / O operation to be executed with the file system.

[0120] Aspect 8. The system according to aspect 7, wherein the merging of multiple user operations comprises placing the multiple user operations into a queue and flushing the queue to the file system when a predetermined condition occurs.

[0121] Aspect 9. A system according to any one of Aspects 1-8, wherein the sorting engine is configured to split data from the first LSM tree into multiple shards, each of the multiple shards containing a second LSM tree having fewer layers than the first log-structured merge tree.

[0122] Aspect 10. The system according to Aspect 9, wherein the sequencing engine comprises a job scheduler configured to prioritize flushing operations and zero-to-one compression operations.

[0123] Aspect 11. A system according to any one of Aspects 9-10, wherein the sorting engine includes a shard manager configured to direct the splitting of data so that each LSM tree in the second LSM tree has a number of layers within a predetermined range.

[0124] Aspect 12. A system according to any one of aspects 9-11, wherein the sorting engine includes a timestamp manager configured to enable zero-level or first-level applications to have timestamp values ​​that strictly increase over time for the same key.

[0125] Aspect 13. A system according to any one of Aspects 1-12, wherein the key-value system further comprises a garbage collection module configured to merge multiple layers of the LSM tree into a last layer of the LSM tree.

[0126] Aspect 14. The system according to Aspect 13, wherein the collaboration layer is configured to coordinate garbage collection operations across the key-value system and the file system.

[0127] Aspect 15. The system according to aspect 14, wherein coordinating the garbage collection operation includes the file system exposing region usage information to the key-value system, and the key-value system using the region usage information to determine files to be compressed or garbage collected.

[0128] Aspect 16. The system according to any one of aspects 14-15, wherein the garbage collection module is configured to determine the size of the garbage data within the blob file based on an SST file associated with the blob file.

[0129] Aspect 17. A system according to Aspect 16, wherein the SST file contains a redundant key at the end of the SST file.

[0130] Aspect 18. The system of claim 1, wherein the file system comprises a superblock, a journal, and a data component, the superblock component comprising general file system metadata, and the journal comprising a checkpoint of a file system operation log.

[0131] Aspect 19. A system according to any one of Aspects 1-18, wherein the key-value system includes a scheduler configured to schedule background tasks so that each input / output (I / O) request issued by an upper-layer application has consistent I / O amplification.

[0132] Aspect 20. A system according to any one of aspects 1-19, wherein the system is configured to process instant files and common files, wherein the file system allocates storage space for the common files within a predefined range.

[0133] The examples disclosed in this application should be considered in all respects as illustrative and not restrictive. The scope of the invention is indicated by the appended claims rather than the foregoing description; and all changes within the meaning and equivalent range of the claims are intended to be embraced therein.

Claims

1. A storage system, comprising a key-value system, a file system and a collaboration layer, wherein: The key-value system includes: a logging engine configured to perform parallel writing of multiple log files, a hash engine configured to process point queries by storing index information in a compact index, and a sorting engine configured to process range scans by using a partitioned log structured merge (LSM) tree; The file system includes a log-structured append-only write model; and The collaboration layer is configured to facilitate collaboration between the key-value system and the file system. 2 . The system of claim 1 , wherein the file system is configured to update data based on the key-value system without updating metadata associated with the data. 3 . The system of claim 1 , wherein the key-value system is configured to provide data to the file system to make the data persistent within the file system.

4. The system of claim 1, wherein the hash engine is configured to retain the compact index in a cache memory. 5 . The system according to claim 4 , wherein the index information retained in the cache memory is partial index information selected by omitting some of the index information according to a least recently used policy.

6. The system of claim 1, wherein the journaling engine is configured to merge multiple user operations into a single I / O operation to be performed with the file system.

7. The system of claim 1 , wherein the sorting engine is configured to split data from a first LSM tree into a plurality of shards, each of the plurality of shards containing a second LSM tree having fewer layers than the first log-structured merge tree.

8. The system of claim 7, wherein the sequencing engine comprises a job scheduler configured to prioritize flush operations and level zero to level one compression operations.

9. The system of claim 8, wherein the sorting engine comprises a shard manager configured to direct the splitting of the data so that each of the second LSM trees has a number of layers within a predetermined range.

10. The system of claim 1, wherein the key-value system further comprises a garbage collection module configured to merge multiple layers of the LSM tree into a last layer of the LSM tree. 11 . The system of claim 10 , wherein the garbage collection module is configured to determine a size of garbage data within a blob file based on an SST file associated with the blob file.

Images