Key value storage engine for range scan ordered queries

Abstract

The embodiment of the invention relates to a key value storage engine for range scan ordered queries. By segmenting data within a large LSM tree structure into smaller tree structures to reduce the number of layers in such structure, a write amplification factor (WAF) is effectively reduced. By further classifying and tagging each I / O based on type, lower level file systems can prioritize schedules between different types of I / Os, thereby facilitating stable delays of individual connections and individual I / O operations within the file system layer.

Description

Technical Field

[0001] Embodiments described herein generally relate to facilitating efficient storage and retrieval of data in solid-state devices (SSDs). Background Art

[0002] Key-value (KV) storage engines render SSDs more efficiently than existing block and object storage systems. However, write amplification factor (WAF) and tail latency can negatively impact the performance of KV storage engines and system efficiency. WAF is calculated as a measurement of how much data is written to the SSD compared to the amount of data requested by the host system. In other words, WAF can be expressed as the ratio of writes submitted to the SSD to writes from the host system. Summary of the invention

[0003] In an example embodiment, a method for optimizing a non-volatile storage device includes partitioning data in a first log structure merge (LSM) tree structure into partition fragments (shards) to reduce the number of levels for data represented in the first LSM tree structure. Each partition fragment represents an independent LSM tree structure, thereby providing scalability and flexibility for the data represented in the first LSM tree structure. The method also includes: when the amount of data of the corresponding partition fragment in the partition fragments reaches a threshold level, splitting the corresponding partition fragment in the partition fragments into at least a parent fragment and a child fragment, or when the amount of data of the corresponding partition fragment in the partition fragments is reduced to an amount less than the threshold level, merging the corresponding partition fragment in the partition fragments into adjacent partition fragments in the partition fragments.

[0004] According to at least one other example embodiment, a non-volatile storage device has an executable component stored thereon, the non-volatile storage device including a shard manager configured to split data in a first log structure merge (LSM) tree structure into partition fragments to reduce the number of layers for data represented in the first LSM tree structure. Each partition fragment represents an independent LSM tree structure, thereby providing scalability and flexibility for the data represented in the first LSM tree structure. The shard manager is also configured to: when the amount of data of the corresponding partition fragment in the partition fragments reaches a threshold level, split the corresponding partition fragment in the partition fragments into at least a parent fragment and a child fragment, or when the amount of data of the corresponding partition fragment in the partition fragments is reduced to an amount less than the threshold level, merge the corresponding partition fragment in the partition fragments into adjacent partition fragments in the partition fragments. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] In the following detailed description, the embodiments of the non-volatile storage device and operation for facilitating weighted memory scanning are therefore described only as illustrations, as various changes and modifications will be apparent to those skilled in the art from the following detailed description. The use of the same reference numerals in different drawings indicates similar or identical items.

[0006] Figure 1 shows an example architecture of a key-value store and file system according to at least some embodiments described and enumerated herein;

[0007] Figure 2A A non-limiting example of a ranking engine is shown, and Figure 2B shows a more detailed view of timestamp management implemented by a ranking engine, the timestamp management being arranged in accordance with at least some of the embodiments described and enumerated herein;

[0008] Figure 3A to Figure 3D 3A to 3C illustrate a non-limiting example of database engine management, all in accordance with at least some of the embodiments described and enumerated herein, and Figure 3D is an illustration of a consistent cross-shard range scan according to a non-limiting example embodiment of data segmentation as described and enumerated herein;

[0009] 4A-4C illustrate non-limiting examples of database engine management, and FIGS. 4A-4C illustrate data merging phases, all in accordance with at least some embodiments described and enumerated herein;

[0010] Figure 5A shows a non-limiting example storage process flow in accordance with at least some embodiments described and enumerated herein; Figure 5B shows an example implementation of I / O job scheduling, and Figure 5C shows a more detailed view thereof; and Figure 5D shows a more detailed view of the I / O classification; and

[0011] Figure 6 An illustrative computing embodiment is shown in which any of the processes and sub-processes of training a unified transformer-based VPR training framework may be implemented as executable instructions stored on a non-transitory computer-readable medium. DETAILED DESCRIPTION

[0012] In the following detailed description, reference is made to the drawings that form a part of the specification. In the drawings, similar symbols generally represent similar components unless the context dictates otherwise. In addition, unless otherwise specified, the description of the consecutive drawings may refer to the features from any previous drawings to provide a clearer context and substantive explanation of the current example embodiment. Nevertheless, the example embodiments described in the detailed description, drawings, and claims are not intended to be restrictive. Other embodiments may be utilized and other changes may be made without departing from the spirit or scope of the subject matter presented herein. It will be readily understood that, as generally described and narrated herein and as shown in the drawings, the various aspects of the present disclosure may be arranged, replaced, combined, separated, and designed in a variety of different configurations, all of which are expressly contemplated herein.

[0013] In addition, various parts of the present disclosure may be described herein in terms of functional block components and various processing steps. It should be understood that such functional blocks may be implemented by any number of firmware, software, and / or hardware components configured to perform the specified functions.

[0014] Figure 1 An example architecture of a key-value store and file system is shown in accordance with at least some embodiments described and enumerated herein.

[0015] As described herein, a "key-value engine" may refer to a network, an algorithm, a program, software, hardware, firmware, or any combination thereof that is configured to or otherwise provides a key-value store to meet the persistence requirements of a cloud service provider. The key-value engine may be based on a log structure merge (LSM) tree architecture, in which key values ​​are separated to reduce write amplification and / or resource contention. The key-value engine may be configured to store data in an in-memory key-value store (KVSe). In some embodiments, the LSM tree may be used to achieve high-speed writes by buffering all updates in an in-memory structure (e.g., MemTable). When the MemTable is full, the data may be flushed to a permanent storage device in an ordered string table (SST), where the SST is immutable. In some embodiments, in order to optimize read operations and reduce space usage, compression may be used to merge several SSTs into one to reduce overlap, for example, due to overlapping key ranges in multiple SSTs.

[0016] The key-value store and file system 100 includes a key-value engine 102. The key-value engine 102 may include one or more of a logging engine 104, a hashing engine 106, a sorting engine 108, and a garbage collection manager module 110. The key-value store and file system 100 may also include a collaboration layer 112 and a file system 114. The key-value store and file system 100 may interact with a kernel space 116, which includes one or more disks 118. The key-value store and file system 100 may also interact with an application 120.

[0017] The key-value store and file system 100 can be used for data storage in cloud applications, for example, to provide data persistence required by cloud services. The key-value engine 102 can be configured to provide key-value storage, for example, as part of the storage backend of a cloud service. Non-limiting examples of cloud services using the key-value engine 102 include Internet-based shopping, social media, metadata management, etc. The file system 114 can be a dedicated user-level append-only file system that is configured to provide storage dedicated to facilitating the operation of the key-value engine 102.

[0018] The journal engine 104 can be configured to allow concurrent writes of multiple log files, thereby reducing the number of compression and garbage collection operations. The journal written by the journal engine 104 can be configured so that the processing of the above-mentioned journal does not need strong sorting. The journal engine 104 can be configured to solve the throughput performance problem in the journal write and improve the recovery speed in the following ways: the synchronous write overhead of the journal is reduced to a single I / O operation from multiple input / output (I / O) operations, the use of lock-free queue aggregation writes to control delays and improve throughput and / or provide an asynchronous interface to enhance the thread model. The key value engine 102 and the file system 114 can be integrated and configured to cooperate with each other, and the journal engine 104 can be used to store a pre-written log (WAL) with a predefined structure, and the predefined structure has a defined actual file size. The file size defined for WAL may then cause the need for less I / O operations, thereby improving performance, while alleviating the potential trade-off in terms of data consistency.

[0019] The hash engine 106 can be configured to process point queries within the key-value engine 102. In particular, the hash engine 106 is configured to reduce tail latency in point queries. The hash engine 106 includes separation of data and index components, and maintaining indexes in cache memory, such as by compressing the index and / or caching partial data. The partial data can be selected using, for example, a least recently used strategy.

[0020] The sorting engine 108 is configured to perform range scan operations while reducing write amplification factors and / or read / write latency associated with such operations. The sorting engine 108 is configured to use a partitioned log structure merge (LSM) tree. Classification of I / O streams and scheduling of tasks can further be performed by the sorting engine 108.

[0021] The garbage collection manager module 110 is configured to perform garbage collection and / or compression operations in the key-value store and file system 100. The garbage collection manager module 110 may be configured to reduce unnecessary data movement during garbage collection and / or compression operations in the key-value store and file system 100. The garbage collection manager module 110 may perform garbage collection and / or compression operations based on awareness of application-side data deletions, such as page expiration. The garbage collection and compression performed by the garbage collection manager module 110 may be configured to arrange data to support other modules, such as the sorting engine 108. The garbage collection manager module 110 may be configured to coordinate the preservation of data during garbage collection and / or compression operations.

[0022] The collaboration layer 112 is configured to facilitate collaboration between the key-value (KV) engine 102 and the file system (FS) 114. The collaboration layer 112 can further facilitate efficient compression and / or garbage collection operations in the key-value engine 102 based on the collaboration between the key-value engine 102 and the file system 114. The collaboration can reduce write amplification problems caused by compression and / or garbage collection operations. In an embodiment, the collaboration layer 112 can expose area usage information from the key-value engine 102 to the file system 114.

[0023] The file system 114 can be configured to separate data from the log and use a log-structured append-only write as a write model. In an embodiment, the file system can further provide pre-allocated data space, where synchronous writes occur only for data persistence, and in an embodiment, metadata persistence is not required. In an embodiment, data persistence and global log persistence for different files can be performed separately. These aspects of the file system can allow the file system to avoid some metadata persistence operations, such as operations caused by a single data write persistence operation.

[0024] The file system 114 can be configured to support general files and instant files. General files and instant files can both be written sequentially, or read sequentially or randomly. General files can be optimized to always maintain low latency in sequential reads or random reads. General files can be used to write data in batches without flushing the data to disk after each write, such as SST files. Storage space is allocated in large units, and a non-limiting example of the unit size is 1MB each. Large allocation units can reduce the metadata size for general files so that the metadata of all general files can be kept in memory during normal file system operations. By keeping the metadata in memory, no further I / O is required for metadata access for read operations on general files regardless of the read offset. This can reduce the read tail latency for general files. Instant files can be optimized for fast incremental synchronous writes while having good sequential and random read performance near the tail. Instant files can be used to write data that needs to be frequently flushed to disk to obtain instant persistence, such as pre-written log files of key-value systems. The data and metadata written by each individual can be bundled together for instant files. The bundled data and metadata can be written to a log file shared by all instant files. Bundling data and writing to a log file can increase the speed of incremental writes and synchronization operations. This approach supports sequential reads, but may trade off in terms of random reads. Since instant files are expected to be mostly read sequentially and random reads are mostly concentrated near the tail, the most recently written data of each instant file that is being actively written can be beam cached to improve read performance.

[0025] The file system 114 may include a user space I / O scheduler for assigning I / O priorities to different I / O types. The I / O scheduler marks the foreground I / O as a high priority, while the background I / O will be marked as a low priority. In addition, the key-value engine 102 may include a scheduler for scheduling its background tasks to ensure that each I / O issued by the upper layer application has a consistent I / O amplification rate. Through this common design of I / O scheduling in both the key-value engine 102 and the file system 114, since both I / O amplification and I / O latency are consistent, tail latency can remain stable and low. In addition, reading common files from the file system does not require I / O for metadata, and using a large space for common files can ensure that most read operations require a single I / O.

[0026] Kernel space 116 may contain disks 118. Disks 118 may include one or more storage media, such as solid-state drives (SSDs). In one embodiment, at least some of disks 118 are zoned storage (ZNS) SSDs.

[0027] The application 120 may be any suitable application that utilizes the key-value store and file system 100, such as online shopping, social media, metadata management applications, etc. The application 120 may interface with the key-value store and file system 100 via any suitable application programming interface (API). In an embodiment, the API may be specific to a particular type of file, such as having a file property as a generic file or an instant file determined by the API through which the file is received.

[0028] FIG. 2 illustrates a non-limiting example of a ranking engine arranged in accordance with at least some embodiments described and enumerated herein.

[0029] The sequencing engine 200 may be implemented and arranged according to at least some of the embodiments described and enumerated herein. Although shown as discrete components, various components may be divided into more components, combined into fewer components, or eliminated entirely while being considered within the scope of the disclosed subject matter. Those skilled in the art will appreciate that each function and / or operation of a component may be implemented individually and / or collectively by various hardware, software, firmware, or any combination thereof.

[0030] According to the non-limiting example embodiments described and enumerated herein, the sorting engine 200 may include various modules that can reduce WAF, reduce tail latency, support asynchronous input / output (I / O) execution / compression threads to prevent one I / O from blocking another I / O, support failure recovery, support multi-version concurrency control (MVCC), etc.

[0031] The initialization of the database (DB) class provides functions for managing and interacting with the underlying storage engine. The initialized DB class includes methods for initializing the database with specified options, including but not limited to creating an engine with given options, engine name and engine type, returning an engine handle, discarding the sorting engine 200 and its related data, retrieving an engine handle of a specified engine name, etc.

[0032] Initialization of the engine reader class provides an interface for reading data from the sorting engine 200; data is retrieved asynchronously and synchronously, respectively, from the sorting engine 200. Initialization of the iterator class provides an iterator for traversing data in a partitioned data structure, which will be discussed further below; initialization of one or more retrieval classes provides methods for, for example, returning a handle to a portion of a partitioned data structure, retrieving properties of a specific portion of a partitioned data structure, and retrieving the type of the sorting engine 200.

[0033] Initialization of the reader class provides interfaces for reading data from a specific portion of a partitioned data structure, and for retrieving data asynchronously and synchronously, respectively, within the scope of a portion of a partitioned data structure, for example.

[0034] Initialization of the write class provides functionality for writing data to the sort engine 200, as well as, for example, synchronous writing to store key-value pairs, merge entries for a given key with new values, delete key-value pairs, remove key-value ranges, and the like.

[0035] Initialization of the allocation class provides functions for allocating key ranges to the key-value sorting engine 200, rejecting access to keys that do not belong to any allocated range, etc. Initialization of the recovery class provides functions for checking range consistency between the application and the key-value layer, splitting a portion of a partitioned data structure by splitting the key, merging adjacent portions of a partitioned data structure, etc.

[0036] Initialization of a garbage collection (GC) filter provides functionality for reclaiming obsolete key-value pairs during a background GC process, and according to at least some non-limiting example embodiments, provides a virtual filter that can be overridden to define filtering logic.

[0037] Hereinafter, the above-mentioned one or more parts of the partitioned data structure may be alternatively referred to as one or more shards, which may refer to partitioned data in a database or engine structure.

[0038] Therefore, the initialization of the shard writer class provides functions for writing data to the shard, merging entries, deleting entries, removing a range of entries, etc.

[0039] As shown, the sorting engine 200 is implemented as a component or module for implementing database management of a non-volatile storage device (e.g., SSD 265). The device 20 includes at least a shard manager 205, an I / O classifier 215, a job scheduler 220, an asynchronous API manager 225, a fault tolerance manager 230, a timestamp manager 235, a multi-tenant manager 240, and a compression / garbage collection (GC) manager 250. Further described is a file system 260.

[0040] File system 260 refers to a data structure that controls the storage and retrieval of data to and from a storage medium (eg, SSD 265 ).

[0041] The shard manager 205 refers to a component or module that is programmed, designed, or otherwise configured to automatically or manually split data in a large LSM tree structure into smaller LSM tree structures (i.e., fragment 1 210A, fragment 2 210B, ..., fragment N 210N); thereby reducing the total number of layers in the LSM tree and minimizing write amplification. That is, when the data within the LSM tree structure reaches a certain threshold, the data structure can be manually or automatically split into two sub-shards. Conversely, when the data within the data structure is reduced due to deletions and its amount drops below a specific threshold, the data structure can be manually or automatically merged with one or more adjacent fragments to form a single fragment. Therefore, the shard manager 205 is further programmed, designed, or otherwise configured to merge two or more shards whose corresponding data amounts are below a predetermined threshold.

[0042] The shard manager 205 ensures the atomicity of the split operation, that is, the split is completely successful or completely failed without splitting any data in the corresponding shard in the case of failure with other data. Any failed operation must be properly rolled back to maintain data integrity. In addition, the shard manager 205 operates in a way that minimizes the impact on the front-end I / O operations.

[0043] I / O classifier 215 refers to a component or module that is programmed, designed, or otherwise configured to classify and label each I / O based on its type to facilitate the lower-level file system to prioritize the scheduling between different types of I / O to and from the storage device, thereby ensuring stable latency for individual I / O operations. That is, in order to ensure timely thread scheduling of high-priority tasks, multiple different background thread pools are maintained, each thread pool is designed to perform different types of background tasks, including but not limited to refresh operations, L0->L1 compression tasks, and L1->LN compression tasks in descending order of priority. L0->L1 compression directly preempts L1->LN compression.

[0044] Using separate thread pools for different tasks does not completely solve the I / O problem between different tasks that fairly compete for limited backend I / O bandwidth. The I / O bandwidth of high-priority tasks may be preempted by low-priority tasks, resulting in untimely processing and affecting foreground latency stability. Therefore, in order to prevent blocking their I / O execution, the bandwidth thresholds for flushing and L0->L1 compression are set to W bytes / second respectively. Therefore, the bandwidth thresholds of the three tasks are dynamically adjusted based on their priorities to gradually increase or decrease the bandwidth caps for different tasks based on their priority order.

[0045] Job scheduler 220 refers to a component or module that is programmed, designed, or otherwise configured to cooperate with FS layer 260 to ensure stable latency of individual I / O operations to and from storage devices, such as SSD 265. Job scheduler 220 provides stable read amplification in the KV layer and prevents adverse conditions, such as WriteStall and WriteStop, to enhance or even optimize data retrieval and reduce tail latency.

[0046] The asynchronous API manager 225 refers to a component or module that is programmed, designed, or otherwise configured to manage I / O operations to and from a storage device (e.g., SSD 265) so that I / O waits do not block upper-layer threads from performing other tasks. That is, the asynchronous API manager 225 collaborates with the underlying file system 260 to alleviate the blocking of I / O operations, thereby improving parallelization, latency, and response time by collaboratively selecting executors for executing asynchronous tasks and asynchronous task callbacks.

[0047] Note that an executor is a component that manages the execution of tasks in a concurrent manner to provide control over factors such as thread pools, thread priorities, and task scheduling. It is crucial to determine which entity or component will have the authority to decide and configure the executor that handles the body of an asynchronous task. Furthermore, a callback is a function or executable component or module that is executed when an asynchronous task is completed. The executor assigned to execute the callback facilitates the results of the task and subsequent actions. Defining who will be responsible for setting up and configuring this callback executor is a priority to ensure the correct execution and handling of callbacks associated with asynchronous tasks.

[0048] Fault tolerance manager 230 refers to a component or module that is programmed, designed, or otherwise configured to provide sector-level fault tolerance so that a single sector corruption within a file does not affect data consistency and visibility.

[0049] That is, in order to prevent damage to individual sectors on the disk from making specific files or data unreadable, thereby causing the entire DB data of the upper-level distributed system to be reconstructed, or to generate data redundancy blocks for key file data to ensure that even if several consecutive sectors within the file are damaged, the file data can be correctly restored, or the file system 260 provides redundant protection for the metadata therein to prevent the unavailability of metadata from causing the entire file system 260 to be unreadable.

[0050] The timestamp manager 235 refers to a component or module that is programmed, designed, or otherwise configured to automatically or manually timestamp each KV pair so as to conveniently implement MVCC-related features.

[0051] The timestamp manager 235 ensures that upper layer applications (e.g., L0 or L1) have timestamp values ​​for the same key that strictly increase over time, thereby ensuring expected behavior during reads. The user timestamp can be combined with the user key and stored as a unified key within the sorting engine 200. During encoding, the timestamp can be used to ensure that internal keys in the sorting engine 200 are sorted first.

[0052] like Figure 2B As shown, the timestamp manager 235 addresses MVCC by enabling a timestamp to be assigned to each KV pair. This timestamp functionality seamlessly integrates MVCC related features. Higher timestamp values ​​represent newer versions of the data, and upper layer applications ensure that the timestamp values ​​of the same key strictly increase over time, thereby ensuring expected behavior during reading. The timestamp will be combined with the user key and stored as a unified key within the engine 200. During encoding, the internal keys in the engine 200 will first be sorted based on the timestamp and then sorted based on the internal segment number.

[0053] Even if the upper layer application ensures strict incremental ordering of timestamps for the same key, the timestamp does not replace the internal segment number in the engine 200, because the timestamps from the upper layer application usually follow strict increments within a key or a key range. For example, the database LSN (log sequence number) provides incremental writes within a segment, while the LSN corresponding to the data between segments does not necessarily provide sequential ordering. If the timestamp is used directly as the internal segment number, the internal snapshot mechanism is to maintain an independent snapshot number for each incremental unit (e.g., segment), resulting in excessive coupling between the layer and the complex snapshot mechanism. The internal segment number is globally incremented and can be used to traverse the internal snapshot, which means that a single value can represent a snapshot, such as Figure 2B shown.

[0054] KV storage (i.e., KV layer) is used to enable read, write, and delete functions to support timestamps to allow users to associate timestamps with KV operations; implement garbage data recovery (deletion) through the compression process to remove expired timestamp data and ensure data integrity; and trigger active compression when capacity limits are reached, thereby identifying SST / Blobs with the highest number of garbage versions. This initiates the compression process to efficiently reclaim storage space. By combining these features, robustness and efficiency are enhanced, providing comprehensive support for timestamp management, ensuring data integrity, and optimizing storage utilization.

[0055] These features enhance the robustness and efficiency of the engine 200, providing comprehensive support for timestamp management, data integrity, and efficient storage utilization.

[0056] The multi-tenant manager 240 refers to a component or module that is programmed, designed, or otherwise configured to provide fragment-level resource limits and isolation, because the upper layer application (e.g., L0 or L1) can have different resource usage limits for different fragments. Non-limiting examples of such resources include I / O bandwidth, memory size, and number of threads, such as the number of asynchronous I / O execution threads / compression threads, because the upper layer application is configured to have resource usage caps for various engines and fragments. In addition, the multi-tenant manager 240 is programmed, designed, or otherwise configured to provide periodic monitoring and resulting statistics on the use of each resource type, thereby allowing the upper layer to dynamically adjust its quota value based on the resource monitoring status of each resource on different fragments, thereby facilitating multi-tenant functionality.

[0057] The multi-tenant manager 240 is programmed, designed, or otherwise configured to monitor the real-time resource usage status of each shard so that, for example, an upper-layer application can dynamically adjust the quota value of each shard accordingly to maximize resource utilization.

[0058] Compression / GC manager 250 refers to a component or module that is programmed, designed, or otherwise configured to coordinate the background mechanism of GC / compression in storage (e.g., SSD 265) with GC in file system 260 to proactively reduce end-to-end WAF.

[0059] 3A-3C illustrate various stages of a non-limiting example of data segmentation performed and managed by the shard manager 205. As shown in FIG3A-3C, data segmentation is designed to be completed quickly to minimize any impact on the front-end I / O. In addition, the data is segmented into a left half and a right half in order to make distribution and management in a distributed database system more efficient by, for example, reducing the height of the LSM tree.

[0060] FIG3A illustrates stage 1 of data segmentation according to a non-limiting example embodiment. Assume that shard 1 is considered as a parent shard and shard 2 is considered as a child shard. In addition, the range of parent shard 1 before segmentation is [X, Y), and after segmentation at the split point (splitpoint), the updated range of parent shard 1 is [X, SplitPoint). Therefore, the range of child shard 2 is [SplitPoint, Y). Shard 1 has data of L0, L1, L2, L-1 layers.

[0061] In stage 1 shown in the box of FIG. 3A , the front-end write I / O is blocked, while the read I / O is not affected; background tasks including compression, flushing, and GC are temporarily suspended; the file deletion switch is temporarily disabled; the current write memory table is immutable, that is, it has immutable content; child shard 2 is created, and the memory version of parent shard 1 (including the immutable memory table) is cloned and assigned to shard 2. Therefore, shard 1 and shard 2 completely share immutable data until the end of the split process. Then, a new memory table is created for child shard 2, and it is added to the memory version corresponding to shard 2, and the key range metadata of shard 1 and shard 2 is updated. Then, the front-end write I / O blocking is released, and the background task execution switch is reactivated.

[0062] In phase 2, as shown in the box of FIG3B, the memory table and SST including the split point are split by setting the split point to create a temporary SST file, and then the subsequent refresh and compaction tasks in the corresponding shards generate SST files that do not include data before and after the split point at the same time; the existing SST containing data before and after the split point is rewritten to avoid overlap. Data within the original shard 1 key range but before the split point continues to be written to shard 1, while data after the split point is written to shard 2.

[0063] In phase 3, as shown in the box of FIG3C , the version data of the split shards 1 and 2 are written to the storage device while maintaining atomicity. This is done by temporarily pausing background tasks, such as compaction, flushing, GC, etc.; removing files that do not intersect with the corresponding range from the memory version. Atomically flush the version data of shards 1 and 2 from memory to disk. After completion, the suspended background tasks (e.g., compaction, flushing, GC, etc.) are reopened for execution.

[0064] Note that if the phase 1 split fails before phase 2, the data on disk remains unaffected. The in-memory split process can be rolled back by, for example, removing the temporarily created shard 2 and reverting to the phase 1 split point.

[0065] If phase 1 succeeds, but an I / O exception during phase 3 causes the split to fail, two options for proceeding include performing a proper rollback of phases 1 to 3 and returning to the previous state. Phases 1 and 2 require rolling back the memory state, while phase 3 requires rolling back the disk state. Subsequent write requests are blocked, and the upper layer L0 is instructed to restart the database. In this case, restarting the database clears any data that was not successfully processed at the last moment.

[0066] After the split, some blob files storing values ​​in key-value (KV) format can be shared among multiple shards. Therefore, care needs to be taken during garbage collection (GC) and blob file deletion. To ensure efficient management, information about shard references for each blob file involved in adding references and removing references is maintained in the storage device 265.

[0067] During the split process, a reference is added to the shared blob file and the blob file is migrated from the parent shard 1 to the child shard 2.

[0068] As used herein, "blob file" is a term and may refer to an object storage solution for a cloud service, which may be a binary file used to store data outside of an LSM tree, e.g., storing (multiple) values ​​in a separate key-value system. An SST entry may include a key or pointer to one or more blob files, e.g., a blob file may be shared between one or more SST entries.

[0069] Figure 3A belongs to memory segmentation, which is designed to complete the process quickly and minimize the impact on front-end I / O. Memory segmentation includes the following, assuming that shard 1 [1, 10) is split into shard 1 and shard 2 at point 5: Front-end write I / O is blocked, while read I / O is not affected. Background tasks such as compression, flushing, and GC are temporarily suspended. The file deletion switch is temporarily disabled. SwitchMemtable: The current write Memtable is converted to an immutable Memtable with immutable content (for example, Mem1 in the figure switches to Imm1). A new shard 2 is created, and the memory version of shard 1 (which includes the immutable part (immutable Memtable and SST)) is cloned and assigned to shard 2. This means that shard 1 and shard 2 completely share immutable data until the segmentation process ends. A new Memtable is created for shard 2 and added to its memory version (for example, Mem3 is a newly created Memtable for shard 2). Update the KeyRange metadata of shard 1 and shard 2. After stage 2, data within the original Shard 1 KeyRange but before the SplitPoint should continue to be written to Shard 1, and data after the SplitPoint should be written to Shard 2 (for example, Shard 1 KeyRange is updated to [1, 5), and Shard 2 KeyRange is updated to [5, 10), as shown in the figure). The front-end write I / O blocking is released. The background task execution switch is reactivated.

[0070] Figure 3B involves preparation for disk splitting, which is to shorten the processing time of phase 3, because phase 3 blocks flushing and compaction. In order to limit the total size of WriteBuffer, writes can only be unblocked for a short time. In the preparation for disk splitting phase, the current SST and the Imm containing the SplitPoint are split at the SplitPoint to create a new temporary SST file. These steps include:

[0071] Setting SplitPoint: After setting the SplitPoint, all subsequent flush and compaction tasks in that shard will generate SST files that do not include both data before and after the SplitPoint. All existing SSTs that contain data both before and after the SplitPoint are rewritten to ensure that they no longer overlap, allowing for a smooth separation of all SST files in phase 3, although these newly generated SST files are temporarily invisible to shards 1 and 2 (for example, SST01 in the figure is rewritten as<SST05,SST06> , SST02 is rewritten as<SST07,SST08> ).

[0072] Note that in this step, no split rewrite operation is performed on the BlobFile, because almost all BlobFiles are out of order, and the design rewrite will cause most BlobFiles to need to be rewritten during the split process, resulting in significant write amplification overhead and extending the split execution time. Instead, the BlobFile is managed through shared file management.

[0073] Figure 3C belongs to disk splitting, that is, the version data of shard 1 and shard 2 after splitting is written to disk, and the writing process must maintain atomicity. The specific steps include temporarily pausing background tasks (compression, refresh, GC); removing SSTs and BlobFiles in shard 1 and shard 2 that do not intersect with their corresponding ranges from the memory version. Since most SSTs are arranged sequentially (L1~LN), the overlapping SSTs in stages 1 and 3 have been rewritten to ensure complete separation. However, in the case of BlobFile, two shards may still share the same BlobFile after splitting. The version data of shard 1 and shard 2 is refreshed from memory to disk atomically. After successful completion, the background tasks (e.g., compression, refresh, GC) will be reopened.

[0074] Regarding shard split consistency, if the phase 1 in-memory split fails before phase 2, the data on disk remains unaffected. The in-memory split process can be rolled back, including removing the temporarily created shard 2 and restoring the split point setting of phase 1.

[0075] If phase 1 succeeds, but an I / O exception during phase 3 causes the split to fail, options include performing a proper rollback of phases 1 through 3 and returning to the previous state. Phases 1 and 2 require rolling back some in-memory state, while phase 3 requires rolling back some disk state.

[0076] Similar to RocksDB, BGError is set to block subsequent write requests and notify the upper layer to restart the DB. In this case, restarting the DB will clear the data that was not successfully processed at the last moment (considering the low probability of local disk I / O errors, this method is simpler and more reliable).

[0077] Figure 3D is an illustration of a consistent cross-shard range scan according to a non-limiting example embodiment of data partitioning described and enumerated herein.

[0078] As mentioned above, reference Figure 1 , the collaboration layer 112 is configured to facilitate collaboration between the key-value engine 102 and the file system 114. According to the non-limiting example embodiments of Figures 3A to 3C, due to the adoption of a multi-fragment strategy, the upper-layer application may encounter one or more fragments when performing a range search using an iterator. In addition, the range search operation can trigger multiple find and next sub-operations. In such a process, the fragments related to the range search may be affected by split and merge operations. Therefore, the range search operation is a result of traversing without any duplicate or missing entries, and returns them to the upper-layer application in the correct order.

[0079] like Figure 3D As shown, for descriptive purposes only, assume that there are four shards at time T1. At this time, the KV layer receives a range query request that traverses the range [B, +∞). After receiving the request, the KV layer obtains the snapshot number. Again, for descriptive purposes only, assume that the returned snapshot number is 2. Snapshots of all shards are managed uniformly in the DB to ensure that once a snapshot is taken, all shards in the DB can access the snapshot. This consistency allows range queries across multiple shards. After taking (multiple) snapshots, the range search will start from position B5 in shard 1.

[0080] At time T2, Shard 1 is split into Shard 1 and Shard 5. The split shards can still access the snapshot created by the DB at time T1 and save the data related to the snapshot during the compaction process, such as A2, C1, D2.

[0081] After traversing all the data in fragment 1 at time T4, the current cursor position can be used to locate the position of the next fragment. For example, if the cursor is located at C1, the first fragment greater than C1 is fragment 7. Therefore, starting from a position greater than C1 (i.e., cursor = D3), traversal continues within fragment 7. Similarly, after traversing the last element F1 in fragment 7 at time T5, traversal continues from position G1 in fragment 8. This process is repeated until all relevant elements are traversed.

[0082] In a range search, multiple shards may be traversed, which may incur additional search overhead because the first search for each shard uses a seek operation. To mitigate this overhead, collaboration between the KV layer and upper layers can be considered. For example, during shard splitting, appropriate split points can be selected to avoid scanning operations across shard ranges. If range searches in NDB always include a fixed segment ID prefix, each shard split process ensures that KV data with the same segment ID prefix is ​​assigned to the same shard, thereby avoiding the performance impact of cross-shard read operations.

[0083] 4A-4C illustrate various stages of a non-limiting example of data merging performed and managed by shard manager 205. Merging at least shard 1 and shard 2 includes merging two adjacent shards having overlapping ranges into one.

[0084] Assuming that the leader shard 1 and the adjacent follower shard 2 have a volume less than a predetermined threshold level and are therefore to be merged, the range of shard 1 is [X, Y) and the range of shard 2 is [Y, Z). The range of the resulting merged shard will be [X, Z).

[0085] In the data merge phase 1, as shown in the box of Figure 4A, basic metadata is added to the leading shard 1 to provide access to the data stored in the following shard 2. Therefore, all new read and write requests in the range [X, Z) are directed to the leading shard 1, and no new requests are accepted by the following shard 2. Reads will not be blocked, but writes, flushes, and compaction operations will be blocked.

[0086] In data merge phase 2, as shown in the box of Figure 4B, the memory state merged in phase 1 is executed together with the disk operation, and the follow-up shard 2 is removed from the DB. Reading and writing will not be blocked in the short term, but flushing and compaction operations are still blocked.

[0087] Merging shards is depicted in FIG. 4C to allow efficient merging of shards, ensure data integrity, and minimize interference with read and write operations during the merging process.

[0088] In data merging, when a shard creates a new blob file, it automatically establishes a reference to the blob file. On the other hand, when a shard intends to delete a blob file, it must first confirm whether the blob file is exclusively referenced by itself. If so, the shard will immediately delete and remove the reference entry. Otherwise, the shard should only remove its own reference to the blob file. In addition, the latest snapshot of the blob file reference information will be written to the new list. This periodic checkpoint effectively organizes the historical reference changes of the blob files stored in the list, thereby ensuring data consistency and minimizing redundancy.

[0089] In addition, when the upper distributed system layer migrates the entire shard data from one node to another, all relevant files (e.g., SST, blob, etc.) of the shard are transmitted to the target node over the network, and the ingest file operation is performed. However, since blob files can be shared between multiple column families (CFs), the same blob file can contain valid data from multiple CFs. In order to reduce the amount of data transmitted over the network, a garbage collection (GC) operation can be performed on all shared blob files before migration, thereby copying all relevant data of the shard to a new set of exclusive blob files.

[0090] Shard migration includes a preparation phase, through which shared blob files are selectively cleaned up to reduce the amount of data transmitted over the network. Shard migration also includes an execution phase, in which, when initiated by the source node, all SST and blob files are exported from the shard in file format. The relevant files are directly copied and exported, and the latest version snapshot of the shard is written to the temporary manifest file. In addition, when initiated by the destination node, a new empty shard is created at the destination node based on the data received from the source node over the network. In order to avoid file number conflicts, all SST and blob files to be imported are reassigned new file numbers in the target database, followed by the rename operation. The contents of the received temporary manifest file are parsed to generate a version snapshot, and the new file numbers of all corresponding files are applied to the version snapshot. The new version snapshot will be attached to the manifest of the DB.

[0091] Shard migration involves transferring the relevant data files of the shard to the target node and carefully managing the shared blob files to reduce data duplication during the migration process. The migration process ensures data consistency and minimizes network traffic, resulting in efficient and successful shard relocation.

[0092] The sharding manager 205 implements sharding manually or automatically.

[0093] 3A to 3C, and even FIG. 4A to 4C described below, the upper storage system calls the splitShard / MergeShard interface. According to a non-limiting example embodiment of manual sharding, data splitting phases 2 and 3 are performed asynchronously in the background to allow phase 1 to return immediately after completion.

[0094] Automatic sharding is implemented inside the KV engine, which periodically selects target shards for data splitting and data merging operations based on specific rules. Both splitting and merging use the following two basic operations:

[0095] Automatic shard splitting: When the upper-layer system cannot predict the distribution range of keys in advance, the shard size is dynamically monitored at runtime. When the database is opened, a dedicated background thread is used to monitor the size status of the shards and issue instructions for data splitting or data merging based on the monitored size / volume conditions.

[0096] Automatic fragment merging: In order to control the upper limit of the total number of fragments in an instance, it is necessary to automatically merge consecutive small fragments to limit the management pressure of fragments in a single database. The thresholds for initiating automatic fragment merging include:

[0097] The size of the leading fragment 1 + the size of the following fragment 2 ≤ the merge size threshold

[0098] Leading Shard 1 TPS ≤ TPS Threshold & Following Shard 2 TPS ≤ TPS Threshold

[0099] When performing a range search using an iterator, the upper-layer application of the corresponding shard may encounter one or more shards. In addition, the range search operation triggers multiple seek and next child operations. Therefore, the shards related to the range search are also affected by the data splitting and data merging operations. Therefore, the range search operation should traverse the results without any duplicate or missing entries and return them to the upper-layer application in the correct order.

[0100] When a snapshot is taken, all shards in the database have access to that snapshot. This consistency allows range queries across multiple shards.

[0101] 4A shows that fragment merging is used to merge two adjacent fragments with overlapping ranges into one. The fragment merging process is similar to fragment segmentation, but it does not require a preparation phase because there will be no range overlap between fragments after segmentation.

[0102] The fragment merging includes (1) assuming that the two fragments to be merged are adjacent: the left fragment range is [X, Y), and the right fragment range is [Y, Z]. The fragment range after merging will be [X, Z); and the left range fragment before merging is called the leading fragment, and the right range fragment is called the following fragment.

[0103] Figure 4B relates to memory merging, by which basic metadata is added to the leading shard to allow it to access data stored in the following shards. Next, new read and write requests in the range [X, Z) are directed to the leading shard, and the following shards no longer accept new requests. This process does not block reads, but blocks writes, refreshes, and compaction operations.

[0104] Figure 4C involves a disk merge, where the memory state merged in phase 1 is performed along with the actual disk operation, and the following fragment is removed from the DB (the removal process is managed through reference counting, and the fragment is only released when all references to it reach zero). This process does not block reads and writes in the short term, but it still blocks flush and compaction operations.

[0105] Shard merging allows for efficient merging of shards, ensuring data integrity and minimizing disruption to read and write operations during the merging process.

[0106] Figure 5A A non-limiting example storage process flow is shown in accordance with at least some embodiments described and enumerated herein.

[0107] As shown, process flow 500 includes operations, actions, or functions, as shown in representative blocks 505, 510, 515, 520, 525, 530, 535, 540, and 545. These various operations, functions, or actions may correspond to software, programming code, or programming instructions executable, for example, by firmware that causes these functions to be performed. Although shown as discrete blocks, obvious modifications may be made, for example, two or more blocks may be reordered; furthermore, one or more blocks may be added; and various blocks may be divided into more blocks, combined into fewer blocks, or even eliminated, depending on the desired implementation. Process flow 500 may begin at block 505.

[0108] Block 505 (Initialize DB) refers to the initialization of a database (DB) class, which is used to provide functions for managing and interacting with the underlying storage engine. The initialized DB class includes methods for initializing the database, for example, with specified options, including but not limited to creating an engine with given options, engine name and engine type, returning an engine handle, discarding the sorting engine 200 and its related data, retrieving an engine handle of a specified engine name, etc.

[0109] Block 510 (Split Data) refers to the shard manager 205 automatically or manually splitting data from a large LSM tree structure into smaller LSM tree structures, i.e., shards, thereby reducing the total number of layers in the LSM tree and minimizing write amplification. Alternatively, when the data within the data structure decreases due to deletions and its amount drops below a certain threshold, block 510 may refer to manually or automatically merging the data structure with one or more adjacent shards to form a single shard.

[0110] Box 515 (Classify and Tag I / O) refers to the I / O classifier 215 classifying and tagging each I / O based on its type to facilitate the lower level file system to prioritize the scheduling between different types of I / O to and from the storage device, thereby ensuring stable latency for individual I / O operations and thus ensuring timely thread scheduling of high priority tasks.

[0111] According to the non-limiting example embodiments described and listed herein, I / O sent to FS 260 may be tagged to distinguish between foreground and background tasks. On the file system side, I / O scheduler 220 may prioritize I / O based on such tags to minimize foreground latency. To ensure that high priority tasks receive timely thread scheduling, three background thread pools may be maintained to schedule different types of background tasks, including (1) flushing using a high priority thread pool, (2) L0->L1 compression using a medium priority thread pool, and (3) L1->LN compression using a low priority thread pool.

[0112] refer to Figure 5D For example, when L0->L1 compression is executed, L1->L2 compression can be preempted directly. When L0->L1 compression is needed, the current execution of L1->L2 compression should be stopped immediately. The I / O bandwidth of high-priority tasks may be preempted by low-priority tasks, resulting in untimely processing and affecting the stability of foreground latency. Therefore, assuming that the upper limit of foreground writes is W bytes / second, in order to prevent blocking I / O execution, the bandwidth limits of flush and L0->L1 compression are set to W bytes / second respectively. Using three independent rate limiters for each type of thread task may cause L1->LN compression to continue to execute within a very low bandwidth limit, even when no flush or L0->L1 compression is executed. Therefore, the bandwidth thresholds of the three tasks are dynamically adjusted based on their priorities within the same rate limiter. In order to gradually increase or decrease the bandwidth upper limit of different tasks based on their priorities, the following operations can be implemented.

[0113] Assuming the total rate limiter threshold is T bytes / second, the bandwidth thresholds for refresh and L0->L1 compression can be dynamically controlled within the range of [W / 5, W]. Initially, the I / O bandwidth thresholds for refresh tasks and L0->L1 compression can be set to their lower limits of W / 5 bytes / second, and the threshold for L1->LN compression can be set to the remaining bandwidth (T-2W / 5) bytes / second. At every fixed interval, such as 100ms, the new I / O bandwidth thresholds for refresh, L0->L1, and L1->LN compression are adjusted in turn. When the bandwidth thresholds for the current two types of tasks are determined, the bandwidth threshold allocated for the lowest priority task can be obtained by subtracting the sum of the thresholds of the first two tasks from the total threshold. Specific adjustment considerations for the bandwidth threshold of each task: (1) the adjustment is mainly based on the statistical proportion of triggering the rate limiter upper limit in the previous cycle of the task, assuming that the proportion is P; (2) when P is less than the minimum threshold (for example, 20%), the I / O bandwidth threshold for the task can be set to Min(T-used, prevValue*95%); (3) when P is greater than the maximum threshold (for example, 90%), the I / O bandwidth threshold for the task can be set to Min(T-used, prevValue*105%); and (4) otherwise, the I / O bandwidth threshold for the task remains unchanged in the next cycle (maintaining prevValue).

[0114] The subsequent RateLimit of SortedEngine limits the background task traffic of the sorting engine layer and collaborates with the file system to provide a unified front end.

[0115] Block 520 (Prioritize I / O) refers to the job scheduler 220 cooperating with the FS layer 260 to ensure stable latency for individual I / O operations to and from the storage device (e.g., SSD 265). The job scheduler 220 provides stable read amplification in the KV layer and prevents adverse conditions such as WriteStall and WriteStop to enhance or even optimize data retrieval and reduce tail latency. See Figure 5B for its example implementation.

[0116] The main cause of write stalls and / or write stops can be a full write buffer, which is mainly caused by a slow flush process. Higher levels of concurrent compaction may eat up the I / O bandwidth for flushing, causing the flush rate to be slower than the write buffer can write.

[0117] I / O amplification can be determined by the total number of levels in the LSM within each shard and the number of L0 files. The size of each shard through dynamic partitioning can be controlled to keep the total number of LSM levels within a fixed range. Therefore, read I / O amplification is mainly affected by the number of L0 files. An increase in the number of L0 files may be due to slow L0->L1 compression, which may be caused by the following reasons: (1) a single thread pool for the current compression, which may cause queuing delays for L0->L1 compression when a higher-level compression is in progress, or (2) concurrent higher-level compressions occupying the I / O bandwidth of L0->L1 compression.

[0118] like Figure 5B As shown in more detail Figure 5C As shown, in order to ensure the stability of latency, different background tasks can be distinguished based on their priorities, and different I / O bandwidth resources can be allocated accordingly. Background tasks can be classified.

[0119] Fast flushing can be prioritized, meaning that flushing has the highest priority, clearing enough space in the memory's write buffer to accommodate foreground write requests in a timely manner. The speed of the flush directly affects the latency of the write tail. In addition, the L0->L1 compression task can be set to the second priority. If L0->L1 compression is slow, it will directly increase the number of read I / O operations, thereby extending the read latency tail. In addition to flushing and L0->L1 compression, L1~LN compression has the lowest priority. These compression tasks can be used to maintain the form of the LSM and will not have a significant impact on read and write latency in the short term.

[0120] Not all background tasks need to share system resources equally. Tasks such as flushing and L0->L1 compaction have higher priority. If these operations are not completed in a timely manner, it will lead to increased write stall events and amplified read I / O operations.

[0121] Block 525 (Predict Workload Patterns and Prioritize R / W) refers to the job scheduler 220 further predicting workload patterns and adjusting the scheduling of read and write operations accordingly, thereby maintaining stable read amplification and preventing performance degradation.

[0122] Box 530 (coordination background mechanism) refers to the I / O classifier 215 optionally ensuring timely thread scheduling of high priority tasks and maintaining multiple different background thread pools to perform different types of background tasks, including but not limited to flush operations, L0->L1 compression tasks and L1->LN compression tasks in descending priority order. L0->L1 compression directly preempts L1->LN compression, thereby reducing end-to-end WAF.

[0123] Block 535 (prioritize and execute asynchronous read requests) refers to the asynchronous API manager 225 optionally supporting asynchronous reads, thereby allowing I / O to wait without blocking upper-level threads to perform other tasks.

[0124] Box 540 (Recovering from Single Sector Damage) refers to the fault-tolerance manager 230 optionally providing sector-level fault-tolerance capabilities so that a single sector damage in a file does not affect data consistency and visibility. For example, the fault-tolerance manager 230 generates data redundancy blocks for critical file data to ensure that even if several consecutive sectors in the file are damaged, the data of the file can be correctly recovered. Alternatively, the file system 260 provides redundancy protection for the metadata therein to prevent the unavailability of the metadata from causing the entire file system 260 to be unreadable.

[0125] Box 545 (allocate quotas) refers to the multi-tenant manager 240 optionally providing fragment-level resource limits and isolation, because the upper layer application (e.g., L0 or L1) can have different resource usage limits for different fragments by, for example, providing periodic monitoring and resulting statistics on the usage of each resource type, thereby allowing the upper layer to dynamically adjust the quota value on each resource type on different fragments based on its resource monitoring status, thereby facilitating multi-tenant functionality.

[0126] Figure 6 An illustrative computing embodiment is shown in which any of the processes and sub-processes of training a unified transformer-based visual place recognition (VPR) training framework can be implemented as executable instructions stored on a non-transitory computer-readable medium. For example, the computer-readable instructions can be executed by a processor of a device having a network element and / or any other device corresponding thereto, as referenced herein, particularly applicable to the above-mentioned application and / or program corresponding to the ranking engine 200.

[0127] In a very basic configuration, computing device 500 may typically include at least one or more processors 602 , memory 604 , one or more input components or modules 606 , one or more output components or modules 608 , a display component or module 610 , computer-readable media 612 , and a transceiver 614 .

[0128] The processor 602 refers to, for example, a microprocessor, a microcontroller, a digital signal processor, or any combination thereof.

[0129] The memory 604 refers to, for example, a volatile memory, a non-volatile memory, or any combination thereof. The memory 604 stores therein an operating system, one or more applications corresponding to the model 205, and / or program data. That is, the memory 604 stores executable instructions for implementing any of the above functions or operations, and therefore, the memory 604 can be regarded as a computer-readable medium.

[0130] Input component or module 606 refers to a built-in or communicatively coupled keyboard, touch screen, or telecommunication device. Alternatively, input component or module 606 includes a microphone configured to cooperate with a speech recognition program that may be stored in memory 604 to receive voice commands from a user of computing device 600. In addition, input component or module 606, if not built-in to computing device 600, may be communicated via a short-range communication protocol (including but not limited to radio frequency or ) is communicatively coupled to computing device 600.

[0131] Output component or module 608 refers to a component or module built-in or removable from computing device 600 that is configured to output commands and data to an external device.

[0132] The display component or module 610 refers to a solid-state display that can have a touch input function, for example. That is, the display component or module 610 can include functions that can share or replace the functions of the input component or module 606.

[0133] Computer readable medium 612 refers to a separable machine readable medium that is configured to store one or more programs that embody any of the above functions or operations. That is, the computer readable medium 612 that can be received or otherwise connected to a drive component or module of the computing device 600 can store executable instructions for implementing any of the above functions or operations. These instructions can be complementary or otherwise independent of the instructions stored by the memory 604.

[0134] The transceiver 614 refers to a network communication link of the computing device 600, which is configured as a wired network or a direct wired connection. Alternatively, the transceiver 614 is configured as a wireless connection, such as radio frequency (RF), infrared, and other wireless protocols.

[0135] From the foregoing, it can be understood that various embodiments of the present disclosure have been described herein for illustrative purposes, and various modifications may be made without departing from the scope and spirit of the present disclosure. Therefore, the various embodiments disclosed herein are not intended to be limiting, and the true scope and spirit are indicated by the following claims.

[0136] aspect

[0137] Aspect 1. A method for optimizing a non-volatile storage device by performing an operation, the operation comprising:

[0138] splitting the data in a first log structure merge (LSM) tree structure into partition fragments to reduce the number of levels for the data represented in the first LSM tree structure,

[0139] wherein each partition shard represents an independent LSM tree structure, thereby providing scalability and flexibility for the data represented in the first LSM tree structure,

[0140] When the data amount of a corresponding partition fragment of the partition fragments reaches a threshold level, splitting the corresponding partition fragment of the partition fragments into at least a parent fragment and a child fragment; and

[0141] When the data amount of a corresponding partition fragment among the partition fragments decreases to an amount less than the threshold level, the corresponding partition fragment among the partition fragments is merged into an adjacent partition fragment among the partition fragments.

[0142] Aspect 2. The method according to aspect 1, wherein the method further comprises directing new read and write requests to the sub-shard.

[0143] Aspect 3. A method according to aspect 1 or 2, wherein the splitting includes splitting an existing ordered string table (SST) between the parent fragment and the child fragment.

[0144] Aspect 4. The method according to any one of aspects 1 to 3, wherein

[0145] the fragment having a volume less than the threshold level is a following fragment, and the adjacent fragment into which the following fragment is merged is a leading fragment,

[0146] Basic metadata from the following shards is added to the leading shard to provide access to data stored in the following shards, and

[0147] New read and write requests are directed to the boot fragment.

[0148] Aspect 5. The method according to any one of aspects 1 to 4, further comprising:

[0149] I / O to and from each of the shards is categorized based on type.

[0150] Aspect 6. The method according to any one of aspects 1 to 5, wherein the classification comprises:

[0151] Tag the corresponding I / O with labels to distinguish between foreground and background tasks, and

[0152] Scheduling of corresponding I / Os for each of the shards is prioritized based on the corresponding tags.

[0153] Aspect 7. The method according to any one of aspects 1 to 6, further comprising:

[0154] Maintain predictable latency bounds for read and write operations by:

[0155] predicting a workload pattern for each of the corresponding shards based on the access pattern, and

[0156] Scheduling of read / write operations for each of the partition tree structures is prioritized based on the predicted workload pattern.

[0157] Aspect 8. The method according to any one of aspects 1 to 7, wherein the delay includes write stall, write stop and I / O amplification.

[0158] Aspect 9. The method according to any one of aspects 1 to 8, further comprising:

[0159] Concurrently manages thread pool execution, thread priorities, and task scheduling at the file system level.

[0160] Aspect 10. The method according to any one of aspects 1 to 9, wherein the management includes alleviating blocking I / O operations to facilitate parallelization, reduce latency, and increase response time.

[0161] Aspect 11. A method according to any one of Aspects 1 to 10, wherein the job scheduler is used to predict workload patterns and prioritize the scheduling of read / write operations for each partition tree structure in the partition tree structure based on access patterns.

[0162] Aspect 12. The method according to any one of aspects 1 to 11, further comprising:

[0163] Establish resource usage limits for different shards; and

[0164] Monitor real-time resource usage for each shard,

[0165] The monitoring is used to improve resource utilization.

[0166] Aspect 13. The method according to any one of aspects 1 to 12, further comprising:

[0167] Redundancy protection is provided by generating data redundancy blocks for critical file data by protecting metadata at the file system level.

[0168] Aspect 14. The method according to any one of aspects 1 to 13, further comprising:

[0169] outsource compression outside the file system, and

[0170] Allocate storage space by aligning the size of the upper layer file.

[0171] Aspect 15. A non-volatile storage device having an executable component stored thereon, comprising:

[0172] The shard manager is configured to:

[0173] splitting the data in a first log structure merge (LSM) tree structure into partition fragments to reduce the number of levels for the data represented in the first LSM tree structure,

[0174] wherein each partition shard represents an independent LSM tree structure, thereby providing scalability and flexibility for the data represented in the first LSM tree structure,

[0175] When the data amount of a corresponding partition fragment of the partition fragments reaches a threshold level, splitting the corresponding partition fragment of the partition fragments into at least a parent fragment and a child fragment; and

[0176] When the data amount of a corresponding partition fragment among the partition fragments decreases to an amount less than the threshold level, the corresponding partition fragment among the partition fragments is merged into an adjacent partition fragment among the partition fragments.

[0177] Aspect 16. The non-volatile storage device according to aspect 15, further comprising:

[0178] An input / output (I / O) classifier is used to classify and mark I / O to and from each of the partition shards based on type to facilitate prioritized scheduling between different types of I / O.

[0179] Aspect 17. The nonvolatile storage device according to aspect 15 or 16, further comprising:

[0180] A job scheduler is configured to predict workload patterns and prioritize scheduling of read / write operations for each of the partition shards to provide stable read amplification.

[0181] Aspect 18. The nonvolatile storage device according to any one of Aspects 15 to 17, further comprising:

[0182] An asynchronous API manager is configured to prioritize and execute asynchronous read requests for each of the partition fragments.

[0183] Aspect 19. The nonvolatile storage device according to any one of Aspects 15 to 18, further comprising:

[0184] A fault-tolerance manager is provided for recovering from a single sector corruption by utilizing error correction codes and redundant storage for each of the partition fragments.

[0185] Aspect 20. The nonvolatile storage device according to any one of Aspects 15 to 20, further comprising:

[0186] A multi-tenant manager is configured to allocate resource quotas between each of the partition shards based on workload demand and resource monitoring.

Claims

1. A method for optimizing a non-volatile storage device by performing operations, the operations comprising: Splitting the data in the first log-structure merge LSM tree structure into partition fragments to reduce the number of levels for the data represented in the first LSM tree structure, wherein each partition shard represents an independent LSM tree structure, thereby providing scalability and flexibility for the data represented in the first LSM tree structure, When the amount of data of a corresponding partition fragment among the partition fragments reaches a threshold level, splitting the corresponding partition fragment among the partition fragments into at least a parent fragment and a child fragment; as well as When the data amount of a corresponding partition fragment among the partition fragments decreases to an amount less than the threshold level, the corresponding partition fragment among the partition fragments is merged into an adjacent partition fragment among the partition fragments.

2. The method of claim 1, wherein the method further comprises directing new read and write requests to the sub-shard.

3. The method of claim 1, wherein the splitting comprises splitting an existing ordered string table (SST) between the parent shard and the child shard.

4. The method according to claim 1, wherein the fragment having the amount less than the threshold level is a following fragment, and the neighboring fragment into which the following fragment is merged is a leading fragment, Basic metadata from the following shards is added to the leading shard to provide access to data stored in the following shards, and New read and write requests are directed to the boot fragment.

5. The method according to claim 1, further comprising: I / O to and from each of the shards is categorized based on type.

6. The method of claim 1, wherein the classification comprises: Tag the corresponding I / O with labels to distinguish between foreground and background tasks, and Scheduling of the corresponding I / O of each of the shards is prioritized based on the corresponding tag.

7. The method according to claim 1, further comprising: Maintain predictable latency bounds for read and write operations by: predicting a workload pattern for each of the respective shards based on the access pattern, and Scheduling of read / write operations for each of the partition tree structures is prioritized based on the predicted workload pattern.

8. The method of claim 7, wherein the delays include write stalls, write stops, and I / O amplification.

9. The method according to claim 1, further comprising: Concurrently manages thread pool execution, thread priorities, and task scheduling at the file system level.

10. The method of claim 1, wherein the managing includes mitigating blocking I / O operations to facilitate parallelization, reduce latency, and increase response time.

11. The method according to claim 1, further comprising: Establish resource usage limits for different shards; as well as Monitor real-time resource usage for each shard, The monitoring is used to improve resource utilization.

12. The method of claim 1, wherein I / O scheduling includes implementation of multi-version concurrency control based on timestamps for each task.

13. The method according to claim 1, further comprising: Redundancy protection is provided by generating data redundancy blocks for critical file data by protecting metadata at the file system level.

14. The method according to claim 1, further comprising: outsource compression outside the file system, and Allocate storage space by aligning the size of the upper layer file.

15. A non-volatile storage device having an executable component stored thereon, comprising: The shard manager is configured to: Splitting the data in the first log-structure merge LSM tree structure into partition fragments to reduce the number of levels for the data represented in the first LSM tree structure, wherein each partition shard represents an independent LSM tree structure, thereby providing scalability and flexibility for the data represented in the first LSM tree structure, When the amount of data of a corresponding partition fragment among the partition fragments reaches a threshold level, splitting the corresponding partition fragment among the partition fragments into at least a parent fragment and a child fragment; as well as When the data amount of a corresponding partition fragment among the partition fragments decreases to an amount less than the threshold level, the corresponding partition fragment among the partition fragments is merged into an adjacent partition fragment among the partition fragments.

16. The nonvolatile memory device according to claim 15, further comprising: An input / output I / O classifier is used to classify and mark I / O to and from each of the partition fragments based on type to facilitate prioritized scheduling between different types of I / O.

17. The nonvolatile memory device according to claim 15, further comprising: A job scheduler is configured to predict workload patterns and prioritize scheduling of read / write operations for each of the partition shards to provide stable read amplification.

18. The nonvolatile memory device according to claim 15, further comprising: An asynchronous API manager is configured to prioritize and execute asynchronous read requests for each of the partition fragments.

19. The nonvolatile memory device according to claim 15, further comprising: A fault-tolerance manager is provided for recovering from a single sector corruption by utilizing error correction codes and redundant storage for each of the partition fragments.

20. The nonvolatile memory device according to claim 15, further comprising: A multi-tenant manager is configured to allocate resource quotas between each of the partition shards based on workload demand and resource monitoring.

Images