Method and key-value storage device for data storage

Abstract

A method for data storage and a key-value storage device are provided. The method includes: receiving a write request including a user key; determining that the user key exists in a cache; generating or updating metadata corresponding to the user key; writing data corresponding to the write request to the storage device; converting the metadata into a device format corresponding to the storage device; and storing the metadata in the storage device.

Description

[0001] This application claims the benefit and priority of U.S. Provisional Application No. 63 / 004,321, filed April 2, 2020, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This disclosure generally relates to a key-value storage architecture, and more specifically, to a key-value storage architecture for key-value devices. Background Technology

[0003] Key-value solid-state drives (KVSSDs) provide a key-value (KV) interface at the device level, resulting in improved performance and simplified storage management. This, in turn, enables high-performance scaling, simplified transformation processing (e.g., data conversion between object and block data), and expanded drive capabilities. By incorporating KV storage logic within the KVSSD's firmware, such KVSSDs can respond to direct data requests from applications while reducing the involvement of host software. Such KVSSDs can be further enhanced with SSD hardware, which can be boosted to provide processing power using Flash Translation Layer (FTL) software. Summary of the Invention

[0004] The embodiments described herein provide improvements in data storage.

[0005] According to some embodiments of this disclosure, a data storage method is provided, the method comprising: receiving a write request including a user key; determining that the user key exists in a cache; generating or updating metadata corresponding to the user key; writing data corresponding to the write request to a storage device; converting the metadata into a device format corresponding to the storage device; and storing the metadata on the storage device.

[0006] The steps to determine that a user key exists in the cache may include: determining that the metadata table to which the user key belongs exists; and determining that the metadata has a user key.

[0007] The steps of generating or updating metadata corresponding to a user key may include: generating metadata to replace metadata that does not exist in the cache; or updating metadata that exists in the cache.

[0008] The steps for generating metadata may include: sending a prefetch request to prefetch metadata; and adding the metadata to the cache.

[0009] The steps for updating metadata may include: creating a user key; inserting the user key into the key mapping of the metadata table; and adding the key information to the metadata table.

[0010] The steps of converting metadata into a device format corresponding to the storage device may include: processing requests on a pending request queue; and constructing the device format.

[0011] The method may further include: retrieving a metadata table from a cache; determining that metadata is missing from memory; loading the metadata table from a prefetch buffer or from a storage device; determining that a user key is found in the corresponding key mapping; adding the key information to a key list; and submitting the metadata to the storage device.

[0012] According to other embodiments of this disclosure, a key-value storage device is provided for storing data to a storage device, the key-value storage device being configured to: receive a write request including a user key; determine that the user key exists in a cache; generate or update metadata corresponding to the user key; write data corresponding to the write request to the storage device; convert the metadata into a device format corresponding to the storage device; and store the metadata on the storage device.

[0013] The KV storage device can be configured to determine that a user key exists in the cache by: determining that the metadata table to which the user key belongs exists; and determining that the metadata has a user key.

[0014] The KV storage device can be configured to generate or update metadata corresponding to user keys by: generating metadata to replace metadata that does not exist in the cache; or updating metadata that exists in the cache.

[0015] The KV storage device can be configured to generate or update metadata corresponding to user keys by: sending a prefetch request for prefetching metadata; and adding metadata to the cache.

[0016] The KV storage device can be configured to update metadata by: creating user keys; inserting user keys into the key mapping of the metadata table; and adding key information to the metadata table.

[0017] The KV storage device can be configured to convert metadata into a device format corresponding to the storage device by: processing requests on a pending request queue; and constructing a device format.

[0018] The KV storage device can also be configured to: retrieve the metadata table from the cache; determine that the metadata is missing from the memory; load the metadata table from the prefetch buffer or from the storage device; determine that the user key is found in the corresponding key mapping; add the key information to the key list; and submit the metadata to the storage device.

[0019] According to other embodiments of this disclosure, a non-transitory computer-readable medium implemented using a key-value storage device for storing data to a storage device is provided. The non-transitory computer-readable medium has computer code that, when executed on a processor, implements a method for storing data. The method includes: receiving a write request including a user key; determining that the user key exists in a cache; generating or updating metadata corresponding to the user key; writing data corresponding to the write request to the storage device; converting the metadata to a device format corresponding to the storage device; and storing the metadata on the storage device.

[0020] The steps to determine that a user key exists in the cache may include: determining that the metadata table to which the user key belongs exists; and determining that the metadata has a user key.

[0021] The steps of generating or updating metadata corresponding to a user key may include: generating metadata to replace metadata that does not exist in the cache; or updating metadata that exists in the cache.

[0022] The steps for generating metadata may include: sending a prefetch request to prefetch metadata; and adding the metadata to the cache.

[0023] The steps for updating metadata may include: creating a user key; inserting the user key into the key mapping of the metadata table; and adding the key information to the metadata table.

[0024] The steps of converting metadata into a device format corresponding to the storage device may include: processing requests on a pending request queue; and constructing the device format.

[0025] Therefore, embodiments of this disclosure improve data storage by providing a high-performance KV storage device architecture for KV devices that enable advanced features, efficient KV management, and reduced write amplification factor (WAF), while also omitting conventionally used file systems and conventionally used write-ahead logs (WAL). Attached Figure Description

[0026] Non-limiting and non-exhaustive embodiments of this example are described with reference to the accompanying drawings, wherein, unless otherwise specified, the same reference numerals denote the same parts throughout the various views.

[0027] Figure 1 Block diagrams depicting a first key-value (KV) store according to some embodiments of the present disclosure, a second KV storage device using a KV solid-state drive, and a KV storage device using a KV solid-state drive;

[0028] Figure 2A and Figure 2BThis is a flowchart depicting a method of data storage according to some embodiments of the present disclosure;

[0029] Figure 3 This is a block diagram depicting a computational-storage-based KV store (CSKVS) according to some embodiments of the present disclosure;

[0030] Figure 4 This is a block diagram depicting a device key format according to some embodiments of the present disclosure;

[0031] Figure 5 It is a block diagram in the form of a key table depicting an ordering according to some embodiments of the present disclosure;

[0032] Figure 6 This is a block diagram depicting the vBlock and kBlock formats according to some embodiments of the present disclosure;

[0033] Figure 7 This is a block diagram comparing a KV storage device with a compute-based KV storage device according to some embodiments of the present disclosure;

[0034] Figure 8 This is a block diagram depicting a mechanism for linking key blocks in a chain to ensure data consistency, according to some embodiments of this disclosure;

[0035] Figure 9 This is a block diagram depicting a mechanism for sorting key tables in a linked chain to ensure data consistency, according to some embodiments of this disclosure.

[0036] Figure 10 It is a block diagram depicting the relationship between KV memory objects, value blocks, key blocks, and key block groups according to some embodiments of the present disclosure;

[0037] Figure 11 This is a block diagram depicting a KV memory object, column node, and request node according to some embodiments of the present disclosure;

[0038] Figure 12 This is a flowchart depicting placement / deletion operations according to some embodiments of the present disclosure;

[0039] Figure 13 This is a block diagram depicting a key block format according to some embodiments of the present disclosure;

[0040] Figure 14 This is a block diagram depicting a key block group format according to some embodiments of the present disclosure;

[0041] Figure 15 It is a flowchart depicting a manifest format;

[0042] Figure 16 This is a block diagram depicting a key queue format according to some embodiments of the present disclosure;

[0043] Figure 17A and Figure 17B This is a block diagram depicting a key mapping structure according to some embodiments of the present disclosure and a key node format corresponding to the key mapping structure; and

[0044] Figure 18 This is a flowchart depicting different background threads and their relationships according to some embodiments of the present disclosure.

[0045] Throughout the various views of the accompanying drawings, corresponding reference numerals indicate the respective components. Those skilled in the art will understand that the elements in the drawings are shown for simplicity and clarity and are not necessarily drawn to scale. For example, the dimensions of some elements, layers, and areas in the drawings may be exaggerated relative to other elements, layers, and areas to aid clarity and understanding of the various embodiments. Furthermore, common but easily understood elements and components unrelated to the description of the embodiments may not be shown to facilitate less obstructive views of these various embodiments and to make the description clearer. Detailed Implementation

[0046] The features of the inventive concept and the methods for implementing it can be more readily understood by referring to the detailed embodiments and accompanying drawings. Hereinafter, embodiments will be described in more detail with reference to the accompanying drawings. However, the described embodiments can be implemented in various different forms and should not be construed as being limited to the embodiments shown herein. Rather, these embodiments are provided as examples so that this disclosure will be thorough and complete, and will fully convey to those skilled in the art aspects and features of the inventive concept. Therefore, processes, elements, and techniques not essential for a full understanding of the aspects and features of the inventive concept by those skilled in the art may not be described.

[0047] In the detailed description, numerous specific details are set forth for illustrative purposes to provide a comprehensive understanding of the various embodiments. However, it will be clear that the various embodiments may be practiced without these specific details or with one or more equivalent arrangements. In other instances, well-known structures and apparatuses are illustrated in block diagram form to avoid unnecessarily obscuring the various embodiments.

[0048] It will be understood that although the terms “first,” “second,” “third,” etc., may be used herein to describe various elements, components, regions, layers, and / or portions, these elements, components, regions, layers, and / or portions should not be limited by these terms. These terms are used only to distinguish one element, component, region, layer, or portion from another element, component, region, layer, or portion. Therefore, without departing from the spirit and scope of this disclosure, the first element, first component, first region, first layer, or first portion described below may be referred to as a second element, second component, second region, second layer, or second portion.

[0049] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising,” “having,” and “including” are used in this specification, they indicate the presence of the stated features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof. As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed items.

[0050] As used herein, the terms “substantially,” “about,” “approximately,” and similar terms are used as terms of approximation rather than terms of degree and are intended to take into account the inherent biases of measured or calculated values ​​that would be recognized by one of ordinary skill in the art. As used herein, “about” or “approximately” includes stated values ​​and indicates an acceptable range of deviation for a particular value as determined by one of ordinary skill in the art, taking into account the measurement in question and errors associated with the measurement of that particular quantity (i.e., limitations of the measurement system). For example, “about” may mean within one or more standard deviations, or within ±30%, ±20%, ±10%, or ±5% of the stated value. Furthermore, the use of “may” in describing embodiments of this disclosure indicates “one or more embodiments of this disclosure.”

[0051] When a particular embodiment can be implemented differently, the specific order of processing can be executed differently than the order in which they are described. For example, two consecutively described processes can be executed substantially simultaneously or in the reverse order of their description.

[0052] Electronic or electrical devices and / or any other related devices or components according to embodiments of the present disclosure described herein can be implemented using any suitable hardware, firmware (e.g., application-specific integrated circuits), software, or a combination of software, firmware, and hardware. For example, various components of such devices may be formed on a single integrated circuit (IC) chip or separate IC chips. Furthermore, various components of such devices may be implemented on a flexible printed circuit film, a tape-and-carrier package (TCP), a printed circuit board (PCB), or formed on a substrate.

[0053] Furthermore, the various components of these devices may be processes or threads that execute computer program instructions and interact with other system components to perform the various functions described herein, running on one or more processors in one or more computing devices. The computer program instructions are stored in memory, which may be implemented in the computing device using standard memory devices, such as random access memory (RAM). The computer program instructions may also be stored in other non-transitory computer-readable media, such as CD-ROMs, flash drives, etc. Moreover, those skilled in the art will recognize that, without departing from the spirit and scope of the embodiments of this disclosure, the functions of various computing devices may be combined or integrated into a single computing device, or the functions of a particular computing device may be distributed across one or more other computing devices.

[0054] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which the inventive concept pertains. It will also be understood that, unless clearly defined herein, terms (such as those defined in a general dictionary) shall be interpreted as having the same meaning as they have in the context of the relevant field and / or in this specification, and shall not be interpreted in an idealized or overly formalistic manner.

[0055] Key-value (KV) stores in this art may not support special operations (e.g., range queries, iterators / snapshots, prefix extractors, time-to-live (TTL) operations, column family transactions, etc.). Furthermore, KV stores in this art can use device input / output (I / O) operations to retrieve key information (e.g., information indicating key existence, value size checks, number of keys, etc.) from the storage device.

[0056] However, some embodiments of this disclosure provide a high-performance KV storage device architecture for KV devices (e.g., key-value solid-state drives (KVSSDs)) that enables advanced features, efficient KV management, and reduced write amplification factor (WAF) while omitting the file system, thereby eliminating the need for KV-to-file conversion while eliminating the WAL typically used for crash recovery. Therefore, the disclosed embodiments provide an improved KV storage device architecture, thereby improving KVSSD technology.

[0057] As will be described below, some embodiments of this disclosure provide a key-value storage device architecture for supporting key-value storage device operators (e.g., put, get, delete, iterators, etc.) used in conjunction with KVSSDs, while providing a WAL-less mechanism for reading, writing, and deleting operations using data paths. Furthermore, instead of using a file system, some embodiments of this disclosure use a metadata buffer for storage management that efficiently supports key iterator operations by involving multiple buffer writes within the data path. Additionally, instead of a write path involving two disk writes including WAL and data, some embodiments may involve only a single disk write.

[0058] Furthermore, some embodiments of the KV storage device can run on KVSSD and can support all the special database operations that are difficult for KVSSD to provide separately due to the associated overhead of existing databases (DBs). According to some embodiments of this disclosure, user key-to-block mapping management can be offloaded to one or more KVSSDs having computing units for managing user key-to-block mappings. However, because KVSSDs may have limited computing resources, they may not be able to provide all the special DB operations in other ways.

[0059] Therefore, some embodiments can identify "complex" operations (e.g., transactions, iterators, or recovery) and can process these operations on the host side while KVSSD manages simple key-block mappings. Some operations in embodiments compatible with KVSSD technology may correspond to metadata table management, place / delete / get / iterator procedures, transaction management, recovery, and cache management.

[0060] Figure 1 Includes respective block diagrams depicting a first KV storage device according to some embodiments of the present disclosure, a second KV storage device using a KV solid-state drive, and a KV storage device using a KV solid-state drive.

[0061] Reference Figure 1In the first KV storage device 110, user application 112 can use write-ahead log (WAL) 114 for write operations and can transfer KV pairs between file system 116 and user application 112 on application side 111. Furthermore, system software 117 corresponding to host 113a may include file system 116 and block device driver 118 to enable data transfer between user application 112 and storage device 119 (e.g., block device or KV device). Storage device 119 may be on device side 113b.

[0062] In a second KV storage device 120 using a KV device driver 128 for operation with KVSSD 129, a limitation may be that the second KV storage device 120 cannot support special operations (such as range queries, iterators / snapshots, prefix extractors, time-to-live (TTL) operations, column family transactions, etc.). Furthermore, device I / O operations can be used to retrieve key information (e.g., information from KVSSD 129 indicating key existence, value size checks, number of keys, etc.).

[0063] Unlike the first KV storage device 110, some embodiments of this disclosure include a KV storage device 130 based on KVSSD (e.g., hereinafter referred to as...). Figure 3 The computational-storage-based KV store (CSKVS), discussed in further detail, can omit the additional host sidekey management on host 133a. Furthermore, the KVSSD-based KV store 130 can omit the WAL 114 and file system 116 used in the first KV store 110.

[0064] Therefore, user application 132 can transfer data to and from KV storage device 139 (e.g., KVSSD) via one or more KV application programming interfaces (APIs) 135, which may result in a small or negligible amount of overhead. KV API 135 may be similar to KV API 125 of the second KV storage device 120. KV API 135 may be implemented on the application side 131 or as system software 137. For example, system software 137 may be executed by the processor of the KV storage device. Although some advanced operations (e.g., iterator (ordered) operations, range query operations, snapshot operations, transaction operations, large operations, value and key support, TTL operations, non-existent key filtering in the host, etc.) may not be supported, the KVSSD-based KV storage device 130 can use KV API 135 to provide advanced functionality.

[0065] The KVSSD-based KV storage device 130 can also use the KV device driver 138 to drive the KVSSD 139. Therefore, after a user sends a command via the user application 132, the command can be translated into a device command. That is, the KVSSD-based KV storage device 130 of this disclosure can manage KV data and can convert KV data from KV format to a file format corresponding to the user application 132. The KVSSD-based KV storage device 130 can then send the KV data to the KVSSD 139. Therefore, embodiments of this disclosure provide improved performance while utilizing the KV API 135 to support the KVSSD 139.

[0066] The in-memory database (InSDB) 134 of the KVSSD-based KV storage device 130 can represent a relatively high-performance, memory-space-efficient embedded database for storing KV data. The InSDB 134 according to embodiments of this disclosure can be a KV database compatible with one or more databases in the art and can provide an improved framework for efficiently utilizing the KVSSD 139 for I / O-bound workloads. The InSDB 134 can be stored and executed on the host side 133a of the KVSSD-based KV storage device 130 (e.g., as computer code) and can include a KV cache while performing key and cache management.

[0067] Therefore, InSDB 134 can support special operations and features on the host side 133a (e.g., iterator operations, range query operations, snapshot operations, comparator operations, transaction operations, TTL operations, key existence confirmation operations, column families, data compression, cyclic redundancy check (CRC), etc.). Furthermore, InSDB 134 allows user application 132 to create requests for communication with KVSSD 139 (e.g., place, retrieve, delete, etc.), and KVSSD 139 can store key-value pairs and support basic key-value operations created by user application 132.

[0068] InSDB 134's key management operations manage valid keys in host memory. Some disclosed embodiments of InSDB 134 reduce KV management overhead (e.g., get value size operations, key presence operations, key count, free space management, etc.). InSDB 134 also supports special operations with relatively low overhead (e.g., range query operations, iterators / snapshots, prefix extractors, TTL, column families, etc.). InSDB 134's key management can also support transactions even without device support. Some features of key management may include avoiding the need to keep keys in memory by maintaining hotkeys, reducing or minimizing key table rewrites by dividing key ranges into multiple key tables, using a limited key table size, and updating occasionally changed key tables. These features can be performed in the background to reduce or minimize the impact on application performance.

[0069] Figure 2A and Figure 2B This is a flowchart depicting a method for data storage according to some embodiments of the present disclosure.

[0070] Reference Figure 2A and Figure 2B The KV storage device according to some embodiments of this disclosure can be operated as follows. As a first operation, at 210, a user application according to an embodiment of this disclosure (e.g., Figure 1 User application 132) can submit KV requests (e.g., place / write requests) with user keys. That is, users can send KV requests to KV storage devices (e.g., Figure 1 KV storage device 130 or based on KVSSD Figure 3 The compute-based storage key-value storage device (CSKVS) 330. For example, the key-value storage device can receive write requests that include user keys.

[0071] As a second operation, at 220, the KV storage device according to embodiments of this disclosure may perform foreground operations. For example, at 221, the KV storage device may begin foreground operations. The KV storage device may then check to determine whether a user key exists in memory (e.g., whether the user key for a place / write request exists in a cache). This can be achieved by determining at 222 whether the corresponding metadata table (to which the user key belongs) exists, and subsequently at 223 whether the metadata actually includes the user key.

[0072] However, the metadata corresponding to the user key may not exist in the cache. In this case, the metadata can be prefetched from the device for that key in a background operation. That is, if the metadata corresponding to the user key is not in the cache, a dummy (or empty) metadata table can be inserted into the cache and populated when the actual metadata is ready in memory. This can be done before the user key is created. After the user key is created, the dummy metadata table can be inserted into the key mapping of the metadata table.

[0073] In one example, the KV storage device can generate or update metadata corresponding to user keys. For instance, the KV storage device can generate metadata or update metadata that exists in the cache in response to determining that the metadata does not exist in the cache. For example, if it is determined at 222 that the metadata does not exist in memory, then at 224, the system can send a prefetch request to prefetch the metadata. Then, at 225, the system can add the metadata table to the table cache.

[0074] In step 226, the system can create a user key and insert it into the key mapping of the metadata table. This can happen even if it is determined in step 223 that the metadata does not have a user key, because the metadata table exists somewhere in the cache.

[0075] Subsequently (or conversely, if it is determined at 222 that the metadata is in memory, and also at 223 that a given user key already exists in the metadata), then at 227, key information and a list of keys can be added, and at 228, the key value can be inserted into the pending request queue. Thus, the value can be added to the end of the list of values ​​for the user key, and the system can terminate the foreground work at 229.

[0076] As a third operation, at 230, regardless of whether the metadata has been prefetched or found and subsequently updated, the data corresponding to the place / write request can be written to the device in the background. That is, user key-value pairs can be written to the device in chunks. If a threshold is reached, everything can be written to the device. This can be achieved by processing requests on the pending request queue. The metadata can then be converted to a device-specific format and subsequently stored in a KV device (e.g., ...). Figure 1 On KVSSD 139). That is, the system can construct device formats and submit key-value pairs to KV devices. Note that the following regarding Figure 18 A more detailed description of the write thread.

[0077] For example, at 231, the KV storage device can begin a KV commit operation. At 232, the KV storage device can retrieve a request from the pending request queue. Then, at 233, the KV storage device can construct the device format. At 234, the KV storage device can commit the KV to the KV device. Then, at 235, the KV storage device can end the KV commit operation.

[0078] As a fourth operation, at 240, the metadata table can be flushed (e.g., the metadata table can be submitted to a KV device (such as KVSSD 139)). That is, old values ​​of the metadata can be deleted during background operations. Note that the following regarding... Figure 18 A more detailed description of the SKTable flushing thread.

[0079] For example, at 241, the KV storage device can begin the metadata flushing process. At 242, the metadata table can be retrieved from the table cache. At 243, it can be determined whether the metadata exists in memory. If the metadata is determined to exist in memory, the metadata can be flushed at 244. Conversely, if the metadata is not found in memory at 243, the metadata table can be loaded from the prefetch buffer or from the KV device at 245. Then, at 246, if the same key is found in the key mapping table, the old key information can be added to the end of the key list, and the metadata can then be flushed at 244. The KV storage device can then end the metadata flushing process at 247.

[0080] Figure 3 This is a block diagram depicting a computationally based storage KV storage device (CSKVS) according to some embodiments of the present disclosure.

[0081] Reference Figure 3 Some embodiments of this disclosure provide a CSKVS 330, which implements (e.g., offloading to KVSSD 339) KV functionality to the storage device. Offloading reduces communication overhead between the host and the device (e.g., between the host KV storage device (HKVS) 340 and KVSSD 339) and simplifies the storage software stack, among other advantages.

[0082] CSKVS 330 may include HKVS 340 and KVSSD 339. CSKVS 330 may correspond to... Figure 1 The KV storage device 130 is based on KVSSD. Furthermore, KVSSD 339 can correspond to... Figure 1 The KVSSD139. Additionally, the HKVS 340 can be matched with... Figure 1 InSDB 134 and KV API 135.

[0083] HKVS 340 can manage user keys 352 to support “complex” operations (e.g., iterator operations, transaction operations, column family operations, etc.). KVSSD 339 can manage KV hash table 360 ​​and may include a “garbage collector” 370 for performing garbage collection operations (e.g., for identifying unusable data, invalid data, or low-usefulness data for evicting or deleting from KVSSD 139).

[0084] In HKVS 340, user keys 352 can be categorized by one or more corresponding user key ranges to which they belong, and each user key range can be managed by a sorted key table (SKTable) manager 344. When a user applies 332 (e.g., Figure 1 When user application 132) accesses user key 352 belonging to SKTable 341, HKVS 340 can load SKTable 341. HKVS 340 can prefetch SKTable 341 according to a prefetching strategy. To determine which SKTable 341 will be prefetched and loaded, HKVS 340 can maintain information corresponding to the range of user keys and each key 352 of SKTable 341. The information maintained by HKVS 340 can be stored in a manifest (hereinafter referred to as...). Figure 15 (See the list for further details).

[0085] The HKVS 340 provides a variety of application programming interfaces (APIs) (e.g., place, get, and iterator APIs 315, 325, and 335). APIs 315, 325, and 335 can correspond to... Figure 1 The KV API 135. The placement API 335 can support operations such as store, delete, and update, and can internally create KV memory objects (KVMO) 351 (e.g., objects in temporary memory), and can insert KVMO 351 into three data structures (e.g., into a pending request queue (PRQ) 342, an in-memory hash table or an in-memory global hash (e.g., an in-memory hash table) 343, and a key queue (e.g., a key queue) 345 of unsorted keys 352 existing in each SKTable 341).

[0086] The first data structure can be PRQ 342. PRQ 342 can deliver KVMO 351 to write threads 346. Each write thread 346 (e.g., write threads WT0, WT1, ..., WTn) can retrieve KVMO 351 from PRQ 342 and can create value blocks (vBlock 349) and key blocks (kBlock 350) management 347 to submit the corresponding requests to KVSSD 339.

[0087] The second data structure can be an in-memory hash table 343, which may have KVMO 351 in a key mapping (e.g., key mapping 348) that has not yet been inserted into SKTable 341. When key 352 exists in in-memory hash table 343, HKVS 340 can provide key 352 from the corresponding hash. Keys inserted into key mapping 348 of SKTable 341 can be removed from in-memory hash table 343.

[0088] The third data structure can be a key queue 345 of SKTable 341. That is, SKTable 341 can include a key queue 345 with unsorted keys 352 and a key map 348 with sorted keys 352. The key queue 345 can be used to hold keys 352 before they are inserted into the key map 348. Keys in the key queue 345 can be inserted into the key map 348 by a flush thread or by iterator operations. The key map 348 and the key queue 345 are discussed further below.

[0089] The Get API 325 supports value retrieval operations. Get API 325 checks if key 352 exists in the hash table 343 in memory; if it exists, the value is read from the hash and then provided. Otherwise, Get API 325 can find key 352 in key map 348 of SKTable 341. If key map 348 has not yet been retrieved, it can be read from KVSSD 339.

[0090] Key map 348 may have an internal key (iKey) for vBlock 349 and may have an offset index for key 352. When key 352 exists in key map 348, HKVS 340 can use iKey to look up vBlock 349 in the cache. When vBlock 349 does not exist in the cache, vBlock 349 can be read from KVSSD 339. The value corresponding to user key 352 can then be found in vBlock 349 using the offset index in key map 348.

[0091] Iterator API 315 supports "seek", "previous", "next", and "value" operations. For this purpose, SKTable 341 can be sorted based on a start key (e.g., a start key or a recovery start key), and the keys 352 in each SKTable 341 can also be managed in a sorted manner. Therefore, Iterator API 315 can find the SKTable 341 containing a given key 352, and subsequently find the key 352 in key map 348. When a key 352 belonging to Iterator API 315 is in key queue 345, Iterator API 315 can insert the key 352 from key queue 345 into key map 348 before accessing key map 348. When vBlock 349 has not yet been fetched, Iterator API 315 can prefetch the next vBlock 349. Iterator API 315 can send a prefetch request to the device driver used to drive KVSSD 339 (e.g., ...). Figure 1 The KV device driver 138 can perform a read-ahead operation to read the requested vBlock 349 and can hold vBlock 349 in a read-ahead buffer in the device driver.

[0092] Unlike the KVSSD 129 of the second KV storage device 120, the KVSSD 339 according to some embodiments of this disclosure can receive KV pairs directly from the host (e.g., directly from the HKVS 340). In the KVSSD 339, the key 352 can be managed based on the KV hash table 360, and the user application 332 can read the user value via the corresponding key 352. However, in some embodiments, a KV pair can be written atomically. That is, the overhead for ensuring data consistency can be reduced or minimized by writing the value and key 352 atomically.

[0093] User values ​​can include vBlock 349. Therefore, garbage collection can be performed when some values ​​are invalid. However, when garbage collection is run on the host, data transfer between the host and the device may be involved. Therefore, garbage collection can be performed on the device (e.g., by the garbage collector 370 of the KVSSD 339). The vBlock 349 used to perform garbage collection can be selected on the HKVS 340 and requested from the KVSSD 339.

[0094] Figure 4 This is a block diagram depicting a device key format according to some embodiments of the present disclosure.

[0095] Reference Figure 4Device key 410 may include sixteen bytes. The higher four bytes of device key 410 may indicate database identifier (DB ID) 411, which may be a 4-byte hash value of the database name. The next single byte of device key 410 may indicate key type 412.

[0096] The lower eight bytes of device key 410 (e.g., lower block 417) can be configured differently depending on the corresponding key type 412. In vBlock 349 or kBlock 350, the lower block 417 of device key 410 can be filled with iKey 413. Furthermore, vBlock 349 and kBlock 350 can use the same iKey 413. kBlock groups can be created for every 127 kBlocks 350. In embodiments using SKTable 341, the lower block 417 may include a 4-byte SKTable identifier (SKTID) 414 and may also include a four-byte key table sequence number 415. The remaining three bytes of device key 410 may be reserved (RSVD) 416.

[0097] Figure 5 This is a block diagram depicting a key table format of sorting according to some embodiments of the present disclosure.

[0098] Reference Figure 5 KV storage devices (e.g., Figure 3 The CSKVS 330 supports a variety of commands that can be grouped into "simple" or "complex" commands. A factor that can be used to determine whether a command is classified as simple or complex can be whether the command has relatively high hardware resource utilization. For example, key sorting operations can be considered complex commands with high hardware utilization because they have high CPU and memory utilization. As discussed here, place, delete, and get commands can be classified as simple commands, while iterator (range scan) commands, transaction commands, and snapshot commands can be classified as complex commands.

[0099] Regarding place and / or retrieve commands in some key-value storage devices, key-value pairs can be converted to a file format and subsequently written to or read from the storage device via a file system (as used herein, storage refers to computational storage, where computation utilizes storage devices such as storage devices, e.g., ...). Figure 3 (This is supported by CSKVS 330 and KVSSD 339). In contrast, iterator commands can use their own context to execute iterator commands, which may include the serial number, current position, etc.

[0100] Another example is transaction commands (e.g., write batch transactions), which can also consume relatively large amounts of memory resources. When using compute storage, multiple key-value pairs can be written atomically to support transactions. However, when there are more than hundreds of thousands of key-value pairs in a transaction, maintaining a log in the storage device for commit and rollback commands / operations can be relatively cumbersome, as the log stores the changes performed as a result of the transaction commands.

[0101] Commit and rollback commands can be two transaction statements used to execute or undo a transaction, respectively. A transaction operation can have a sequence of query or update statements that can be used to modify the database. Therefore, when a transaction operation is executed successfully, the commit command makes the corresponding database modifications made by the transaction operation permanent. Conversely, when a transaction operation fails to execute successfully, the rollback command undoes all updates from the first statement of the transaction.

[0102] According to some embodiments of this disclosure, storage (e.g., in) is possible. Figure 3 All of the above commands are executed in the CSKVS 330 (KVSSD 339). However, problems may arise when supporting complex commands in storage. For example, there may be overhead when maintaining sorted data structures (such as trees and skip lists, or key mappings 348 such as SKTable 341). In addition, there may be direct overhead, computational overhead, and overhead associated with maintaining the data structures.

[0103] User applications (e.g., Figure 3 The user application (332) can provide its own comparator for performing sorting operations, and the corresponding values ​​can be sorted by the comparator. To find the key 352 in a sorted data structure (e.g., key map 348), one or more intermediate keys can be loaded and compared by the comparator. Therefore, insertion, deletion, and search operations can utilize relatively large amounts of CPU and memory resources.

[0104] Furthermore, the execution time for completing operations in compute storage (e.g., KVSSD 339) can be longer than the corresponding execution time on the host system because the storage may employ an embedded processor, which is typically less efficient than the host system's (e.g., Figure 3 The HKVS 340's host processor is powerful and typically has less storage space than the host system.

[0105] Other issues may relate to indirect overhead and latency caused by “atomic” execution. Even if the computing power within the storage device is similar to that of the host system, the interaction between the host and the storage device (e.g., between the HKVS 340 and KVSSD 339) can cause relatively significant bottlenecks. For example, when a sorted data structure (e.g., SKTable 341) is only held in storage, all iterator commands (e.g., next command, prey command, and find command) can be submitted to the device. This can result in non-negligible submission overhead, and user processes may be blocked until the device completes sending and is interrupted.

[0106] Iterator commands can be determined to be complex commands. Each iterator can have its own context, which may include a sequence number, current position, etc. The current position can be in a sorted data structure (e.g., in...). Figure 5 The key in the key mapping 348 is shifted forward or backward. When the user executes the next or previous command on the iterator, the next or previous key can be found in the sorted data structure. When key 352 is found, the sequence number and time can be compared to determine if the key belongs to the iterator. If key 352 does not belong to the iterator, the search for another key 352 can continue until the located key 352 meets the conditions.

[0107] Therefore, some embodiments of this disclosure may support simple commands only in the KVSSD 339 of the CSKVS 330, which efficiently limits the use of computational storage. However, complex commands can be processed efficiently by making efficient use of the underlying storage. For example, some complex commands can be supported by managing user keys 352, and key tables (e.g., SKTable 341) can be used for this purpose.

[0108] Each SKTable 341 can manage keys 352 within a corresponding key range (e.g., a CSKVS may have multiple SKTables 341 for managing keys 352 within a corresponding key range, respectively). Because the host system (e.g., HKVS 340) may not be able to retain all keys 352 in host memory, the entire key range can be divided into multiple SKTables 341. Each key node 353 in an SKTable 341 may have key attributes (e.g., column, sequence number, creation time, etc.). In a CSKVS 330 according to some embodiments of this disclosure, the entire key range can be divided when only a subset of SKTables 341 is in memory, allowing multiple SKTables 341 to be retained, thereby making more efficient use of the host system's memory.

[0109] Therefore, the CSKVS 330 can have multiple SKTables 341, and some commands (e.g., get, place, delete, and exist commands) can be offloaded to the KVSSD 339, while some commands (e.g., iterator (range scan), transaction, and snapshot commands, as well as other optional operations such as time-to-live (TTL)) can be provided on the HKVS 340. In other words, many operations that the host system can support can be implemented by managing user keys 352 using SKTables 341.

[0110] Figure 6 This is a block diagram depicting the vBlock and kBlock formats according to some embodiments of the present disclosure.

[0111] Reference Figure 6 Some embodiments of this disclosure can reduce the interaction between the host and storage (e.g., Figure 3 Interaction between HKVS 340 and KVSSD 339 of CSKVS 330. For example, it can be managed using keys (e.g., Figure 3 The SKTable Manager 344 is used to reduce interaction with the KVSSD 339.

[0112] As described above, some commands can be processed through HKVS 340 to avoid interaction with KVSSD 339. For example, HKVS 340 can check if a key exists before submitting a command to KVSSD 339. When key 352 is not managed on HKVS 340, all get and delete commands can be submitted to KVSSD 339 even if the key does not exist on the system. When key 352 and attributes are managed through HKVS 340, operations related to existing commands and TTL commands do not involve interaction with KVSSD 339. Furthermore, when key 352 is sorted and managed through HKVS 340, operations related to determining the key count or size within a specific key range do not involve device interaction because HKVS 340 does not need to submit commands to KVSSD 339.

[0113] Some key-value (KV) storage devices (e.g., HKVS 340) can merge KV pairs to reduce the number of I / O operations. For example, some KV storage devices can use a "memtable" (e.g., ...). Figure 3The memory table (343) is used to collect key-value pairs in memory. Then, when the size of the memory table exceeds a reference size (e.g., a maximum size), the memory table can be converted to a file format and written to the device. Therefore, multiple key-value pairs can be written to the device in a single operation. Even though the CSKVS 330 can receive key-value pairs directly from the user application 332, a similar mechanism can be used for computational storage according to some embodiments of this disclosure to reduce the number of interactions with the KVSSD 339.

[0114] Therefore, embodiments of this disclosure provide vBlock and kBlock formats for value packing and key packing, respectively. vBlock 349 includes a value 612 and an offset 614 of the value 612.

[0115] The key node 353 in SKTable 341 may have two pieces of information to retrieve the value 612. The first piece of information may be the iKey 413 of vBlock 349. iKey 413 may be assigned to HKVS 340. The second piece of information may be an index of the value offset (e.g., directly relative to the offset). Therefore, key node 353 does not need to be updated when the offset changes after any garbage collection operation.

[0116] kBlock 350 may include key 352 and key attributes 616 (e.g., TTL, column ID, transaction information, key size, etc.). kBlock 350 can be deleted when all keys 352 in kBlock 350 are recorded in the corresponding SKTable 341. Because kBlock 350 is separate from vBlock 349, kBlock 350 can be deleted without deleting vBlock 349.

[0117] Conversely, when a single block is used for both key 352 and value 612, that single block can be updated when all keys 352 are updated in kBlock 350. The update could involve, for example, reading the entire value 612, as well as writing value 612 back to KVSSD 339. Therefore, CSKVS 330 can store kBlock 350 in KVSSD 339 to enable data recovery operations after a crash.

[0118] Figure 7 This refers to a KV storage device (e.g., according to some embodiments of this disclosure) Figure 1 A block diagram comparing the first KV storage device with CSKVS.

[0119] Reference Figure 7Some embodiments of this disclosure can reduce offloading management overhead. For example, CSKVS 330 can offload garbage collection to KVSSD 339. CSKVS 330 may include vBlock 349 for encapsulating KV pairs. However, some KV pairs may be invalid, while others remain valid. Therefore, when vBlock 349 has both valid and invalid values, garbage collection operations can be performed on vBlock 349 to mark invalid values ​​for eviction. However, garbage collection can use read-modify-write (RMW) operations.

[0120] When performing garbage collection operations on the host side, it can be done from the device (e.g., Figure 1 The storage device 119, which may be a solid-state drive (SSD), reads the entire vBlock 349. The vBlock 349 can then be modified, and the modified vBlock 349 can be written back to the device, thereby causing additional I / O.

[0121] In contrast, in the CSKVS 330 system of the embodiments of this disclosure, garbage collection can be performed in the KVSSD 339 without any data transformation between the host and the KVSSD 339.

[0122] Furthermore, when some key-value pairs are deleted from vBlock 349, the offsets of the remaining key-value pairs can be changed, thereby ensuring a corresponding change in the position of the values ​​in SKTable 341. However, according to some embodiments of this disclosure, the vBlock 349 format can be used (e.g., refer to...). Figure 6 This is to avoid updating SKTable 341.

[0123] Key nodes 353 in SKTable 341 may have indexes with value offsets, rather than directly having value offsets. Although the index offsets can be modified accordingly, the number of indexes remains unchanged during garbage collection. Therefore, key nodes 353 in SKTable 341 do not need to be updated. In some embodiments, garbage collection may be performed using a field-programmable gate array (FPGA).

[0124] Additionally, the CSKVS 330 can offload block management. For example, the file system 116 can be a key component of a modern computer system. The primary function of the file system 116 can be to manage block space and maintain the mapping between files and blocks. Therefore, KV storage devices (e.g., Figure 1 The first KV storage device 110) can convert KV pairs into files so that KV pairs can be written to block devices and can write KV pairs to files.

[0125] In addition, the file system 116 can create metadata (such as "inodes") for files. Furthermore, metadata and data that are the content of the files can be written to the storage device 119 via the block layer 117 and the block device driver 118. The storage device 119 may have a mapping table 719 for translating logical block addresses (LBAs) to physical block addresses (PBAs).

[0126] However, according to some embodiments of this disclosure, the CSKVS 330 can bypass the file system 116 and block layer 117 when storing KV pairs. That is, the CSKVS 330 can omit the operation of converting KV pairs to files, and can also omit the operation of converting LBAs to PBAs. In addition, the CSKVS 330 can avoid creating metadata in the file system 116. Instead of conversion, the CSKVS 330 may include a hash table 739 for managing key and value PBA mappings.

[0127] Figure 8 This is a block diagram depicting a mechanism for ensuring data consistency using kBlocks in a linked chain according to some embodiments of the present disclosure.

[0128] Reference Figure 8 WAL (for example, Figure 1 WAL 114 can be used to secure KV storage devices (e.g., Figure 1 Data consistency in the first KV storage device 110. However, WAL 114 can cause redundant write operations (e.g., double writes), which can double I / O traffic. Furthermore, when the file system (e.g., Figure 1 When the file system 116 uses a log to ensure file system consistency, the key-value pairs are writable devices (e.g., Figure 1 The storage device 119) is accessed four times. Therefore, not only is overall performance sacrificed to ensure data consistency, but the WAF is also increased, thereby reducing the lifespan of the corresponding SSD. Therefore, some embodiments of this disclosure can provide a consistency model that omits WAL 114 (e.g., data consistency can be ensured without using WAL).

[0129] To eliminate redundant write operations, some embodiments of this disclosure provide a method for ensuring data consistency by using the characteristics of a KVSSD 339 with compute storage to link kBlock 350 in a kBlock chain. The KVSSD 339 can store individual KV pairs atomically. Therefore, by linking kBlock 350 to each other, all kBlock 350 not recorded in SKTable 341 can be traversed during recovery operations. kBlock 350 can be created on the host side and can be committed to the KVSSD 339.

[0130] According to some embodiments of this disclosure, transaction rollback operations can be supported without in-place updates in KVSSD 339. Therefore, all write, update, and delete requests can be added as new kBlock 350s to the end of the kBlock chain 850.

[0131] To create kBlock chain 850, the new kBlock 350 can use a pre-assigned iKey 855, which is stored in the previous kBlock as the iKey of the previous kBlock. Then, the new kBlock 350 can have a new pre-assigned iKey 854 pre-assigned for the subsequent / next kBlock 350, and the new iKey 854 can be stored in kBlock 350.

[0132] Simultaneously, it may be useful to instruct the first / starting kBlock 853 to enable the recovery operation. Therefore, the recovery start iKey 852 of the start kBlock 853 can be recorded in the KVSSD 339. The recovery start iKey 852 can be updated when all keys 352 of the start kBlock 853 are stored in the corresponding SKTable 341. The recovery start iKey 852 can be updated after all "dirty" SKTables 341 (e.g., SKTables 341 that cannot be confirmed to contain valid data) have been updated, thus avoiding frequent updates to the recovery start iKey 852.

[0133] When kBlock 350 is within the recovery range from the start of kBlock 853 to the end of kBlock 856 in the recovery of the kBlock chain 850, kBlock 350 is protected from deletion even if any valid key 352 is missing, thus ensuring that the kBlock chain 850 is maintained. In other words, according to some embodiments of this disclosure, the system can delete only kBlock 350 outside the recovery range.

[0134] Figure 9 This is a block diagram depicting a mechanism for sorting an SKTable in a linked chain to ensure data consistency, according to some embodiments of this disclosure.

[0135] Reference Figure 9The SKTable chaining mechanism can be used to ensure data consistency. As described above, SKTable 341 can manage the corresponding user keys 352 within the key range associated with SKTable 341. Therefore, SKTable 341 can be updated in the device when a user key 352 is added to or deleted from the key range.

[0136] When the size of SKTable 341 exceeds the reference size (e.g., the maximum SKTable size) after adding user key 352, SKTable 341 can be split into two or more SKTables 341 (e.g., into a main SKTable 942 and one or more child SKTables 943). Therefore, child SKTable 943 can be inserted into SKTable chain 950.

[0137] like Figure 9 As shown, operations 901, 902, and 903 depict a method for inserting an SKTable 941 into an SKTable chain 950. Each SKTable 941 may have two additional keys 352. One key may be the next SKTable key 953, and the other key may be a pre-assigned SKTable key 954, which can be used as a key for the first child SKTable.

[0138] Therefore, if a crash occurs during the splitting of SKTable 341, the pre-assigned SKTable key 954 can be used to find the child SKTable 943. The next SKTable key 953 of the last SKTable in the SKTable chain 950 can be the same as the key of the original SKTable, thus completing the SKTable chain 950.

[0139] When all child SKTable 943 are successfully stored in the device (e.g., Figure 3 In KVSSD 339, the original SKTable can be updated with a reduced range of corresponding keys. When replacing the next SKTable key 953 with a pre-assigned SKTable key 954, or when assigning a new SKTable key to a pre-assigned SKTable key 954, the original SKTable key 954 does not need to be changed. Therefore, if a data corruption or power failure occurs before updating the original SKTable 341, all child SKTables 943 can be deleted, and the operation for splitting SKTables can begin from the initial steps. The newly added key can be found in the kBlock chain 850.

[0140] Conversely, when one or more SKTables 341 have no key 352 or have a small number of keys 352, SKTable 341 can be merged into the previous SKTable 341. For example... Figure 9 As shown, operations 903, 904, and 905 illustrate methods for merging and deleting SKTable 341 from SKTable chain 950. Figure 9 In the example shown, SKTable "SKT 10" 910 and SKTable "SKT 4" 914 are merged into SKTable "SKT 3" 913. "SKT 3" 913 can be updated before deleting "SKT 10" 910 and "SKT 4" 914 to avoid corrupting kBlock chain 850.

[0141] The next SKTable key 953 of “SKT 3” 913 can be updated to the next SKTable key 953 of “SKT 4”, where SKTable key 953 can correspond to the last SKTable key in the SKTable chain 950. Then, the pre-assigned SKTable key 954 of “SKT 3” 913 can be the SKTable key of “SKT 10” 910, which can be the next SKTable key of “SKT 3” 913. Therefore, in the event of a crash during a merge operation, “SKT 10” 910 and “SKT 3” 913 can be found from the pre-assigned SKTable key 954, and “SKT 10” 910 and “SKT 3” 913 can be deleted during a recovery operation. To maintain the link between SKTables 341 that are candidates to be deleted, the deletion operations should be performed in reverse order, so that the earlier SKTable 341 is deleted last. Therefore, in this example, “SKT 4”914 can be deleted first, and then “SKT 10”910 can be deleted.

[0142] Figure 10 This is a block diagram depicting the relationship between KVMO, vBlock, kBlock, and kBlock group according to some embodiments of the present disclosure.

[0143] Reference Figure 10 CSKVS 330 can be implemented using objects for managing key-value pairs. As described below, different objects can be used to manage key-value pairs and the correspondence between them.

[0144] The first object can be one or more KVMOs 351. KVMOs 351 can be created in the user environment. During a write operation of a write thread, a KVMO 351 can be packed together with vBlock 349 and kBlock 350. vBlock 349 can pack the value 612, and kBlock 350 can pack the key 352 and other attributes. When the transaction is committed, kBlock 350 can be deleted, the key information can be recorded in SKTable 341, and it can be merged into a kBlock group 354, which includes the number of keys 352 and the valid key map (hereinafter referred to as...). Figure 14 (Further detailed key mapping information will be provided).

[0145] Figure 11 This is a block diagram depicting KVMO, column nodes, and request nodes according to some embodiments of the present disclosure. Figure 12 This is a flowchart depicting placement / deletion operations according to some embodiments of the present disclosure.

[0146] Reference Figure 11 KVMO 351, column node (CN) 1151, and request node (RN) 1152 are memory objects used to submit requests to KVSSD 339. A KVMO 351 can be created for each key 352, and a CN 1151 can be created for each user request. In other words, a KVMO 351 can have multiple columns, and each column can have multiple CNs 1151. Furthermore, an RN 1152 can be created for common columns in key 352 that have the same transaction group ID (TGID), which is found in transaction information 1153. Additionally, a request node (RN) 1152 can have multiple column nodes (CNs) 1151, with only the last CN 1151 being submitted to KVSSD 339. Therefore, RN 1152 can have a "last CN pointer" pointing to the last CN 1151.

[0147] Reference Figure 12 When a user application sends a place or delete command at 1201, at 1202, the HKVS 340 can first determine whether the KVMO 351 is in the hash.

[0148] If KVMO 351 is found to exist at 1202, KVMO 351 can be used for user requests. Otherwise, at 1203, a new KVMO 351 can be created, and at 1204, KVMO 351 can be inserted into hash and SKTable 341.

[0149] Then, at 1205, a new CN 1151 can be created and inserted into KVMO 351. Then, at 1206, it can be determined whether the previous CN 1151 exists, and at 1207, it can be determined whether the previous CN 1151 has the same TGID, which can be calculated by right-shifting the transaction ID. The number of bits used for shifting can be configurable. Transactions can be grouped to allow in-place updates. When it is determined at 1207 that the previous CN 1151 has the same TGID, the new CN 1151 can be inserted into RN 1152 at 1208 to merge with the previous CN 1151. The merge can be achieved by changing the last CN 1151 pointer in RN 1152 to the new CN 1151.

[0150] Otherwise, at 1209, a new RN 1152 can be created, and at 1210, the new RN 1152 is inserted into the pending request queue (e.g., Figure 3 In PRQ 342).

[0151] Figure 13 This is a block diagram depicting the kBlock format according to some embodiments of the present disclosure.

[0152] Reference Figure 13 As mentioned earlier, in the process of packing KV pairs, user key 352 and value 612 can be packed into kBlock 350 and vBlock 349, respectively. During KV pair packing, the write thread can have multiple RNs 1152 from PRQ 342. Within each RN 1152, the most recent CN 1151 is available for vBlock 349. The remaining CNs 1151 are considered to be updated in-place. However, when the remaining CNs 1151 are part of a transaction, transaction information used for commit and rollback operations can be recorded in kBlock 350.

[0153] After submitting vBlock 349, the iKey of vBlock 349 (e.g., Figure 4 The size of iKey 413 and vBlock 349, as well as the index of the value offset in vBlock 349, are stored in CN 1151 to record information to SKTable 341. The user value pointer in CN 1151 can also be replaced by the vBlock 349 pointer, thereby releasing the user value.

[0154] kBlock 350 can store value attributes (e.g., user key, column ID, sequence number, transaction information, etc.). This information can be used for recovery processing. However, vBlock 349 may not include any information for deletion requests. Therefore, only kBlock 350 can have information about deletion requests.

[0155] Each block of column information can have multiple columns updated by the last CN before commit. When the merged columns are part of a transaction, transaction information can be stored in kBlock 350 for recovery. If a crash occurs before the write to kBlock 350 is complete, all transactions belonging to kBlock 350 can be considered uncommitted transactions.

[0156] Figure 14 This is a block diagram depicting a kBlock group format according to some embodiments of the present disclosure.

[0157] Reference Figure 14 After all keys 352 belonging to kBlock 350 have been recorded in the corresponding SKTable 341, kBlock 350 can be deleted. However, it can be useful to keep bitmap 1410 as an indicator of which values ​​in vBlock 349 are valid or invalid.

[0158] However, bitmap 1410 can use only eight bytes per kBlock 350. Therefore, 127 kBlocks 350 can be merged into kBlock group 354 to reduce the number of I / O operations. kBlock group 354 can have an 8-byte header 1412 for determining whether kBlock group 354 can be deleted.

[0159] Each kBlock 350 entry can use eight bytes for both the raw key count 1414 and the valid key bitmap 1410. The raw number of keys 352 can be used to trigger garbage collection. For example, garbage collection can be triggered using the keys of vBlock 349 and bitmap 1410 when the number of valid keys 352 differs from the raw number of keys 352 indicated by the raw key count 1414. For example, garbage collection can be performed when half of the values ​​in vBlock 349 are invalid.

[0160] Figure 15 It is a flowchart depicting a list format.

[0161] Reference Figure 15 Key 352 can be managed using SKTable 341 and a manifest. As mentioned above, SKTable 341 can be a key table used to manage the corresponding key range, where user key 352 is stored in sorted form. HKVS 340 can manage SKTable 341 with skip tables. When the database is closed, SKTable metadata, including key 352 of SKTable 341 and the starting user key (e.g., recovery start iKey 852), can be stored in the manifest. When the database is open, the SKTable metadata can be read from the manifest to construct the SKTable skip tables.

[0162] In this example, fields 3, 4, and 5 represent unused or not yet used numbers. For example, field 4 could represent an iKey that has not yet been used for vBlock 349 and kBlock 350. That is, a new pre-assigned iKey (e.g., Figure 8 The new pre-assigned iKey 854 can begin at the number recorded in field 4.

[0163] In field 7, iKeys can be reserved for subsequent iKeys. That is, after the database is opened, these values ​​can be stored in the vBlock chain information, and in the event of a crash, recovery can begin from the corresponding iKey. Therefore, a list of up to one billion keys can be created, with a size of 2MB or less.

[0164] Figure 16 This is a block diagram depicting a key queue format according to some embodiments of the present disclosure.

[0165] Reference Figure 16 The SKTable 341 may include three main parts: a key queue 345, a key map 348, and a DeltaTable. The key queue 345 may receive new keys from the user in an unsorted manner. The key map 348 may retrieve keys from the key queue 345 and may store keys 352 in a sorted manner. Furthermore, the DeltaTable may store only new keys 352 to avoid redundant writes.

[0166] Key queue 345 may have KVMO 351, which is not stored in SKTable 341 in chronological order (e.g., in key map 348). In this example, key queue 345 may include an active key queue 1646 and a deactivated key queue 1647. When SKTable 341 is flushed (e.g., written to a device and subsequently deleted from memory), operations may cause the active key queue 1646 and the deactivated key queue 1647 to be swapped, and KVMO 351 may be recorded in the deactivated key queue 1647 in key map 348.

[0167] Then, key map 348 may temporarily hold pointers corresponding to KVMOs, rather than key information for uncommitted requests and uncommitted transactions. These keys 352 can be managed separately in SKTable 341, and when the next SKTable 341 is written to the device and deleted, keys 352 can be processed first before being retrieved from key queue 345. When key queue 345 has keys 352 belonging to an iterator, the iterator can flush key queue 345 to key map 348 because iterator operations can be performed using sorted keys 352.

[0168] Figure 17A This is a block diagram depicting a key mapping structure according to some embodiments of the present disclosure.

[0169] Figure 17A The skip list and the corresponding key-mapped memory block are shown. Key map 348 manages keys 352 with skip lists. Each key node 353 can be allocated within a corresponding memory block to store key map 348 (skip list) to KVSSD 339 without translation.

[0170] To point to the next key node 353, each key node 353 can have an offset relative to the base address, rather than having the address of the next key node 353. Therefore, the position of the next key node 353 can be calculated as the sum of the next offset 1753 and the base address. Thus, the skip table does not need to rely on virtual addresses. In other words, even when different virtual addresses are allocated during the reloading of the key mapping 348 from the KVSSD 339, the address of the next node in the skip table can be dynamically calculated.

[0171] On the other hand, in order to store the key mapping 348 "as is" into the KVSSD 339, each key node 353 of the skip table can reside in contiguous virtual memory. Therefore, memory for the key nodes 353 can be allocated within a memory block that can be up to 2 MB in size. When storing the key mapping 348 into the KVSSD 339, the memory block can be stored between the base address and the end of the memory used. When the size of the memory used exceeds the default size, the size of the memory block can be adjusted through a reallocation operation, or a new SKTable 341 can be created while splitting the key mapping 348.

[0172] When key mapping 348 is split, key nodes 353 can be sequentially inserted into the new memory block, and invalid key nodes 353 can be removed. Because key mapping 348 allows in-situ updates, invalid key nodes 353 can be created solely due to deletion operations. Furthermore, invalid key nodes 353 in the memory block can be reused for new key nodes 353.

[0173] Additionally, when all keys 352 in key mapping 348 are deleted, keys 352 can be merged into the previous key mapping 348. Merging operations can be performed between adjacent SKTables 341, and for the efficiency of SKTable 341 management, keys 352 can be merged even when SKTable 341 is not empty.

[0174] Figure 17BThis is a block diagram depicting the format of key nodes corresponding to a key mapping structure according to some embodiments of the present disclosure. Each key node 353 may include an offset of the next key node 353 in the skip list, user key information, and column information. The column information may include vBlock information. In the user key information, the "relocation" field and KVMO can be referred to the following.

[0175] Regarding the repositioning of fields, when updating key node 353, if the size of the existing key node 353 is smaller than the newly requested memory size, new memory can be allocated for column information, and the offset can be recorded at the repositioned location.

[0176] Regarding the KVMO field, when vBlock 349 has not yet been committed to KVSSD 339, or when the transaction to which the column belongs has not been committed, a pointer to KVMO 351 can be stored instead of the column insertion information. Therefore, key node 353 does not need to be inserted into the DeltaTable.

[0177] Regarding DeltaTable, whenever a new key 352 is inserted into key map 348, duplicate information can be repeatedly written to KVSSD 339 if the entire key map 348 can be stored in KVSSD 339. For example, when key map 348 has a size of 512KB, even if only some bytes of key map 348 are changed, 512KB of data should be written to KVSSD 339. In this case, not only will there be an increase in I / O traffic, but the WAF (Web Application Firewall) may also increase.

[0178] Therefore, a DeltaTable can be used to avoid rewriting any key nodes 353 that have not been updated. The DeltaTable can be a new set of key nodes 353, where vBlock 349 can be committed and transactions can be completed. That is, when a key node 353 is inserted into the key map 348 in memory, the key node 353 can also be inserted into the DeltaTable.

[0179] Subsequently, the DeltaTable, instead of the key mapping 348, can be stored in the KVSSD 339. The DeltaTable's ID may include the SKTable ID and the DeltaTable ID (e.g., a serial number). When the SKTable 341 is evicted or when the DB is shut down, the modified key mapping 348 can be stored in the KVSSD 339.

[0180] After storing key mapping 348, a DeltaTable can be deleted from KVSSD 339. The ID range of the deleted DeltaTable can be stored in the metadata area of ​​key mapping 348 to ensure recovery operations after a crash. The next DeltaTable ID can start from the next sequence number after the last DeltaTable ID.

[0181] Figure 18 This is a flowchart depicting different background threads and their relationships according to some embodiments of the present disclosure.

[0182] Reference Figure 18 The HKVS 340 can have four types of background threads. For example, background threads may include worker thread 1860, SKTable flush thread 1870, vBMG update thread 1880, and eviction thread 1890. Each type of background thread can have a unique role and can be linked via data structures. For example, operations 1801 through 1819 can indicate the processing flow for place / delete commands.

[0183] Regarding worker threads (e.g., write threads) 1860, the HKVS 340 may have multiple write threads. Each write thread can package multiple KVMO 351s into vBlock 349 and kBlock 350 to reduce the total number of device commits to the KVSSD 339, and the KVMO 351s can be written to the KVSSD 339 in operations 1801 and 1802.

[0184] To avoid rewriting the entire vBlock 349 to record updated information, metadata related to key 352 can be recorded separately in kBlock 350. The write thread can obtain the iKey from the iKey generator. vBlock 349 and kBlock 350 can share the iKey and can be accessed via key type fields (e.g., ...). Figure 4 The key type 412 fields are used for differentiation. When iKey is reserved for kBlock group 354, kBlock group 354 can be created using iKey, and kBlock group 354 can be inserted into hash table 1851 of kBlock group 354.

[0185] Subsequently, another iKey can be retrieved from the iKey generator. When vBlock 349 and kBlock 350 are created, vBlock 349 and kBlock 350 can be inserted into vBlock hash 1852 and kBlock chain 850 respectively, and can be submitted to the device in operations 1803, 1804 and 1805.

[0186] Then, the corresponding kBlock group 354 can be obtained from kBlock group hash 1851, and the corresponding kBlock group 354 can be updated.

[0187] Then, in operations 1806, 1804, and 1808, the user value of the relevant KVMO351 can be replaced using vBlock 349. On the other hand, when kBlock 350 is committed simultaneously from various write threads, the order of writing kBlock 350 may not be guaranteed, and the kBlock chain 850 may be corrupted. Therefore, a kBlock chain 850 can be created for each write thread.

[0188] Regarding SKTable flushing thread 1870 (e.g., a flushing thread where SKTable 341 is written to KVSSD 339 and subsequently deleted), SKTable flushing thread 1870 is responsible for recording DeltaTable to KVSSD 339 whenever key mapping 348 changes. In this example, key 352 in key queue 345 can be inserted into key mapping 348, and DeltaTable can be created in operations 1809 and 1810.

[0189] As described above, in order to reduce the overhead associated with rewriting the entire key map 348 when the number of newly inserted keys 352 is small, only the newly inserted keys 352 are collected separately in DeltaTable, which can then be stored in KVSSD 339 in operation 1811.

[0190] When updating key mapping 348, the kBlock group 354 of key 352 recorded in key mapping 348 can also be updated in operations 1812 and 1813.

[0191] For example, SKTable flush thread 1870 can clear the bitmap of the kBlock group 354 to which the old key 352 belongs (e.g., Figure 14 The corresponding bit in bitmap 1410. For the new key 352, the KVMO count of the kBlock that may only exist in memory can be reduced. When the KVMO count reaches zero, the kBlock group 354 can be submitted to KVSSD 339, and the kBlock 350 belonging to the kBlock group 354 can be deleted. Then, in operation 1814, the kBlock group 354 can be inserted into the kBlock group update queue 1853 to transmit the updated kBlock group 354 to the update thread 1880.

[0192] When SKTable refresh thread 1870 updates one of the kBlock groups 354 in multiple SKTables 341, kBlock group 354 can be inserted multiple times, and the last request can be the only request processed in kBlock update thread 1880.

[0193] Regarding the vBMG update thread (e.g., the kBlock update thread) 1880, the kBlock update thread 1880 can write or rewrite the updated kBlock group 354 to the KVSSD 339. The kBlock update thread 1880 can also update the start kBlock information of the kBlock chain 850 and can delete kBlock 350 outside the chain. For example, the kBlock update thread 1880 can first retrieve the kBlock group 354 from the kBlock group update queue 1853 in operation 1816.

[0194] When all keys 352 in kBlock 350 have been recorded in the corresponding SKTable 341, kBlock 350 can be submitted to KVSSD 339 in operation 1817. Then, the kBlock chain 850 can be updated accordingly. When the scope of kBlock chain 850 changes, kBlock 350 outside the kBlock chain (e.g., off-chain kBlock) can be deleted from KVSSD 339 in operations 1818 and 1819.

[0195] Before deleting kBlock 350, the start key information of the updated kBlock chain 850 (e.g., the restored start key 852) and the key 352 of kBlock 350 can be recorded in the kBlock chain 850 record to prevent damage caused by a crash, thereby ensuring data consistency.

[0196] Meanwhile, key 352 in kBlock group 354 can belong to different SKTable 341, and multiple keys 352 can be updated within a given time period. Therefore, kBlock update thread 1880 can delay updating kBlock group 354 to avoid multiple rewrites to kBlock group 354. However, the length of kBlock chain 850 can be long when kBlock group update operations take too long, or when kBlock group update operations may be considered late. Therefore, regular maintenance can be potentially guaranteed.

[0197] Regarding eviction thread 1890, when cache memory is insufficient, eviction thread 1890 can evict vBlock 349, kBlock group 354, and key map 348. Each object can be managed using a Least Recently Used (LRU) policy and can be evicted according to priority. That is, initially vBlock 349 or kBlock group 354 can be evicted. However, if there is still insufficient memory even after evicting vBlock 349 and / or kBlock group, key map 348 can be evicted. Therefore, key information can be evicted conservatively. When key map 348, which can be an eviction candidate, has been updated, the thread can write key map 348 to KVSSD 339.

[0198] It should be noted that the metadata table format may be as follows. As described above, the metadata table may reside in memory. A key mapping table may exist for each metadata table, and the corresponding key mapping table may be for updated keys. The key mapping table may include a tree or skip table, which includes multiple keys, each key corresponding to a list including key information, which may include value information, a serial number, and user key-to-device key mapping information. However, when no key is updated, the key mapping table may be empty. Furthermore, when the metadata table has not yet been retrieved, the metadata does not have a device format table. The metadata table device format according to embodiments of this disclosure may be constructed from a conventional metadata table device format and a key mapping table, thereby being submitted to a storage device.

[0199] Therefore, embodiments of this disclosure provide a high-performance KV storage architecture for KV devices that enables advanced features, efficient KV management, and WAF reduction, while omitting the file system, thereby eliminating the need for KV-to-file conversion, and also omitting the WAL conventionally used for crash recovery.

[0200] Although embodiments of the present disclosure have been specifically shown and described with reference to the accompanying drawings, the specific terminology used herein is for descriptive purposes only and is not intended to limit the meaning of the terminology or the scope of the invention as set forth in the claims. Therefore, those skilled in the art will understand that various modifications and other equivalent embodiments of the present disclosure are possible. Consequently, the true scope of protection of the present disclosure must be determined based on the technical spirit of the appended claims, and functional equivalents of the invention will be included therein.

Claims

1. A method for data storage, the method comprising: Receive write requests including user keys; Determine if the metadata corresponding to the user key exists in the cache; When it is determined that the metadata does not exist in the cache, the metadata is generated. Once it is determined that the metadata exists in the cache, determine whether the metadata has a user key; When it is determined that the metadata does not have a user key, update the metadata; When it is determined that the metadata has a user key, the key information is added to the metadata table; Write the data corresponding to the write request to the storage device; Convert metadata into a device format corresponding to the storage device; and Storing metadata on a storage device The step of writing the data corresponding to the write request to the storage device includes: inserting the user key value into the pending request queue.

2. The method according to claim 1, wherein, The steps for generating metadata include: Send a prefetch request for prefetching metadata; and Add metadata to the cache.

3. The method according to claim 1, wherein, The steps to update metadata include: Create user keys; and Insert the user key into the key mapping of the metadata table.

4. The method according to claim 1, wherein, The steps of converting metadata into a device format corresponding to the storage device include: Process requests in the pending request queue; Constructing device formats; and Submit the user's key-value pair to the storage device.

5. The method according to any one of claims 1 to 4, further comprising: Retrieve the metadata table from the cache; It was determined that metadata was missing from memory; Load the metadata table from the prefetch buffer or from storage; Ensure that the user key is found in the corresponding key mapping; Add key information to the key list; and Submit metadata to the storage device.

6. A key-value storage device, comprising: Storage device; and The processor is configured as follows: Receive write requests including user keys; Determine if the metadata corresponding to the user key exists in the cache; When it is determined that the metadata does not exist in the cache, the metadata is generated. Once it is determined that the metadata exists in the cache, determine whether the metadata has a user key; When it is determined that the metadata does not have a user key, update the metadata; When it is determined that the metadata has a user key, the key information is added to the metadata table; Write the data corresponding to the write request to the storage device; Convert metadata into a device format corresponding to the storage device; and Storing metadata on a storage device The step of writing the data corresponding to the write request to the storage device includes: inserting the user key value into the pending request queue.

7. The key-value storage device according to claim 6, wherein, The processor is configured to generate metadata by: Send a prefetch request for prefetching metadata; and Add metadata to the cache.

8. The key-value storage device according to claim 6, wherein, The processor is configured to update metadata by: Create user keys; and Insert the user key into the key mapping of the metadata table.

9. The key-value storage device according to claim 6, wherein, The processor is configured to convert metadata into a device format corresponding to the storage device by performing the following operations: Process requests in the pending request queue; Construct device format; and Submit the user's key-value pair to the storage device.

10. The key-value storage device according to any one of claims 6 to 9, wherein, The processor is also configured as follows: Retrieve the metadata table from the cache; It was determined that metadata was missing from memory; Load the metadata table from the prefetch buffer or from storage; Ensure that the user key is found in the corresponding key mapping; Add key information to the key list; and Submit metadata to the storage device.

11. A non-transitory computer-readable medium implemented using a key-value storage device for storing data into a storage device, the non-transitory computer-readable medium having a computer program that, when executed on a processor, implements a method for storing data, the method comprising: Receive write requests including user keys; Determine if the metadata corresponding to the user key exists in the cache; When it is determined that the metadata does not exist in the cache, the metadata is generated. Once it is determined that the metadata exists in the cache, determine whether the metadata has a user key; When it is determined that the metadata does not have a user key, update the metadata; When it is determined that the metadata has a user key, the key information is added to the metadata table; Write the data corresponding to the write request to the storage device; Convert metadata into a device format corresponding to the storage device; and Storing metadata on a storage device The step of writing the data corresponding to the write request to the storage device includes: inserting the user key value into the pending request queue.

12. The non-transitory computer-readable medium according to claim 11, wherein, The steps for generating metadata include: Send a prefetch request for prefetching metadata; and Add metadata to the cache.

13. The non-transitory computer-readable medium according to claim 11, wherein, The steps to update metadata include: Create user keys; and Insert the user key into the key mapping of the metadata table.

14. The non-transitory computer-readable medium according to claim 11, wherein, The steps of converting metadata into a device format corresponding to the storage device include: Process requests in the pending request queue; Constructing device formats; and Submit the user's key-value pair to the storage device.

Images