Method, system, and non-transitory computer-readable medium for searching a target key

Abstract

A method, system, and non-transitory computer-readable medium for searching a target key are provided, the method comprising: populating a hash offset table of a sorted key table with hash offset table entries, the hash offset table entries having hash values ​​and hash offsets corresponding to corresponding keys; sorting the hash offset table entries based on the hash values; searching the hash offset table for a target hash value corresponding to the target key among the hash values; locating a target key-value pair corresponding to the target key based on the target hash value; and saving the location of the target key-value pair.

Description

Technical Field

[0001] One or more aspects of embodiments of the present disclosure generally relate to methods, systems, and non-transitory computer-readable media for searching a database for a target key. Background Art

[0002] Searching and retrieving random keys in a table in a key-value (KV) store can result in significant overhead. This overhead is often caused by searches that utilize multiple key comparisons. Consequently, some key search techniques often use a significant number of CPU cycles associated with the various key comparisons. Therefore, it would be useful to define the table's data structure more efficiently and effectively. Summary of the Invention

[0003] Embodiments described herein provide a method for searching a target key in a database to improve data storage by mapping a hash offset table to a key-value table.

[0004] According to one embodiment of the present disclosure, a method for searching for a target key in a database is provided, the method comprising: filling a hash offset table of a sort key table with hash offset table entries, the hash offset table entries having a hash offset and a hash value corresponding to a corresponding key; sorting the hash offset table entries based on the hash value; searching the hash offset table for a target hash value corresponding to the target key in the hash value; locating a target key-value pair corresponding to the target key based on the target hash value; and saving the location of the target key-value pair.

[0005] Searching for the target hash value may include performing a binary search.

[0006] The method may further include calculating a target hash value from the target key.

[0007] The operation of saving the location of the target key-value pair may include mapping the target hash value to the target key-value pair.

[0008] The operation of locating the target key-value pair may include locating the target key-value pair based on a hash offset.

[0009] The sort key table may further include a key-value table, the key-value table includes a plurality of key-value table entries, and the key-value table entries include target key-value pairs.

[0010] The number of key-value table entries can be the same as the number of hash offset table entries.

[0011] According to another embodiment of the present disclosure, a system for searching for a target key in a database is provided, the system comprising an on-memory data structure, a storage format data structure, and a storage device, wherein the system is configured to: fill a hash offset table of a sort key table with hash offset table entries in the storage format data structure, the hash offset table entries having a hash offset and a hash value corresponding to a corresponding key; sort the hash offset table entries based on the hash value according to the on-memory sorting structure of the on-memory data structure; search the on-memory data structure or the storage format data structure to obtain a target hash value corresponding to the target key in the hash value in the hash offset table; locate a target key-value pair corresponding to the target key based on the target hash value in the on-memory data structure or the storage format data structure; and save the location of the target key-value pair to the on-memory data structure or the storage format data structure.

[0012] The system may also be configured to search for the target hash value by performing a binary search.

[0013] The system may also be configured to calculate a target hash value from the target key.

[0014] The system may also be configured to save the location of the target key-value pair by mapping the target hash value to the target key-value pair.

[0015] The system may also be configured to locate a target key-value pair based on a hash offset.

[0016] The sort key table may further include a key-value table, the key-value table includes a plurality of key-value table entries, and the key-value table entries include target key-value pairs.

[0017] According to another embodiment of the present disclosure, a non-transitory computer-readable medium is provided, which is implemented on a system for searching for a target key in a database. The non-transitory computer-readable medium has computer code that, when executed on a processor, implements a method for data storage, the method comprising: filling a hash offset table of a sorted key table with hash offset table entries, the hash offset table entries having hash offsets and hash values ​​corresponding to corresponding keys; sorting the hash offset table entries based on the hash values; searching the hash offset table for a target hash value corresponding to the target key in the hash values; locating a target key-value pair corresponding to the target key based on the target hash value; and saving the location of the target key-value pair.

[0018] When the computer code is executed by the processor, the computer code may further implement the data storage method by searching for the target hash value by performing a binary search.

[0019] When the computer code is executed by the processor, the computer code may further implement the data storage method by calculating a target hash value from a target key.

[0020] The operation of saving the location of the target key-value pair may include mapping the target hash value to the target key-value pair.

[0021] The operation of locating the target key-value pair may include locating the target key-value pair based on a hash offset.

[0022] The sort key table may further include a key-value table, the key-value table includes a plurality of key-value table entries, and the key-value table entries include target key-value pairs.

[0023] The number of key-value table entries can be the same as the number of hash offset table entries.

[0024] Therefore, the system of the embodiments of the present disclosure can improve data storage by providing a system and method for improving search and retrieval of data by mapping entries of a hash offset table to entries of a key-value table. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Non-limiting and non-exhaustive embodiments of the present invention are described with reference to the figures, wherein like reference numerals refer to like parts throughout the various views unless otherwise specified.

[0026] Figure 1 is a block diagram depicting an example of a search hierarchy for a sorted string table;

[0027] Figure 2 is a block diagram depicting a method for searching key-value pairs according to an embodiment of the present disclosure;

[0028] Figure 3 is a flowchart depicting a method of searching for key-value pairs according to an embodiment of the present disclosure; and

[0029] Figure 4 is a block diagram depicting a system for searching key-value pairs according to an embodiment of the present disclosure.

[0030] Corresponding reference numerals indicate corresponding components throughout the several views of the accompanying drawings. It will be understood by those skilled in the art that the elements in the drawings are shown for simplicity and clarity and are not necessarily drawn to scale. For example, the dimensions of some elements, layers, and regions in the drawings may be exaggerated relative to other elements, layers, and regions to help improve the clarity and understanding of the various embodiments. In addition, common but well-known elements and components that are not relevant to the description of the embodiments may not be shown to facilitate less obstructed viewing of these various embodiments and to make the description clear. DETAILED DESCRIPTION

[0031] By referring to the detailed description and the accompanying drawings of the embodiments, the features of the inventive concept and the methods for realizing the inventive concept can be more easily understood. Hereinafter, the embodiments will be described in more detail with reference to the accompanying drawings. However, the embodiments described can be implemented in various forms and should not be construed as being limited to the embodiments shown herein. On the contrary, these embodiments are provided as examples so that this disclosure will be thorough and complete, and will fully convey the aspects and features of the inventive concept to those skilled in the art. Therefore, processes (processes), elements (elements) and technologies that are unnecessary for a complete understanding of the aspects and features of the inventive concept by those of ordinary skill in the art may not be described.

[0032] In the detailed description, for purposes of explanation, numerous specific details are set forth to provide a thorough understanding of the various embodiments. However, it is apparent that the various embodiments can be practiced without these specific details or with one or more equivalent arrangements. In other cases, well-known structures or devices are shown in block diagram form to avoid unnecessarily obscuring the various embodiments.

[0033] It will be understood that although the terms "first," "second," "third," etc. may be used herein to describe various elements, components, regions, layers, and / or parts, these elements, components, regions, layers, and / or parts should not be limited by these terms. These terms are used to distinguish one element, component, region, layer, or part from another element, component, region, layer, or part. Therefore, a first element, component, region, layer, or part described below may be referred to as a second element, component, region, layer, or part without departing from the spirit and scope of the present disclosure.

[0034] The terms used herein are only for the purpose of describing specific embodiments and are not intended to be limitations of the present disclosure. As used herein, unless the context clearly indicates otherwise, the singular forms "one" and "a (species / person)" are also intended to include plural forms. It will also be understood that when the terms "comprises", "having" and "comprising" and variations thereof are used in this specification, the features, integral bodies, steps, operations, elements and / or components stated are present, but the presence or addition of one or more other features, integral bodies, steps, operations, elements, components and / or their groups are not excluded. As used herein, the term "and / or" includes any combination and all combinations of one or more of the relevant listed items.

[0035] As used herein, the terms "substantially," "about," "approximately," and similar terms are used as terms of approximation and not as terms of degree, and are intended to account for the inherent deviations in measured or calculated values ​​that would be recognized by one of ordinary skill in the art. As used herein, "approximately" or "approximately" are inclusive of the stated value and mean within an acceptable range of deviation for the particular value as determined by one of ordinary skill in the art taking into account the measurement in question and the errors associated with the measurement of the particular quantity (i.e., the limitations of the measurement system). For example, "approximately" can mean within one or more standard deviations, or within ±30%, 20%, 10%, or 5% of the stated value. Additionally, when describing embodiments of the present disclosure, the use of "may" means "one or more embodiments of the present disclosure."

[0036] When a certain embodiment can be implemented differently, a specific processing order may be performed in a different order than described. For example, two consecutively described processes may be performed substantially simultaneously, or two consecutively described processes may be performed in an order opposite to the described order.

[0037] The electronic devices or electrical devices according to the embodiments of the present disclosure described herein and / or any other related devices or components can be implemented using any suitable hardware, firmware (e.g., an application specific integrated circuit), software, or a combination of software, firmware, and hardware. For example, the various components of these devices can be formed on an integrated circuit (IC) chip or on separate IC chips. In addition, the various components of these devices can be implemented on a flexible printed circuit film, a tape carrier package (TCP), or a printed circuit board (PCB), or can be formed on a substrate.

[0038] In addition, the various components of these devices can be processes or threads that run on one or more processors in one or more computing devices, execute computer program instructions, and interact with other systems to perform the various functions described herein. Computer program instructions are stored in a memory that can be implemented in a computing device using a standard memory device (such as, for example, random access memory (RAM)). Computer program instructions can also be stored in other non-transitory computer-readable media (such as, for example, a CD-ROM, a flash drive, etc.). In addition, it should be appreciated by those skilled in the art that, without departing from the spirit and scope of the embodiments of the present disclosure, the functions of various computing devices can be combined or integrated into a single computing device, or the functions of a particular computing device can be distributed on one or more other computing devices.

[0039] Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which the present inventive concept belongs. It will also be understood that, unless expressly defined as such herein, terms (such as those defined in common dictionaries) should be interpreted as having a meaning consistent with their meaning in the relevant art and / or the context of this specification, and should not be interpreted in an idealized or overly formal sense.

[0040] Embodiments of the present disclosure can reduce or eliminate the need for search hierarchies and / or algorithms in an internal key hash directory for performing searches of key-value (KV) data, for example, in a key-value solid-state drive (KVSSD), such as a KVRock. Thus, the disclosed embodiments can reduce the number of CPU cycles of a processor of the KVSSD used to locate a random key in a given table stored in the KVSSD.

[0041] The internal key hash directory of the disclosed embodiment includes various entries in the hash offset table that have both the hash value and the offset of the corresponding key. Therefore, the KV table entries of the internal key hash directory can be sorted based on their respective hash values. Therefore, when searching for a random key, because each key has been converted to a corresponding hash value, and because the size of the hash value will be smaller than the size of the key to which it corresponds, the disclosed embodiment can reduce overhead by searching the internal key hash directory for the hash value corresponding to the target key being sought.

[0042] Figure 1 is a block diagram depicting an example of a search hierarchy for a sorted string table.

[0043] Reference Figure 1 , the sorted string table (SST) 110 can be a primary on-storage data format for storing key information. In addition, some log-structured KV databases (such as RocksDB developed by LevelDB (developed by [the company's]) includes a relatively complex search hierarchy or search algorithm within the SST 110 in the key-value store. The search hierarchy can be thought of as the overall sequence of operations followed when searching for data corresponding to a target key so that the data can be returned to the application requesting the search. It should be noted that a single-key search in the SST 110 can be a path in random access mode.

[0044] For example, to search for and locate a KV pair 180, some search algorithms, such as those used in a LevelDB table structure including an index block 120 and a data block 130, may first locate an index chunk 140 handle from the index block 120. The index chunk 140 handle may then be used to locate the index chunk 140. The index chunk 140 may be located by comparing the relevant key with the offset table 150 (e.g., by performing a binary search). That is, some search algorithms may search for the index chunk 140 by performing a binary search using the offset table 150.

[0045] Once the index chunk 140 is located, the search algorithm can then similarly locate the data chunk handle 160 included in the index chunk 140 by performing a scan operation. The data chunk handle 160 includes an offset and a size, thereby indicating the location of the associated data chunk 130.

[0046] The search algorithm may then load the relevant data chunk 130 by using the data chunk handle 160 .

[0047] Furthermore, the search algorithm can seek to locate a data chunk 170 (e.g., corresponding to a target key “abc”) in the data chunk 130 and having the sought target KV data / KV pair 180 by performing a binary search using the offset table 190 in the data chunk 130. Thereafter, the search algorithm performs a scan operation to scan the KV pair 180, thereby locating the KV pair 180 in the data chunk 170.

[0048] Therefore, the above structure performs at least two binary searches and two scans using string comparison operations to obtain the target KV pair 180. Therefore, it is advantageous to provide a format that supports a faster single query.

[0049] Figure 2 is a block diagram depicting a method of searching for KV pairs according to an embodiment of the present disclosure.

[0050] Reference Figure 2 , an example method according to an embodiment of the present disclosure may be generally performed as follows.

[0051] First, the system of an embodiment of the present disclosure can access a table (e.g., a sort key table) 210 of a seek key or target key including a KV pair (e.g., a target KV pair) 280. The KV pair 280 corresponds to a key for identifying stored data and corresponds to a value or object including the data. As further described below, the seek key / target key can be a key searched by the system during a key search. The table 210 can include a KV table 211 and a hash offset table 212 (e.g., an internal key hash directory). The various keys (e.g., target keys) 214 of the KV pairs 280 of the database can be divided into one or more corresponding KV tables 211, each of which can correspond to a corresponding hash offset table 212 so that the keys 214 of the (one or more) KV tables 211 can be located during the search.

[0052] Thereafter, the system can perform a search for a given target key 214. In this example, the system searches for the key "abc". Therefore, the system can first calculate a target hash value 216 by hashing the target key 214 (e.g., by determining a hash value (e.g., target hash value) 216 corresponding to the target key 214 according to the equation "target hash value = Hash("abc")"). For example, the hash value 216 can be calculated from a string of the key 214. Therefore, the number of separate instances of hash information can be the same as the number of keys 214 corresponding thereto. For example, the sort key table can fill the hash offset table 212 with the same number of hash offset table entries 250 as the number of KV table entries 270 in the KV table 211, and then after calculating the hash value 216, the system can sort the hash offset table 212 by the hash value 216.

[0053] In addition, an offset to the actual location of the corresponding KV pair 280 can be included for each hash value 216 (e.g., the system can create a corresponding hash value 216 by using each key 214.) Thus, when the system constructs metadata, the hash value 216 can be saved to a location in the KVSSD, and the KV table entry 270 can be mapped to the hash offset table entry 250.

[0054] The system can then perform a binary search to search for the target hash value 216 within the hash offset table 212. The hash offset table entries 250 of the hash offset table 212 can be sorted based on their respective hash values ​​216, and each hash offset 218 for locating a corresponding KV pair 280 in a corresponding KV table entry 270 is located in the hash offset table 212.

[0055] After locating the hash offset 218 corresponding to the KV pair 280 of the target key 214, the system retrieves the KV pair 280 from the hash offset 218 in the buffer. Thus, if a random target key 214 (e.g., the key "abc") is sought, the system can then calculate the hash value corresponding to the target key 214 (e.g., according to the equation "hash value = Hash ("abc")") to determine the hash number / hash value 216, and can then search for the hash value 216 in the hash offset table 212 by performing a binary search. In this case, the system has already obtained the hash offset 218 and has determined that the hash offset 218 points to a certain location of the located key 214.

[0056] There is usually a large amount of associated data for a random string of keys 214. As described above and in accordance with the disclosed embodiments, keys 214 can be split into separate KV table entries 270 in sort key table 210. Thereafter, sort key table 210 includes KV table 211 and hash offset table 212. By searching for hash value 216 in hash offset table 212, the system can use the hash value 216 that matches the key 214. Figure 1 The example shown in FIG. 28 shows a method for locating a target KV pair 280 using relatively few CPU cycles.

[0057] In the disclosed embodiment, the hash value 216 is specific to a relatively small amount of data, and all keys 214 may be converted to or mapped to a corresponding hash value 216. Because comparing each key 214 is computationally more expensive than comparing the hash value 216 corresponding to each key 214, the hash value 216 may be mapped to a corresponding hash value 216. Figure 1 Compared to the example shown in FIG, the system of the disclosed embodiment can be faster, smaller, and more efficient. That is, as described above, because the need for searching the internal key hash directory can be reduced or eliminated, the internal key hash directory in the sort key table according to the embodiment of the present disclosure is more efficient. Therefore, the number of CPU cycles associated with locating a key in the table in the KV store is reduced.

[0058] Figure 3 is a flowchart depicting a method of searching for KV pairs according to an embodiment of the present disclosure. Figure 4 is a block diagram depicting a system for searching KV pairs according to an embodiment of the present disclosure.

[0059] Reference Figure 3 and Figure 4, the method for searching for a KV pair may include (e.g., in the storage format data structure 410a or 410b) populating a hash offset table (310) of a sort key table with a hash offset table entry. The hash offset table entry may have a hash value and a hash offset corresponding to a corresponding key. The sort key table may include a KV table, the KV table including a plurality of KV table entries, the KV table entries including KV pairs. The number of KV table entries may be the same as the number of hash offset table entries.

[0060] The method may also include sorting the hash offset table entries based on the hash values ​​(320).

[0061] The method may further include searching the hash offset table for a target hash value corresponding to the target key (330).Searching for the target hash value may include performing a binary search.

[0062] The method may also include calculating a target hash value from the target key (340).

[0063] The method may further include locating a target KV pair corresponding to the target key based on the target hash value (350). The operation of locating the target KV pair may include locating the target KV pair based on the hash offset.

[0064] The method may further include saving the location of the target KV pair (360). Saving the location of the target KV pair may include mapping the target hash value to the target KV pair.

[0065] Figure 4 is a block diagram depicting a system for searching KV pairs according to an embodiment of the present disclosure.

[0066] Reference Figure 4 , the on-memory data structure 420 generally relates to the internal key hash directory creation and life cycle. It can be noted that the details of the internal workings of the on-memory data structure 420 are not particularly relevant to the embodiments of the present disclosure. The on-memory data structure 420 includes a hash map (or on-memory hash table) 422 and an on-memory sorting structure 424. The on-memory data structure 420 initially manages an on-memory KV object (e.g., a KV pair) 426.

[0067] When the on-memory data structure 420 causes a flush to occur so that any recorded KV pairs 426 can be marked for deletion to free up memory space, the on-memory data structure 420 can merge or create a storage format data structure 410a or 410b including the internal key hash directory 412. For example, the on-memory data structure 420 can be merged with the pre-existing storage format data structure 410a. The merging or creation of the storage format data structure 410a or 410b is similar to the merging or creation of the storage format data structure 410a or 410b. Figure 3(e.g., filling and sorting the hash offset table, searching and calculating the target hash value, and locating and saving the location of the target KV pair).

[0068] Thereafter, the newly created or newly merged storage format data structure 410 b replaces any existing storage format data structure 410 a and is saved on the storage device 430 .

[0069] Furthermore, any read (retrieval) operation will cause the on-memory data structure 420 to be searched first when the on-memory data structure 420 may have the latest associated KV pair 426. If the on-memory data structure 420 does not have the latest associated KV pair 426, the storage format data structure 410b may be searched to obtain the KV pair 426.

[0070] Therefore, the disclosed embodiments provide a storage data format for a KV database system that allows fast single-key queries, thereby allowing relatively fast key positioning for a single key in a sorted key table by simplifying its search hierarchy and allowing a reduction in comparison overhead, thereby improving single-key read performance.

Claims

1. A method for searching a target key in a database, the method comprising: populating a hash offset table of the sort key table on a storage device with hash offset table entries having hash values ​​and hash offsets corresponding to respective keys; Sort the hash offset table entries based on the hash value; Search the hash offset table for a target hash value corresponding to the target key; Locate the target key-value pair corresponding to the target key based on the target hash value; as well as The location to save the target key-value pairs.

2. The method according to claim 1, wherein The operation of searching for the target hash value involves performing a binary search.

3. The method according to claim 1, further comprising: Computes the target hash value from the target key.

4. The method according to claim 1, wherein The operation of saving the location of the target key-value pair includes mapping the target hash value to the target key-value pair.

5. The method according to claim 1, wherein The operation of locating the target key-value pair includes locating the target key-value pair based on the hash offset.

6. The method according to claim 1, wherein The sort key table further includes a key-value table, the key-value table includes a plurality of key-value table entries, and the key-value table entries include target key-value pairs.

7. The method according to claim 6, wherein: The number of key-value table entries is the same as the number of hash offset table entries.

8. A system for searching a database for a target key, the system comprising an on-memory data structure, a storage format data structure, and a storage device, wherein: The system is configured to: populating a hash offset table of the sort key table on the storage device using hash offset table entries in the storage format data structure, the hash offset table entries having hash values ​​and hash offsets corresponding to respective keys; sorting the hash offset table entries based on the hash values ​​according to an on-memory sorting structure of the on-memory data structure; Searching the on-memory data structure or the storage format data structure to obtain a target hash value corresponding to the target key in the hash value table; Locating a target key-value pair corresponding to a target key in an on-memory data structure or a storage format data structure based on the target hash value; as well as Save the location of the target key-value pair to an on-memory data structure or a storage format data structure.

9. The system according to claim 8, wherein: The system is further configured to search for the target hash value by performing a binary search.

10. The system according to claim 8, wherein: The system is also configured to calculate a target hash value from the target key.

11. The system according to claim 8, wherein The system is further configured to save the location of the target key-value pair by mapping the target hash value to the target key-value pair.

12. The system according to claim 8, wherein: The system is further configured to locate a target key-value pair based on the hash offset.

13. The system according to any one of claims 8 to 12, wherein: The sort key table further includes a key-value table, the key-value table includes a plurality of key-value table entries, and the key-value table entries include target key-value pairs.

14. A non-transitory computer-readable medium implemented on a system for searching a database for a target key, the non-transitory computer-readable medium having computer code that, when executed on a processor, implements a method for storing data, the method comprising: populating a hash offset table of the sort key table on a storage device with hash offset table entries having hash values ​​and hash offsets corresponding to respective keys; Sort the hash offset table entries based on the hash value; Search the hash offset table for a target hash value corresponding to the target key; Locate the target key-value pair corresponding to the target key based on the target hash value; as well as The location to save the target key-value pairs.

15. The non-transitory computer-readable medium of claim 14, wherein: The computer code, when executed by the processor, further implements the method of data storage by searching for the target hash value by performing a binary search.

16. The non-transitory computer-readable medium of claim 14, wherein: The computer code, when executed by the processor, further implements the method of data storage by calculating a target hash value from a target key.

17. The non-transitory computer-readable medium of claim 14, wherein: The operation of saving the location of the target key-value pair includes mapping the target hash value to the target key-value pair.

18. The non-transitory computer-readable medium of claim 14, wherein: The operation of locating the target key-value pair includes locating the target key-value pair based on the hash offset.

19. The non-transitory computer-readable medium of any one of claims 14 to 18, wherein: The sort key table further includes a key-value table, the key-value table includes a plurality of key-value table entries, and the key-value table entries include target key-value pairs.

20. The non-transitory computer-readable medium of claim 19, wherein: The number of key-value table entries is the same as the number of hash offset table entries.

Images