Hash table processing method, device and electronic device

Abstract

The present invention provides a method, device, and electronic device for processing a hash table. The hash table is located in a persistent data layer and includes multiple hash buckets. Each hash bucket stores multiple key-value pairs. A Bloom filter is provided in a volatile filter layer. The key of each key-value pair stored in the hash bucket is inserted into the Bloom filter and mapped to N bits of data in the Bloom filter. The keys of any two key-value pairs in the hash bucket cannot completely overlap in the data bits mapped in the Bloom filter. A data bit in the Bloom filter can be mapped to at least one key. When a value in a key-value pair in the hash table needs to be searched or deleted, the corresponding key is first searched in the Bloom filter. If the corresponding key does not exist, the corresponding key-value pair does not exist in the hash table, and a search failure or deletion failure indication is returned, thereby reducing access overhead to the persistent data layer and improving negative query performance of the hash table. The present invention improves the processing performance of the hash table.

Description

Technical Field

[0001] The present invention belongs to the field of computer storage, and more specifically, relates to a hash table processing method, device and electronic equipment. Background Art

[0002] Persistent Memory (PM) offers a new approach for building large-scale, low-latency memory and storage systems. PM, such as phase-change memory and Intel Optane DC persistent memory modules (DCPMMs), offer desirable characteristics such as large capacity, high performance, non-volatility, and byte addressability. PM is expected to replace DRAM as the next-generation memory candidate, bridging the performance gap between hard disk drives (HDDs) and dynamic random access memory (DRAM) and enriching the storage tier. Intel's released DCPMMs offer a single-device capacity of up to 256GB, with a maximum single-machine capacity of 8TB (512GB / DIMM*16DIMMs), enabling real-time processing systems with high throughput, low latency, and a low total cost of ownership (TCO) for a variety of applications.

[0003] Hash indexes are widely used as in-memory index structures in key-value stores, such as Memcached and Redis, due to their constant, fast lookup performance. A unique weakness of hash indexes is that if multiple entries have key conflicts (i.e., hash collisions), the hash table must be rehashed or resized if it is full. With the development of PM, many researchers have been working on designing efficient PM-based persistent hash indexes, such as level hashing, CCEH, PCLHT, SOFT, Clevel, and Dash.

[0004] A positive search looks for existing elements in the hash table, while a negative search looks for elements that do not exist in the hash table. However, the PM-based hash indexes mentioned above still suffer from poor negative search performance. Our study of their negative search analysis shows that the negative query throughput of the persistent hash index is only 45.1%-79.8% of its positive query throughput. Negative queries in PM-based hash index systems result in a large number of PM accesses, which significantly degrades system performance, especially when dealing with workloads that frequently query non-existent elements. Therefore, improving negative query performance is necessary and important. Moreover, when the data load is too large, the average bucket chain length of the hash bucket is too long, resulting in excessive positive query throughput of the hash table, affecting system performance.

[0005] In summary, the processing performance of existing hash tables needs to be improved. Summary of the Invention

[0006] In view of the shortcomings of the prior art, the purpose of the present invention is to provide a hash table processing method, device and electronic device, aiming to solve the problem that the processing performance of the existing hash table needs to be improved.

[0007] To achieve the above objectives, in a first aspect, the present invention provides a method for processing a hash table, wherein the hash table is located in a persistent data layer and includes multiple hash buckets, each of which stores multiple key-value pairs. The method includes the following steps:

[0008] A Bloom filter is provided in the volatile filter layer; wherein the key of each key-value pair stored in the hash bucket is inserted into the Bloom filter and mapped to N bits of data in the Bloom filter, where N is an integer greater than 1; the data bits mapped by the keys of any two key-value pairs in the hash bucket in the Bloom filter cannot completely overlap, and one data bit in the Bloom filter can be mapped by at least one key;

[0009] When it is necessary to search or delete the value in a key-value pair in the hash table, first search for the corresponding key in the Bloom filter. If the corresponding key does not exist, the corresponding key-value pair does not exist in the hash table, and a search failure or deletion failure indication is returned to reduce the access overhead to the persistent data layer when the key-value pair does not exist in the hash table, thereby improving the negative query performance of the hash table.

[0010] In one possible implementation, the method further includes the following steps:

[0011] When you need to search or delete a value in a key-value pair in the hash table, first search for the corresponding key in the Bloom filter. If the corresponding key exists, search or delete it in the hash table according to the corresponding key-value pair.

[0012] In one possible implementation, the method further includes the following steps:

[0013] When a key-value pair needs to be inserted into the hash table, first determine whether the key-value pair to be inserted already exists in the hash table. If so, return an insertion failure indication;

[0014] If the key-value pair to be inserted does not exist in the hash table, the corresponding head bucket number is calculated according to the key, and the range in the hash bucket corresponding to the head bucket number is locked;

[0015] A search is performed within the locked hash bucket to determine whether there is an idle key-value pair slot. If so, the key-value pair is inserted into the idle slot, and the inserted key-value pair is persisted to the non-volatile memory using a merge refresh method to complete the key-value pair insertion. The merge refresh method is used to reduce the number of cache line refreshes and reduce the insertion delay overhead of the hash table.

[0016] In one possible implementation, a search is performed within the locked hash bucket to determine whether there is an idle key-value pair slot. If not, a head insertion method is used to allocate a new chain bucket from the non-volatile memory space and assign it to a temporary bucket pointer. The first slot of the new chain bucket is set to the position where the key-value pair is to be inserted, the key-value pair to be inserted is inserted, and the inserted key-value pair is persisted to the non-volatile memory using a merge refresh method. Thereafter, the temporary bucket pointer is assigned to the next bucket pointer of the head bucket pointer, and a cache line refresh instruction is used to persist the next bucket pointer of the head bucket to the non-volatile memory.

[0017] In one possible implementation, the method further includes the following steps:

[0018] The initial header bucket length of the hash table is dynamically set according to the data load of the persistent data layer. When the data load is relatively large, the initial header bucket length is relatively long, and the average bucket chain length of each header bucket is relatively short, so as to improve the positive query performance of the hash table.

[0019] In a second aspect, the present invention provides a hash table processing device, wherein the hash table is located in a persistent data layer and includes multiple hash buckets, each hash bucket storing multiple key-value pairs, and the device includes:

[0020] A Bloom filter setting unit, configured to set a Bloom filter in the volatile filter layer; wherein the key of each key-value pair stored in the hash bucket is inserted into the Bloom filter and mapped to N bits of data in the Bloom filter, where N is an integer greater than 1; the data bits mapped by the keys of any two key-value pairs in the hash bucket in the Bloom filter cannot completely overlap, and a data bit in the Bloom filter can be mapped by at least one key;

[0021] The hash table negative query unit is used to first search for the corresponding key in the Bloom filter when it is necessary to search or delete the value in a key-value pair in the hash table. If the corresponding key does not exist, the corresponding key-value pair does not exist in the hash table, and a search failure or deletion failure indication is returned to reduce the access overhead to the persistent data layer when the key-value pair does not exist in the hash table, thereby improving the negative query performance of the hash table.

[0022] In one possible implementation, the device further includes:

[0023] A hash table insertion unit is used to determine whether the key-value pair to be inserted already exists in the hash table when a key-value pair needs to be inserted into the hash table. If so, an insertion failure indication is returned; if the key-value pair to be inserted does not exist in the hash table, the corresponding head bucket number is calculated according to the key, and the range in the hash bucket corresponding to the head bucket number is locked; whether there is an idle key-value pair slot in the locked range of the hash bucket, if so, the key-value pair is inserted into the idle slot, and the inserted key-value pair is persisted to the non-volatile memory using a merge refresh method to complete the key-value pair insertion; wherein the merge refresh method is used to reduce the number of cache line refreshes and reduce the insertion delay overhead of the hash table.

[0024] In one possible implementation, the hash table insertion unit is used to search within the locked hash bucket for whether there is an idle key-value pair slot. If not, a head insertion method is used to allocate a new chain bucket from the non-volatile memory space and assign it to a temporary bucket pointer. The first slot of the new chain bucket is set to the position where the key-value pair is to be inserted, the key-value pair to be inserted is inserted, and the inserted key-value pair is persisted to the non-volatile memory using a merge refresh method; then, the temporary bucket pointer is assigned to the next bucket pointer of the head bucket pointer, and a cache line refresh instruction is used to persist the next bucket pointer of the head bucket to the non-volatile memory.

[0025] In one possible implementation, the device further includes:

[0026] The header bucket length setting unit is used to dynamically set the initial header bucket length of the hash table according to the data load of the persistent data layer. When the data load is relatively large, the initial header bucket length is relatively long, and the average bucket chain length of each header bucket is relatively short, so as to improve the positive query performance of the hash table.

[0027] In a third aspect, the present invention provides an electronic device comprising: at least one memory for storing programs; and at least one processor for executing the programs stored in the memory. When the program stored in the memory is executed, the processor is used to execute the method described in the first aspect or any possible implementation of the first aspect.

[0028] In a fourth aspect, the present invention provides a computer-readable storage medium storing a computer program. When the computer program runs on a processor, the processor executes the method described in the first aspect or any possible implementation of the first aspect.

[0029] In a fifth aspect, the present invention provides a computer program product, which, when executed on a processor, enables the processor to execute the method described in the first aspect or any possible implementation of the first aspect.

[0030] In general, the above technical solutions conceived by the present invention have the following beneficial effects compared with the prior art:

[0031] The present invention provides a method, device and electronic device for processing a hash table. When it is necessary to search or delete a key-value pair that does not exist in the hash table, the traditional approach is to perform a negative query in the hash table of the persistent data layer, which will search all the key-value pairs in the hash table. When the key-value pair does not exist, the search overhead in this case is very large and unnecessary. The present invention can quickly locate non-existent keys through the Bloom filter located in the volatile filter layer, avoiding the search of non-existent key-value pairs in the persistent data layer, thereby improving the negative query performance and the deletion operation of non-existent key-value pairs. When inserting a key-value pair into the hash table of the persistent data layer, the present invention reduces the number of cache line refreshes through the header insertion method and the merge refresh operation, thereby improving the insertion performance. The present invention reduces the average chain bucket length of each header bucket by setting a relatively long header bucket length, thereby improving the positive query performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is a diagram of a non-volatile memory architecture provided by an embodiment of the present invention;

[0033] Figure 2 This is a diagram of a hash bucket architecture provided by an embodiment of the present invention;

[0034] Figure 3 is a mapping relationship diagram between Bloom filters and keys provided by an embodiment of the present invention;

[0035] Figure 4 This is a flow chart of a hash table processing method provided by an embodiment of the present invention;

[0036] Figure 5 This is a lock-based hash bucket concurrency control diagram provided by an embodiment of the present invention;

[0037] Figure 6 This is a flowchart of inserting key-value pairs into a hash table provided by an embodiment of the present invention;

[0038] Figure 7 This is a flowchart of a hash table search provided by an embodiment of the present invention;

[0039] Figure 8 This is a flowchart of deleting a hash table provided by an embodiment of the present invention;

[0040] Figure 9 This is an architecture diagram of a hash table processing device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0042] In the embodiments of the present invention, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present invention should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0043] In the description of the embodiments of the present invention, unless otherwise specified, “plurality” means two or more.

[0044] In response to the defects of the existing technology and the need for improvement, the present invention provides a hash table processing method to reduce negative searches to accelerate persistent hash indexes. Its purpose is to improve the negative query performance of persistent hash indexes, provide high multi-threaded scalability, and ensure low data consistency overhead.

[0045] The embodiment of the present invention provides a non-volatile memory, the structure of which is shown in FIG. Figure 1 As shown in the figure, the hash table can be either traditional chained hashing or other hashing variants such as extendible hashing. The Volatile Filter Layer (VFL) is a DRAM-based Bloom filter that reduces negative lookup operations on the PM hash table, significantly improving negative lookup performance and reducing PM access overhead, which is more expensive than DRAM. The Persistent Data Layer (PDL) is a persistent memory-based hash table (which can be chained hashing, extendible hashing, or other hash tables). Here, using chained hashing as an example, the initial header bucket length is set to N. N can be dynamically set based on the scenario. The chained bucket is divided into a head bucket and a chained bucket, each of which is a multi-slot bucket. Generally speaking, a longer initial header bucket length is preferred, as this reduces the average bucket chain length of each head bucket and reduces the positive lookup overhead. However, if the header bucket length is too long and the load is relatively small, memory space will be wasted. Therefore, those skilled in the art can set the header bucket length according to actual conditions to take into account memory and positive query overhead.

[0046] In addition, the persistent data layer needs to ensure the correctness of multi-threaded lock-based concurrency and low data consistency overhead in the event of system failures, while the VFL needs to quickly recover from the consistent persistent data layer in the event of a system failure.

[0047] Figure 2 This is a mapping relationship diagram between a Bloom filter and a key provided by an embodiment of the present invention. Figure 2 As shown, y1, y2 and y3 represent three keys, each key is mapped to three bits of the Bloom filter, and the three bits are distributed in different positions. Figure 3 As you can see, a bit can be mapped to at least one key, and the three bits mapped to different keys will not completely overlap. If the key exists, the data of the corresponding three bits is set to 1, and if the key does not exist, the data of the corresponding bits is set to 0.

[0048] In order to improve the efficiency of cache lines, the structure size of the head bucket and the chain bucket is an integer multiple of the 64-byte cache line size, which can be 64 bytes, 128 bytes, or 256 bytes. Here, the head bucket and chain bucket with a size of 64 bytes are used as an example. Figure 3 As shown in the figure. Token represents a tag, KV represents a key-value pair, Next BP represents the next bucket pointer (BP), and Dummy represents the extra padding added for 64-byte cache line alignment. A tag value of 0 indicates that the corresponding key-value pair slot is unused, while a value of 1 indicates that the corresponding key-value pair slot is occupied. KV can store fixed-length key-value pairs or 8-byte pointers to variable-length key-value pairs. Here, we use fixed-length as an example, with both keys and values being 8 bytes.

[0049] Figure 4 This is a flow chart of a hash table processing method provided by an embodiment of the present invention. The hash table is located in the persistent data layer and includes multiple hash buckets. Each hash bucket stores multiple key-value pairs. The method includes the following steps:

[0050] S101, setting a Bloom filter in a volatile filter layer; wherein the key of each key-value pair stored in the hash bucket is inserted into the Bloom filter and mapped to N bits of data in the Bloom filter, where N is an integer greater than 1; the data bits mapped by the keys of any two key-value pairs in the hash bucket in the Bloom filter cannot completely overlap, and one data bit in the Bloom filter can be mapped to at least one key;

[0051] S102, when it is necessary to search or delete the value in a key-value pair in the hash table, first search the corresponding key in the Bloom filter. If the corresponding key does not exist, the corresponding key-value pair does not exist in the hash table, and a search failure or deletion failure indication is returned to reduce the access overhead to the persistent data layer when the key-value pair does not exist in the hash table, thereby improving the negative query performance of the hash table.

[0052] In one example, Figure 5 This is a lock-based hash bucket concurrency control diagram provided by an embodiment of the present invention. It can be seen that read operations use shared read locks, write operations use mutually exclusive write locks, and the size of each lock includes a range of 256 64-byte header buckets.

[0053] In one example, Figure 6 The flowchart of inserting a key-value pair into a hash table provided by an embodiment of the present invention includes the following steps:

[0054] (1) Calculate the hash value and the head bucket number according to the given key, and find the bucket pointer corresponding to the chain hash in the persistent data layer.

[0055] (2) Before inserting a key-value pair, first determine whether the key-value pair to be inserted exists in the chain hash in the non-volatile memory. If so, execute (3); otherwise, execute step (4).

[0056] (3) The key-value pair to be inserted already exists in the hash table, and the insertion failure is directly returned.

[0057] (4) Based on the calculated specific head bucket number, an intelligent mutex lock is used to lock the lock range where the head bucket is located.

[0058] (5) Check whether there is an idle key-value pair slot in the initial head bucket and the first chain bucket (if it exists). If there is an idle slot, execute step (6); if not, execute step (7).

[0059] (6) The slot value and key of the key-value pair are modified to the value and key of the key-value pair to be inserted, and the slot flag is set to 1. All modifications are persisted to non-volatile memory using a merge refresh method with a 64-byte cache line size bucket granularity. At this point, the insertion is successful.

[0060] (7) Using the head insertion method, a new chain bucket is allocated from the non-volatile memory space and assigned to the temporary bucket pointer, and the first slot of the chain bucket is set to the position where the key-value pair is to be inserted. The slot value, key and corresponding flag are modified in the same way, and the modification is persisted to the PM by a merged cache line refresh method based on 8-byte atomic updates.

[0061] (8) Set a memory fence instruction, assign the temporary bucket pointer to the next bucket pointer of the head bucket pointer, and use a cache line refresh instruction to persist the next bucket pointer of the head bucket into the non-volatile memory.

[0062] (9) Set the memory fence instruction and insert the key of the key-value pair to be inserted into the Bloom filter located in the volatile filter layer. Return successful insertion.

[0063] In one example, Figure 7 The following is a flowchart of a hash table search provided by an embodiment of the present invention; the following steps are included:

[0064] (1) Check whether the key to be searched is contained in the Bloom filter. If not, execute step (2); if so, execute step (3);

[0065] (2) If the key-value pair to be searched does not exist, it returns empty directly;

[0066] (3) Calculate the hash value and the head bucket number according to the given key, find the bucket pointer corresponding to the chain hash in the persistent data layer, and lock it with a shared read lock;

[0067] (4) Determine whether the bucket pointer is empty. If it is empty, the loop ends and executes step (2); otherwise, execute step (5);

[0068] (5) Traverse the multiple slots in the bucket in order. If the corresponding flag is 1 and the key of the key-value pair slot is equal to the key to be searched, execute step (6); otherwise, execute step (7);

[0069] (6) If the key-value pair to be found exists, return the corresponding value.

[0070] (7) Update the bucket pointer to point to the next bucket pointer. Execute step (4).

[0071] In one example, Figure 8 The following is a flowchart of deleting a hash table provided by an embodiment of the present invention; the following steps are included:

[0072] (1) Check whether the key to be searched is contained in the Bloom filter. If not, execute step (2); if so, execute step (3);

[0073] (2) If the key-value pair to be deleted does not exist, the deletion failure is directly returned;

[0074] (3) Calculate the hash value and the head bucket number according to the given key, find the bucket pointer corresponding to the chain hash in the persistent data layer, and lock it with a smart mutex;

[0075] (4) Determine whether the bucket pointer is empty. If it is empty, the loop ends and executes step (2); otherwise, execute step (5);

[0076] (5) Traverse the multiple slots in the bucket in order. If the corresponding flag is 1 and the key of the key-value pair slot is equal to the key to be searched, execute step (6); otherwise, execute step (7);

[0077] (6) If there is a key-value pair to be deleted, set the flag of the corresponding key-value pair slot to 0, persist it to the non-volatile memory through the cache line refresh instruction, and return a deletion success.

[0078] (7) Update the bucket pointer to point to the next bucket pointer. Execute step (4).

[0079] Figure 9 This is a diagram of the hash table processing device architecture provided by an embodiment of the present invention. Figure 9 Shown, including:

[0080] A Bloom filter setting unit 910 is configured to set a Bloom filter in the volatile filter layer; wherein the key of each key-value pair stored in the hash bucket is inserted into the Bloom filter and mapped to N bits of data in the Bloom filter, where N is an integer greater than 1; the data bits mapped by the keys of any two key-value pairs in the hash bucket in the Bloom filter cannot completely overlap, and a data bit in the Bloom filter can be mapped to at least one key;

[0081] The hash table negative query unit 920 is used to first search for the corresponding key in the Bloom filter when it is necessary to search or delete the value in a key-value pair in the hash table. If the corresponding key does not exist, the corresponding key-value pair does not exist in the hash table, and a search failure or deletion failure indication is returned to reduce the access overhead to the persistent data layer when the key-value pair does not exist in the hash table, thereby improving the negative query performance of the hash table.

[0082] The hash table insertion unit 930 is used to determine whether the key-value pair to be inserted already exists in the hash table when a key-value pair needs to be inserted in the hash table. If so, an insertion failure indication is returned; if the key-value pair to be inserted does not exist in the hash table, the corresponding head bucket number is calculated according to the key, and the range in the hash bucket corresponding to the head bucket number is locked; whether there is an idle key-value pair slot in the locked range of the hash bucket is searched, and if so, the key-value pair is inserted into the idle slot, and the inserted key-value pair is persisted to the non-volatile memory using a merge refresh method to complete the key-value pair insertion; wherein, the merge refresh method is used to reduce the number of cache line refreshes and reduce the insertion delay overhead of the hash table.

[0083] The hash table insertion unit 930 is used to search for free key-value pair slots within the locked hash bucket. If not, a head insertion method is used to allocate a new chain bucket from the non-volatile memory space and assign it to a temporary bucket pointer. The first slot of the new chain bucket is set to the position where the key-value pair is to be inserted, the key-value pair to be inserted is inserted, and the inserted key-value pair is persisted to the non-volatile memory using a merge refresh method. The temporary bucket pointer is then assigned to the next bucket pointer of the head bucket pointer, and the cache line refresh instruction is used to persist the next bucket pointer of the head bucket to the non-volatile memory.

[0084] The header bucket length setting unit 940 is used to dynamically set the initial header bucket length of the hash table according to the data load of the persistent data layer. When the data load is relatively large, the initial header bucket length is relatively long, and the average bucket chain length corresponding to each header bucket is relatively short, so as to improve the positive query performance of the hash table.

[0085] It should be understood that the above-mentioned device is used to execute the method in the above-mentioned embodiment. The implementation principle and technical effect of the corresponding program module in the device are similar to those described in the above-mentioned method. The working process of the device can refer to the corresponding process in the above-mentioned method and will not be repeated here.

[0086] Based on the methods described in the above embodiments, embodiments of the present invention provide an electronic device. The device may include at least one memory for storing programs and at least one processor for executing the programs stored in the memory. When the programs stored in the memory are executed, the processor is configured to execute the methods described in the above embodiments.

[0087] Based on the method in the above embodiment, an embodiment of the present invention provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the method in the above embodiment.

[0088] Based on the method in the above embodiment, an embodiment of the present invention provides a computer program product. When the computer program product runs on a processor, the processor executes the method in the above embodiment.

[0089] It is understood that the processor in the embodiments of the present invention may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0090] The method steps in the embodiments of the present invention can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and the storage medium can be located in an ASIC.

[0091] In the above embodiments, all or part of the embodiments may be implemented using software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted via the computer-readable storage medium. The computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media. The available medium may be a magnetic medium (e.g., a floppy disk, hard disk, tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0092] It should be understood that the various numerical numbers involved in the embodiments of the present invention are only used for the convenience of description and are not intended to limit the scope of the embodiments of the present invention.

[0093] It will be easily understood by those skilled in the art that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for processing a hash table, wherein the hash table is located in a persistent data layer and includes multiple hash buckets, each of which stores multiple key-value pairs, characterized in that: The method comprises the following steps: A Bloom filter is provided in the volatile filter layer; wherein the key of each key-value pair stored in the hash bucket is inserted into the Bloom filter and mapped to N bits of data in the Bloom filter, where N is an integer greater than 1; the data bits mapped by the keys of any two key-value pairs in the hash bucket in the Bloom filter cannot completely overlap, and one data bit in the Bloom filter can be mapped by at least one key; When searching or deleting a value in a key-value pair in the hash table, the corresponding key is first searched in the Bloom filter. If the corresponding key does not exist, the corresponding key-value pair does not exist in the hash table, and a search failure or deletion failure indication is returned. This reduces the access overhead to the persistent data layer when the key-value pair does not exist in the hash table, and improves the negative query performance of the hash table. When a key-value pair needs to be inserted into the hash table, first determine whether the key-value pair to be inserted already exists in the hash table. If so, return an insertion failure indication; If the key-value pair to be inserted does not exist in the hash table, the corresponding head bucket number is calculated according to the key, and the range in the hash bucket corresponding to the head bucket number is locked; A search is performed within the locked hash bucket to determine whether there is an idle key-value pair slot. If so, the key-value pair is inserted into the idle slot, and the inserted key-value pair is persisted to the non-volatile memory using a merge refresh method to complete the key-value pair insertion. The merge refresh method is used to reduce the number of cache line refreshes and reduce the insertion delay overhead of the hash table.

2. The method according to claim 1, characterized in that The following steps are also included: When you need to search or delete a value in a key-value pair in the hash table, first search for the corresponding key in the Bloom filter. If the corresponding key exists, search or delete it in the hash table according to the corresponding key-value pair.

3. The method according to claim 2, characterized in that Search within the locked hash bucket range to see if there is an idle key-value pair slot. If not, use the head insertion method to allocate a new chain bucket from the non-volatile memory space and assign it to the temporary bucket pointer. Set the first slot of the new chain bucket to the position where the key-value pair is to be inserted, insert the key-value pair to be inserted, and use the merge refresh method to persist the inserted key-value pair to the non-volatile memory; then assign the temporary bucket pointer to the next bucket pointer of the head bucket pointer, and use the cache line refresh instruction to persist the next bucket pointer of the head bucket to the non-volatile memory.

4. The method according to any one of claims 1 to 3, characterized in that The following steps are also included: The initial header bucket length of the hash table is dynamically set according to the data load of the persistent data layer. When the data load is relatively large, the initial header bucket length is relatively long, and the average bucket chain length of each header bucket is relatively short, so as to improve the positive query performance of the hash table.

5. A hash table processing device, wherein the hash table is located in a persistent data layer and includes multiple hash buckets, each of which stores multiple key-value pairs, characterized in that: The device includes: A Bloom filter setting unit, configured to set a Bloom filter in the volatile filter layer; wherein the key of each key-value pair stored in the hash bucket is inserted into the Bloom filter and mapped to N bits of data in the Bloom filter, where N is an integer greater than 1; the data bits mapped by the keys of any two key-value pairs in the hash bucket in the Bloom filter cannot completely overlap, and a data bit in the Bloom filter can be mapped by at least one key; The hash table negative query unit is used to search for the corresponding key in the Bloom filter when it is necessary to search or delete the value in a key-value pair in the hash table. If the corresponding key does not exist, the corresponding key-value pair does not exist in the hash table, and a search failure or deletion failure indication is returned. This reduces the access overhead to the persistent data layer when the key-value pair does not exist in the hash table, and improves the negative query performance of the hash table; A hash table insertion unit is used to determine whether the key-value pair to be inserted already exists in the hash table when a key-value pair needs to be inserted into the hash table. If so, an insertion failure indication is returned; if the key-value pair to be inserted does not exist in the hash table, the corresponding head bucket number is calculated according to the key, and the range in the hash bucket corresponding to the head bucket number is locked; whether there is an idle key-value pair slot in the locked range of the hash bucket, if so, the key-value pair is inserted into the idle slot, and the inserted key-value pair is persisted to the non-volatile memory using a merge refresh method to complete the key-value pair insertion; wherein the merge refresh method is used to reduce the number of cache line refreshes and reduce the insertion delay overhead of the hash table.

6. The device according to claim 5, characterized in that The hash table insertion unit is used to search for an idle key-value pair slot within the locked hash bucket. If not, a head insertion method is used to allocate a new chain bucket from the non-volatile memory space and assign it to a temporary bucket pointer. The first slot of the new chain bucket is set to the position where the key-value pair is to be inserted, the key-value pair to be inserted is inserted, and the inserted key-value pair is persisted to the non-volatile memory by a merge refresh method. Then, the temporary bucket pointer is assigned to the next bucket pointer of the head bucket pointer, and a cache line refresh instruction is used to persist the next bucket pointer of the head bucket to the non-volatile memory.

7. The device according to claim 5 or 6, characterized in that Also includes: The header bucket length setting unit is used to dynamically set the initial header bucket length of the hash table according to the data load of the persistent data layer. When the data load is relatively large, the initial header bucket length is relatively long, and the average bucket chain length of each header bucket is relatively short, so as to improve the positive query performance of the hash table.

8. An electronic device, characterized in that: include: at least one memory for storing a program; At least one processor is used to execute the program stored in the memory. When the program stored in the memory is executed, the processor is used to execute the method according to any one of claims 1 to 4.

Images