Write optimization Hash index construction method and system based on separated memory

Abstract

The invention discloses a write optimization Hash index construction method and system based on a separated memory, and the method comprises the steps: building a Hash index structure comprising a directory, a segment and a bucket in a memory node based on the design of the separated memory, building a directory tree structure comprising a leaf item and a non-leaf item, and caching a corresponding directory in a computing node; maintaining a corresponding predictor for each slot in the barrel, and indicating the position where the slot should be placed during splitting by reading a corresponding bit in the predictor during splitting; when the Hash index is introduced and split, starting a single-thread memory crawler in a memory node to regularly search an unupdated predictor; multiple round-trip times are compressed into one by merging multiple RDMA operations into one RDAM request through passive acknowledgement and batch processing. According to the method, the multi-level catalog is combined with the extensible Hash, the Hash index is expanded and synchronized in a fine-grained mode, meanwhile, the splitting overhead is reduced through a predictor component and optimization splitting, and finally the writing performance is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of key-value storage and hash index, and in particular to a write-optimized hash index construction method and system based on separate memory. Background Art

[0002] Hash index, also known as hash index, is widely used as the cornerstone of data-intensive applications, such as in-memory databases, key-value storage systems, etc., due to its constant-level search time complexity to provide fast response. For any given keyword, the hash index determines the location of the key-value pair based on the mapping of the keyword through the hash function. When a search request is required, the hash index substitutes the keyword into the hash function to obtain the address of the record containing the keyword in the table, and accesses the value corresponding to the keyword through the address.

[0003] Traditional data centers consist of multiple monolithic servers, each equipped with a CPU and memory. This architecture tightly couples computing and storage resources, causing data centers to suffer from underutilization, low scalability, and high total cost. Separate memory decouples computing and storage resources into a computing pool and an independently managed and scalable memory pool. The computing pool includes many CPU blades and a small amount of memory space for caching part of application data, while the memory pool has many memory blades and slight computing power. The computing pool can communicate directly with the memory pool via the RDMA network card to achieve high access bandwidth and low latency. Existing hash index research results can be mainly divided into two categories: (1) hierarchical hash index (level, clevel); (2) scalable hash index (cceh, dash, race). Existing hash index work based on separate memory mainly focuses on hash index based on unilateral RDMA operation (RACE). It is observed that existing research results consider the common characteristics of weak computing power of separated memory nodes, high RDMA network bandwidth, and low latency when designing key-value storage systems, but ignore the performance impact caused by splitting and the high synchronization overhead caused by cache design, which leads to poor write performance. There are two reasons for the poor write performance: (1) Existing research results often read the entire remote directory when a single directory entry in the cache directory is found to be invalid. Reading the entire directory will cause serious traffic amplification, squeezing the traffic used for writing, thereby reducing the bandwidth utilization in the RDMA network and causing write performance degradation; (2) In order to achieve complete lock-free, existing research results use remote primitives to operate each key-value pair in splitting, resulting in multiple network requests for a single split, resulting in high overhead, thereby reducing write performance. Therefore, based on the weak computing power of memory nodes and the small storage capacity of computing nodes, how to design hash indexes to fully utilize their device characteristics in separated memories is still a challenging and important problem in the field of hash indexing. Summary of the invention

[0004] The purpose of the present invention is to address the defects and shortcomings in the prior art and propose a write-optimized hash index construction method and system based on separate memory, combining multi-level directories with scalable hashes, expanding and synchronizing hash indexes in a fine-grained manner, and using predictor components and optimized splitting to reduce splitting overhead, ultimately improving write performance.

[0005] The technical solution of the present invention is as follows.

[0006] On the one hand, a write-optimized hash index construction method based on separated memory includes:

[0007] S101, based on the separated memory design, a hash index structure including directories, segments and buckets is established in the memory node, and a directory tree structure including leaf items and non-leaf items is established, and the corresponding directory is cached in the computing node; wherein each bucket has preset slots; the directory item in the leaf item points to the corresponding segment;

[0008] S102, maintaining a corresponding predictor for each slot, and reading a corresponding bit in the predictor to indicate the position where the slot should be placed when splitting;

[0009] S103, when the hash index introduces a split, a single-threaded memory crawler is started in the memory node to periodically search for unupdated predictors;

[0010] S104, when the corresponding computing node successfully obtains the segment lock during the split, the computing node merges multiple remote CAS primitives in the modified directory entry into one request through the doorbell batch processing technology and sends it; in the insertion and update operations, the computing node merges the reading bucket and the writing key value into one request through the doorbell batch processing technology.

[0011] Preferably, the S101 specifically includes:

[0012] S1011, establish a hash index structure consisting of a directory, a segment, and a bucket in the memory node; wherein the directory contains directory entries, each directory entry points to a corresponding segment, the segment contains a fixed number of M buckets, each bucket has N slots, and each slot is used to store a pointer to a key-value pair;

[0013] S1012, establishing a directory tree structure in the memory node; wherein the directory tree includes leaf items and non-leaf items, the leaf items and the non-leaf items include the same number of directory items, the non-leaf item directory item points to the next leaf item, and the directory item in the leaf item points to the corresponding segment;

[0014] S1013, cache the corresponding directory in the computing node.

[0015] Preferably, the S102 specifically includes:

[0016] S1021, storing a corresponding predictor in each segment, the predictor including M×N slots, the number of slots being equal to the number of key-value pairs that can be accommodated in the segment, the slots of each predictor corresponding to the slots in the corresponding segment, and the predictor slot storing a part of the hash value of the key-value pair in the slot of the corresponding segment;

[0017] S1022, when a computing node attempts to insert a key-value pair into a bucket, but the number of key-value pairs in the bucket reaches the maximum upper limit, a split is triggered. At this time, the computing node reads the segment corresponding to the bucket and the predictor corresponding to the segment to perform a split operation on the segment;

[0018] S1023, after reading the segment to be split, the computing node allocates a new segment to accommodate the new key-value pair;

[0019] S1024, the computing node scans each bucket in the segment in ascending order of bucket numbers, and detects whether the slot in each bucket is empty. If it is not empty, it means that the slot stores a pointer to a key-value pair, and searches for the corresponding prediction slot in the predictor according to the position of the slot and the position of the bucket;

[0020] S1025, since the prediction slot stores information about the corresponding hash value of the key-value pair, the information about the corresponding hash value of the key-value pair is used to indicate the location where the key-value pair pointer is stored when it is split;

[0021] S1026, if the location where the key-value pair pointer is stored when it is split is a new bucket, the key-value pair pointer is temporarily written to the local new segment, and after all buckets are traversed, the new segment is written to the remote memory node through a remote write request;

[0022] S1027, after writing the new segment, the computing node sets the directory entry corresponding to the new segment to the address of the new segment through a remote compare and exchange CAS primitive so that the new segment can be discovered by other computing nodes;

[0023] S1028, finally the computing node deletes all key-value pair pointers in the old segment that have been migrated to the new segment through the remote comparison and exchange CAS primitive.

[0024] Preferably, the S103 specifically includes:

[0025] S1031, the memory node starts a memory crawler program, and the memory crawler program continuously scans the predictor part of each segment in the entire hash table;

[0026] S1032, when the crawler finds that the prediction slot in the predictor is empty, it checks whether the slot in the corresponding bucket is empty. If it is empty, it means that the slot in the bucket has not been occupied and does not need to be updated. If it is not empty, it means that a client segment has been inserted into the bucket slot but the corresponding prediction slot has not been updated. At this time, the crawler accesses the position pointed to by the pointer in the bucket slot, reads the corresponding key value and calculates it, intercepts part of the hash value and stores it in the prediction slot;

[0027] S1033, when the computing node has been inserted into the bucket slot but the memory crawler has not scanned the corresponding bucket slot, the computing node adopts split-based predictor maintenance. When the computing node splits, it is found that there is a pointer in the bucket slot but the corresponding prediction slot is empty, then it is considered that the memory crawler has not had time to update the corresponding prediction slot; at this time, the computing node reads the corresponding key-value pair from the remote end for hash calculation and writes the intercepted part of the hash value into the prediction slot through the remote comparison and exchange CAS primitive to complete the prediction slot update.

[0028] Preferably, the S104 specifically includes:

[0029] S1041, the computing node combines multiple remote CAS primitives in the modified directory entry into one request through the doorbell batch processing technology and sends the request;

[0030] S1042, in the insert and update operations, the computing node merges the read bucket and write key value into one request through the doorbell batch processing technology, and pushes the signal request to the front of the read bucket operation. That is, when the read bucket operation is completed, the remote end will send a completion signal to the computing node to notify the computing node that the operation is completed.

[0031] On the other hand, a write-optimized hash index construction system based on separated memory includes:

[0032] The node index and directory cache module is used to establish a hash index structure including directories, segments and buckets in the memory node based on the separated memory design, and to establish a directory tree structure including leaf items and non-leaf items, and cache the corresponding directory in the computing node; wherein each bucket has preset slots; the directory item in the leaf item points to the corresponding segment;

[0033] A split module, used for maintaining a corresponding predictor for each slot, and indicating the position where the slot should be placed during splitting by reading the corresponding bit in the predictor during splitting;

[0034] The memory node crawler module is used to start a single-threaded memory crawler in the memory node to periodically search for unupdated predictors when a hash index is split;

[0035] The RDMA network optimization module is used to merge multiple remote CAS primitives in the modified directory entry into one request and send it through the doorbell batch processing technology after the corresponding computing node successfully obtains the segment lock during the split; in the insertion and update operations, the computing node merges the reading bucket and the writing key value into one request through the doorbell batch processing technology.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] (1) The hash index of the present invention caches the directory in the computing node to reduce the network overhead of reading the remote directory once; the directory is split into multiple unit directories of fixed size and organized in a tree structure for fine-grained synchronization and expansion to reduce the synchronization traffic caused by the update;

[0038] (2) The present invention maintains a corresponding predictor for each slot, and can indicate the position where the slot should be placed when splitting by reading the corresponding bit in the predictor, so as to reduce the reading overhead caused by the splitting;

[0039] (3) Maintaining the predictor: Directly updating the predictor at each insertion will result in an additional operation during the insertion. The additional operation will result in additional round-trip time and higher latency. The hash index of the present invention introduces split-time maintenance and cooperation with the memory crawler. A single-threaded memory crawler is started in the memory node to regularly search for unupdated predictors to achieve low-overhead predictor maintenance.

[0040] (4) Directly using unmodified unilateral RDMA operations will cause a single read operation to require multiple RDMA operations, thereby increasing the number of network round trips and increasing latency. The present invention merges multiple RDMA operations into one RDAM request through passive confirmation and batch processing, compressing multiple round-trip times into one, thereby improving index execution efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 An example diagram of a multi-level index structure according to an embodiment of the present invention;

[0043] Figure 2 This is a flow chart of a write-optimized hash index construction method based on separate memory according to an embodiment of the present invention;

[0044] Figure 3An exemplary diagram of a multi-level structure index search process according to an embodiment of the present invention;

[0045] Figure 4 1 is an example diagram of two strategies in splitting according to an embodiment of the present invention; wherein (a) represents segment splitting; (b) represents directory splitting;

[0046] Figure 5 An exemplary diagram of a splitting process in which a predictor is not used and a splitting process in which a predictor is used in an embodiment of the present invention;

[0047] Figure 6 An example diagram of a process of updating a cache directory unit by a computing node according to an embodiment of the present invention;

[0048] Figure 7 An example diagram of two different RDMA operation optimizations used in an embodiment of the present invention;

[0049] Figure 8 A structural block diagram of a write-optimized hash index construction system based on separate memory according to an embodiment of the present invention;

[0050] Fig. 9 The graph is a result of an experiment on the multi-threaded writing, searching, updating and deleting performance under balanced and tilted loads of an embodiment of the present invention;

[0051] Fig.10 The figure is a graph showing the experimental results of the performance test of the embodiment of the present invention under different real loads;

[0052] Fig.11 This is a graph showing the experimental results of the real-time performance of an embodiment of the present invention when inserting a certain amount of key-value pairs;

[0053] Fig.12 FIG. 1 is an experimental result diagram of delay distribution of an insertion operation according to an embodiment of the present invention;

[0054] Fig.13 The figure is a graph showing the experimental results of synchronous traffic under different insertion numbers according to an embodiment of the present invention;

[0055] Fig.14 This is a diagram of experimental results of using a predictor to reduce dereference traffic during splitting in an embodiment of the present invention;

[0056] Fig.15 A graph showing experimental results of the performance of design decomposition for an embodiment of the present invention. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] The write-optimized hash index construction method based on separated memory of the present invention is applied in the following aspects: Figure 1 The multi-level index structure shown includes memory nodes, computing nodes, and an RDMA network that implements communication between the memory nodes and the computing nodes.

[0059] like Figure 2 As shown, this embodiment discloses a write-optimized hash index construction method based on separate memory, including:

[0060] S101, based on the separated memory design, establishes a hash index structure including directories, segments and buckets in the memory node, and establishes a directory tree structure including leaf items and non-leaf items, and caches the corresponding directory in the computing node; wherein each bucket has preset slots; the directory item in the leaf item points to the corresponding segment.

[0061] Specifically, an extensible hash index is initialized on the memory node. The hash index consists of directories, segments, and buckets. The index contains a directory tree. Each item in a directory is a persistent pointer to a segment. A segment is a collection of multiple buckets. A bucket can hold 32 slots of 8B, and each slot can store a pointer to a key-value pair. Each segment stores key-value pairs with the same hash value in the first M bits, and each bucket in the segment stores pointers to key-value pairs with the same hash value in the last N bits. Memory is allocated to the indexed directories and segments in sequence on the memory node through the allocation function.

[0062] To improve search performance, this index extracts part of the hash value and divides it into v non-overlapping parts (v-layer index structure). Its organization is similar to a prefix tree. Specifically, if the hash value has a total of h bits, the h bits are divided into v parts, each part contains h / v bits, that is, each layer uses h / v bits for indexing. To make it easier to understand, assume that the same number of bits (represented by d) is used in each layer of directory entries. Therefore, each subdirectory contains 2 d Directory item. Figure 3 This is a three-level index structure of this index, and its first level contains four subdirectories. Here, two digits (i.e., d=2) are used in the directory entries to index, and each subdirectory consists of four directory entries (i.e., 2 2 )composition.

[0063] In order to achieve higher space utilization and more flexible hierarchical structure, this index is designed with two types of directory entries, namely internal entries (non-leaf entries) and leaf entries. Internal entries contain pointers to the next level of subdirectories, while leaf entries contain pointers to segments. For example, in Figure 2 The first subdirectory in level 1 contains an internal entry pointing to the next level of subdirectories and three leaf entries pointing to segments.

[0064] If the segment is located only by relying on the index structure, the pointer of the key-value pair is stored sequentially in the segment, that is, when inserting, the segment is pointed to according to the index, and inserted at any free position in the segment, which will result in multiple detections during query. This is because the segment contains a large number of slots. If linear detection is performed in the segment during query, the round-trip delay caused by the detection is unbearable for the system. Assuming that the segment contains 64 slots, the worst case requires 64 slots to finally find the searched key-value pair, which means that 128 remote operations are required, because only pointers are stored in the slots. Each detection requires a remote read to detect the pointer in the slot, and another remote read to read the key-value pair according to the pointer. In order to reduce the number of linear detections. The present invention designs a more fine-grained storage structure-buckets. Each segment of this index consists of multiple buckets, and each key-value pair may only be in one bucket in the segment. If there are n slots in the segment, the first b bits (in binary) of the hash value H of the key-value pair are used to determine which bucket in the segment it should be inserted into, and the segment is logically divided into buckets, each of which contains n / 2b slots. In the segment-only design, the worst-case query detection number is 2*n remote operations, while after adding buckets, the worst-case query detection number is 2*n / 2b remote operations.

[0065] by Figure 3 For example, for the hash value '010..111000', the first two binary digits of the hash value (i.e., b=2) are used to determine the bucket to which it belongs in the segment. Since the first two digits of the hash value H are '01', it is determined that it belongs to the second bucket (counting starts from 0, and '00' belongs to the first bucket). By adding a bucket design, the maximum detection distance is limited and the system performance is improved. At the same time, in order to further accelerate the insertion and search performance, each time the computing node reads the corresponding bucket in the memory node, it will directly read the complete bucket instead of wasting two remote operations for detection each time. Reading the entire bucket is bound to bring about read amplification, but experimental tests have found that reading the entire bucket greatly improves system performance compared to multiple remote operation detections.

[0066] S102, maintaining a corresponding predictor for each slot, and reading a corresponding bit in the predictor to indicate the position where the slot should be placed during splitting.

[0067] like Figure 4 As shown in the figure, when a bucket in a segment is full and becomes an overflow segment, splitting is triggered. When the local depth of the bucket is less than the global depth (such as Figure 4 (a) The local depth of the yellow bucket is 1 less than the global depth 2), the computing node triggers segment splitting, creates a new segment, and adds one to the local depth of the overflow segment and the new segment. When the local depth of the bucket is equal to the global depth (such as Figure 4 (b) The local depth of the yellow bucket is 2, which is equal to the global depth 2). The computing node triggers the directory split, creates a new subdirectory at the next level, and increases the corresponding global depth by d (the number of subdirectory items is (2 d )), and let all new subdirectory entries point to the overflow segment.

[0068] Specifically, the hash index stores a corresponding predictor in each segment. The predictor contains M×N slots, and the number of slots is equal to the number of key-value pairs that can be accommodated in the segment. The slot of each predictor corresponds to the slot in the corresponding segment, and the predictor slot stores a part of the hash value of the key-value pair in the corresponding segment slot.

[0069] When a computing node attempts to insert a key-value pair into a bucket, but the number of key-value pairs in the bucket reaches the maximum limit it can accommodate, a split will be triggered. At this time, the computing node will read the segment corresponding to the bucket and its corresponding predictor to perform a split operation on it.

[0070] After reading the segment to be split, the compute node will allocate a new segment to accommodate the new key-value pairs.

[0071] The computing node scans each bucket in the segment in order from small to large bucket numbers, and detects whether the slot in each bucket is empty. If it is not empty, it means that the slot stores a pointer to a key-value pair. The corresponding prediction slot in the predictor is searched based on the position of the slot and the position of the bucket.

[0072] like Figure 5 As shown, since the prediction value information corresponding to the key-value pair is stored in the prediction slot, the computing node only needs to access the value of the prediction slot in the predictor, without dereferencing the corresponding pointer, to obtain the first K bits of the key-value pair hash value information, and the Lth bit value being split (the 4th bit in the figure) is determined by the depth of the bucket to indicate the location where the key-value pair pointer is stored when it is split.

[0073] If the location where the key-value pair pointer is stored when it is split is a new bucket, the key-value pair pointer is temporarily written to the local new segment, and after all buckets are traversed, the new segment is written to the remote memory node through a remote write request.

[0074] After writing the new segment, the compute node sets the corresponding directory entry of the new segment to the address of the new segment through a remote compare and exchange (CAS) primitive so that the new segment can be discovered by other compute nodes.

[0075] Finally, the computing node deletes all the key-value pair pointers in the old segment that have been migrated to the new segment through the remote CAS primitive.

[0076] When a computing node is split, the memory node directory will be inconsistent with the cache directories of other computing nodes. Figure 6 As shown, when other computing nodes read inconsistent buckets, they will detect that the local depth of the bucket is inconsistent with the corresponding local depth of the cache directory, and thus realize that the cache subdirectory has expired. At this time, the computing node will read the corresponding subdirectory of the remote memory to overwrite the local cache subdirectory to implement sub-directory level cache update.

[0077] S103, when the hash index introduces a split, a single-threaded memory crawler is started in the memory node to periodically search for unupdated predictors.

[0078] First, the memory node will start a computing core to run the memory crawler process, which is responsible for periodically scanning newly inserted items and checking whether their partial hash keys are generated correctly.

[0079] When the crawler finds that the prediction slot in the predictor is empty, the crawler will check whether the slot in the corresponding bucket is empty. If it is empty, it means that the slot in the bucket has not been occupied and does not need to be updated. If it is not empty, it means that a customer segment has been inserted into the bucket slot but the corresponding prediction slot has not been updated. At this time, the crawler accesses the location pointed to by the pointer in the bucket slot. It will access the corresponding key stored locally in the memory node, calculate its hash value, and write it to the predictor through the local atomic comparison and exchange primitive. For the segment undergoing a resize operation, it will be locked at this time to prevent any modification; in this case, the crawler will postpone the inspection of this segment to the next cycle.

[0080] The compute node helps generate predictors during the resizing process. When resizing, the compute node retrieves the overflowed segments and their associated predictors. During the pointer reallocation process, if the compute node finds that a slot has a key-value pair pointer but its corresponding predictor is empty, it means that the slot is newly inserted and its corresponding predictor has not yet been generated by the crawler. In this case, the compute node obtains the corresponding key-value pair from the memory node according to the key-value pair pointer, calculates the hash value and writes it to the relevant predictor. Finally, the compute node writes the two segments (the original and the newly generated) and the updated predictor back to the memory node.

[0081] S104, when the corresponding computing node successfully obtains the segment lock during the split, the computing node merges multiple remote CAS primitives in the modified directory entry into one request through the doorbell batch processing technology and sends it; in the insertion and update operations, the computing node merges the read bucket and the write key value into one request through the doorbell batch processing technology. Figure 7 As shown in the left figure, in insert and update operations, the hash index combines the reading of buckets and the writing of key values ​​into one request through doorbell batch processing technology.

[0082] The hash index of the present invention further explores the optimization of multiple remote atomic and comparison primitives in splitting. Specifically, when a computing node attempts to perform a size adjustment for a full segment, it must first try to compete for the segment lock. The client that successfully obtains the segment lock will have exclusive write rights to the full segment to modify the pointers in the segment and the associated entries in the subdirectory. Other clients that attempt to split the segment will be blocked in the step of obtaining the segment lock, which means that the client that successfully obtains the segment lock has no competition when writing to the full segment. Therefore, in order to further reduce the overhead caused by the split, this index uses doorbell batch processing technology to further optimize multiple remote operations in the split. The main idea is to merge multiple remote atomic comparison and exchange primitives into one request for batch processing while ensuring correctness. Therefore, only one remote request is needed to complete multiple remote atomic and comparison primitives in the split.

[0083] In the insert and update operations, the computing node uses remote requests to perform the following consecutive actions: read the corresponding bucket from the remote memory node to locate the empty slot; write the key-value pair to the allocated memory pool space; use the remote atomic compare and exchange primitive to write the pointer (that is, the storage address where the item is stored) to the empty slot of the corresponding bucket. This index further combines the bucket read and key write into one request through the doorbell batch processing technology. At the same time, this index further pushes the signal request to the front of the bucket read operation, that is, when the bucket read operation is completed, the remote end will send a completion signal to the computing node to notify the computing node that the operation is completed. Figure 7 As shown in the figure on the right, the compute node uses optimized doorbell batch processing to send requests. Once the memory node network card completes the read operation, it immediately sends a confirmation message instead of waiting for the key-value pair to be written before sending the confirmation message, which shortens the waiting time of the compute node.

[0084] The core of the embodiment of the present invention is to realize high-performance and high-concurrency hash indexes on multiple devices equipped with network cards supporting remote direct memory access. Corresponding to the above-mentioned write-optimized hash index construction method based on separate memory, the write-optimized hash index construction system based on separate memory of the present invention can be implemented by the following modules.

[0085] The node index and directory cache module 801 is used to establish a hash index structure including directories, segments and buckets in the memory node based on a separate memory design, and to establish a directory tree structure including leaf items and non-leaf items, and cache the corresponding directory in the computing node; wherein each bucket has preset slots; the directory entry in the leaf item points to the corresponding segment.

[0086] The node index and directory cache module includes a computing node cache directory unit and a memory node index unit.

[0087] The directory cache of the computing node cache directory unit is established in the computing node. By caching the cache directory consistent with the remote memory node in the computing node, additional memory access to the remote directory is reduced. The cache directory contains all sub-directory items in the memory node and their corresponding metadata. Each sub-directory item stores a 1-bit internal flag and a pointer to the next sub-directory item / segment. If the internal flag value is 1, it means that the sub-directory item is an internal item, and the next pointer is still a sub-directory item; if the internal flag value is 0, it means that the sub-directory item is a leaf item, and the next pointer is a segment, which should be read from the remote end. The metadata stores the local depth of the sub-directory item, which is used to compare the local depth with the bucket read from the remote end to confirm whether the cache directory item is invalid.

[0088] The memory node index unit is responsible for writing the key-value pairs of the storage index as a whole. The memory node index unit mainly consists of three parts: (1) Key-value pair pool. The memory node will create a whole continuous area as the key-value pair pool to store the key-value pairs of the computing node and provide it with search functions. (2) Segment pool. As the key path of the splitting process, remote memory allocation has a high overhead. This index reserves a large enough continuous area in advance and divides it into multiple units according to the segment. When the computing node needs to create a new segment, it only needs to take out a segment corresponding pointer from the segment pool through the remote atomic comparison and exchange primitive. (3) Directory tree index structure, which is used to organize the used segments in the segment pool. The segments are organized in a form similar to a prefix tree. Each sub-directory item can be divided into internal items and leaf items. The internal item points to the next sub-directory, and the leaf item points to the segment. The sliding window is used to traverse from the root node according to the hash value to obtain the location of the final segment. Figure 2 The process of multi-level structure index search is given. At this time, the directory tree index structure includes three levels, and each subdirectory has 4 subdirectory items. The computing node first determines the first-level subdirectory item 00 through the first two hash values, and addresses it to the second-level subdirectory 1 according to its pointer. It determines the second-level subdirectory item 10 through the 3-4-bit hash value and addresses it to the third-level subdirectory 5. Finally, it determines the third-level subdirectory item 11 through the 5-6-bit hash value and addresses it to segment 16.

[0089] The split module 802 is used to maintain a corresponding predictor for each slot, and to read the corresponding bit in the predictor to indicate the position where the slot should be placed during the split.

[0090] The memory node crawler module 803 is used to start a single-threaded memory crawler in the memory node to periodically search for unupdated predictors when a hash index is split.

[0091] The memory node crawler module runs in the memory node and is responsible for completing the predictor values ​​corresponding to the key-value pairs inserted from the computing node. This module mainly consists of two parts: (1) a detection program responsible for routine scanning. This program continuously scans the directory tree of the memory node, extracts the corresponding segment, and traverses all the slots in the segment. If a key-value pair pointer is found in the slot and its corresponding predictor is empty, it is found that the slot needs to be filled with the predictor. (2) A calculation program responsible for completing the predictor. When it is found that the slot needs to be filled with the predictor, the program will access the key-value pair pointer in the slot and extract its corresponding key-value pair, determine the value of its predictor through hash function calculation, and write it into the predictor slot through the local atomic comparison and exchange primitive.

[0092] RDMA network optimization module 804 is used to merge multiple remote CAS primitives in the modified directory entry into one request and send it through the doorbell batch processing technology after the corresponding computing node successfully obtains the segment lock during the split; in the insertion and update operations, the computing node merges the reading bucket and the writing key value into one request through the doorbell batch processing technology.

[0093] The performance tests of the present invention are given as follows.

[0094] The prototype system of the present invention is deployed on five servers, each of which is equipped with two 2.40GHz Intel Xeon Silver 4314 computing cores (16 cores each), 256GB of DRAM memory and a 100Gbps Mellanox sixth-generation network card. Each network card is connected to a 100Gbps Mellanox SN2700 switch, running the Ubuntu 20.04 operating system. One server is used as a memory node, and the remaining four servers are used as computing nodes. Specifically, the processes in the memory node are pinned to a single CPU core to simulate close to zero computing power in the memory pool. In order to mitigate the impact of cross-memory access, all threads are pinned to the local computing core to which the network card connection is connected. The key of the key-value pair is set to 16 bytes and the value is set to 32 bytes in the experiment. The test system includes: (1) FaRM, a linked associative hopscotch hash, where each key is stored in a bucket that hashes to or from a neighbor bucket; (2) DrTM, a chained hash, where once it cannot find an empty slot for the inserted key, it will create a new bucket in the hash bucket linked to the key to store the incoming item; (3) RACE, the first hash index designed based on the separate memory feature, which caches the entire directory directly in the computing node; (4) SepHash, a two-layer structure hash index that uses a single remote write operation to merge a small unsorted compressed segment with a larger sorted body, thereby reducing the adjustment overhead; (5) WaSH, a write-optimized hash index based on separate memory, i.e., the present invention. In the absence of a specification for the number of key-value pairs, each test operation defaults to preheating 10 million key-value pairs and performing 100 million operations, repeating the test 5 times, and taking the average as the test result.

[0095] (1) System scalability performance test

[0096] The scalability performance test generates loads of write, positive search, negative search, and delete operations through YCSB. The test evaluates the system throughput of the present invention and other comparison systems in executing four operation scenarios under different threads under balanced and skewed data distribution. The test results are shown in the figure. Fig. 9 As shown, the upper figure shows the throughput under balanced load, and the lower figure shows the throughput under tilted load. In the four operation scenarios, compared with FaRM, DrTM, RACE, and SepHash, WaSH has an average improvement of 13.53 times, 1.90 times, 3.46 times, and 1.57 times. The reason is that WaSH divides the entire directory into multiple subdirectories to reduce synchronization traffic and speeds up the resizing process by using a predictor that avoids reading key-value pairs from remote memory nodes.

[0097] (2) Mixed load simulation performance test

[0098] The real load simulation performance test uses the YCSB tool to simulate 5 real load scenarios with different configurations. The test results are as follows: Fig.10 In all scenarios, the throughput of WaSH is 5.98 times, 6.16 times, 5.54 times, 1.68 times, and 1.85 times higher than that of the existing hash indexes.

[0099] (3) Real-time throughput test

[0100] The real-time insertion performance is further studied as follows, and the stability of the hash index is analyzed. Fig.11 The volatility ratio, which is the ratio of real-time throughput to average throughput, is shown for each hash index. Therefore, a volatility ratio closer to 1 means more stable performance. It can be seen that WaSH achieves the most stable performance compared to the other four hash indices. Specifically, the volatility ratio of WaSH varies between 60.9% and 145.1% because it significantly reduces synchronization traffic and uses a predictor to reduce round-trip latency. FaRM has the highest volatility ratio because it needs to rearrange all key-value pairs during the resizing process. Although SepHash remained stable for a period of time (from 6.25 seconds to 33.00 seconds), significant fluctuations occurred when splitting the main segment, which required retrieving the main segment and key-value pairs from the memory node for rehashing.

[0101] (4) Delay distribution test

[0102] The test evaluates the latency of the hash index by running 256 items to insert 100 million key-value pairs. Fig.12 The cumulative probability distribution of the latency for inserting a key-value pair is shown. Compared with FaRM, DrTM, RACE, and SepHash, WaSH reduces the average latency by 75.8%, 56.7%, 76.3%, and 71.8%, respectively, and the 95% latency by 67.1%, 60.0%, 79.2%, and 79.9%, respectively. In addition, FaRM causes the longest tail latency (31.7s) because it needs to double the entire hash table during resizing, thus blocking incoming insertion operations.

[0103] (5) Synchronous flow test

[0104] The test measures the synchronous traffic of WaSH by changing the number of clients. The test results are as follows: Fig.13As shown. Compared with RACE, WaSH significantly reduces 99.7% of the synchronization traffic. These reductions are due to the multi-level index structure in WaSH, which only needs to transmit the newly added subdirectories for directory synchronization. In contrast, RACE needs to retrieve the entire directory from the remote memory node, which introduces overwhelming synchronization traffic during insertion, which also makes RACE inefficient in mixed workloads. When there are 256 clients, SepHash introduces 5.19× more synchronization traffic than WaSH because it maintains a large amount of additional metadata. FaRM and DrTM have zero synchronization traffic because they directly access the remote hash table instead of caching it on the client; however, they require more remote operations to access key-value pairs, resulting in low search throughput.

[0105] (6) Using predictors to reduce dereference traffic during splitting

[0106] To investigate the effect of the predictor on the resizing traffic, two variants of WaSH, named WaSH-predictor and WaSH-predictor-disabled, were generated and 100 million insertions were performed using 256 clients. Two metrics were measured: (1) traffic, which includes the traffic size of read key-value pairs and the predictor read to help reposition pointers during resizing; (2) prediction rate, which represents the ratio of key-value pair pointers repositioned by the predictor to the key-value pair pointers stored in the overflow segment. The test results are shown in Figure 2. Fig.14 The prediction rate increases from 34.6% to 62.1% and then remains stable after 30 million insertions. The reason is that WaSH uses a memory crawler to verify that the hash key is generated correctly before the first resize, and it supports the client to generate predictors to increase the prediction rate.

[0107] (7) Design decomposition performance test

[0108] In order to analyze the contribution of key technologies, four WaSH variants with different technologies are generated: (1) WaSH-Base, which only retains the multi-level index structure; (2) WaSH-C, which further uses adjustment-assisted updates to generate predictors; (3) WaSH-CM, which uses adjustment-assisted updates and memory crawlers to generate predictors; (4) WaSH-CMR, which is a combination of the above three technologies and optimizations for RDMA network characteristics. The test results are shown in Figure 2. Fig.15 As shown. Compared with WaSH-Base, WaSH-C reduces the 99.9% latency by 28.0% and increases the average throughput by 13.6%. Compared with WaSH-C, WaSH-CM further improves the average throughput by 17.7%, and reduces the average latency and 99.9% latency by 15.1% and 49.0%, respectively. Finally, WaSH-CMR reduces the 99.9% latency by 16.9% compared with WaSH-CM.

[0109] As mentioned above, the existing hash index-based approach does not consider the RDMA network characteristics and computing node cache characteristics. Since the computing node cache needs to update the entire flat directory, it leads to higher update traffic and poorer write performance. The present invention adopts a multi-level cache structure to reduce the update intensity to the sub-directory level, and uses RDMA optimization requests to further optimize the read and write process.

[0110] The principle and operation mode of the present invention are described through the above-mentioned specific implementation cases. These cases are intended to provide readers with a clear understanding framework to grasp the core ideas and operation points of the present invention. However, it should be clear that these cases are not limitations on the scope of application of the present invention. For professionals in this field, various forms of improvements and innovations based on the core ideas of the present invention should be included in the protection scope of the present invention.

Claims

1. A write-optimized hash index construction method based on separate memory, characterized in that: include: S101, based on the separated memory design, a hash index structure including directories, segments and buckets is established in the memory node, and a directory tree structure including leaf items and non-leaf items is established, and the corresponding directory is cached in the computing node; wherein each bucket has preset slots; the directory item in the leaf item points to the corresponding segment; S102, maintaining a corresponding predictor for each slot, and reading a corresponding bit in the predictor to indicate the position where the slot should be placed when splitting; S103, when the hash index introduces a split, a single-threaded memory crawler is started in the memory node to periodically search for unupdated predictors; S104, when the corresponding computing node successfully obtains the segment lock during the split, the computing node merges multiple remote CAS primitives in the modified directory entry into one request through the doorbell batch processing technology and sends it; in the insertion and update operations, the computing node merges the reading bucket and the writing key value into one request through the doorbell batch processing technology.

2. The write-optimized hash index construction method based on separate memory according to claim 1, characterized in that: The S101 specifically includes: S1011, establish a hash index structure consisting of a directory, a segment, and a bucket in the memory node; wherein the directory contains directory entries, each directory entry points to a corresponding segment, the segment contains a fixed number of M buckets, each bucket has N slots, and each slot is used to store a pointer to a key-value pair; S1012, establishing a directory tree structure in the memory node; wherein the directory tree includes leaf items and non-leaf items, the leaf items and the non-leaf items include the same number of directory items, the non-leaf item directory item points to the next leaf item, and the directory item in the leaf item points to the corresponding segment; S1013, cache the corresponding directory in the computing node.

3. The write-optimized hash index construction method based on separate memory according to claim 2 is characterized in that: The S102 specifically includes: S1021, storing a corresponding predictor in each segment, the predictor including M×N slots, the number of slots being equal to the number of key-value pairs that can be accommodated in the segment, the slots of each predictor corresponding to the slots in the corresponding segment, and the predictor slot storing a part of the hash value of the key-value pair in the slot of the corresponding segment; S1022, when a computing node attempts to insert a key-value pair into a bucket, but the number of key-value pairs in the bucket reaches the maximum upper limit, a split is triggered. At this time, the computing node reads the segment corresponding to the bucket and the predictor corresponding to the segment to perform a split operation on the segment; S1023, after reading the segment to be split, the computing node allocates a new segment to accommodate the new key-value pair; S1024, the computing node scans each bucket in the segment in ascending order of bucket numbers, and detects whether the slot in each bucket is empty. If it is not empty, it means that the slot stores a pointer to a key-value pair, and searches for the corresponding prediction slot in the predictor according to the position of the slot and the position of the bucket; S1025, since the prediction slot stores information about the corresponding hash value of the key-value pair, the information about the corresponding hash value of the key-value pair is used to indicate the location where the key-value pair pointer is stored when it is split; S1026, if the location where the key-value pair pointer is stored when it is split is a new bucket, the key-value pair pointer is temporarily written to the local new segment, and after all buckets are traversed, the new segment is written to the remote memory node through a remote write request; S1027, after writing the new segment, the computing node sets the directory entry corresponding to the new segment to the address of the new segment through a remote compare and exchange CAS primitive so that the new segment can be discovered by other computing nodes; S1028, finally the computing node deletes all key-value pair pointers in the old segment that have been migrated to the new segment through the remote comparison and exchange CAS primitive.

4. The write-optimized hash index construction method based on separate memory according to claim 3 is characterized in that: The S103 specifically includes: S1031, the memory node starts a memory crawler program, and the memory crawler program continuously scans the predictor part of each segment in the entire hash table; S1032, when the crawler finds that the prediction slot in the predictor is empty, it checks whether the slot in the corresponding bucket is empty. If it is empty, it means that the slot in the bucket has not been occupied and does not need to be updated. If it is not empty, it means that a client segment has been inserted into the bucket slot but the corresponding prediction slot has not been updated. At this time, the crawler accesses the position pointed to by the pointer in the bucket slot, reads the corresponding key value and calculates it, intercepts part of the hash value and stores it in the prediction slot; S1033, when the computing node has been inserted into the bucket slot but the memory crawler has not scanned the corresponding bucket slot, the computing node adopts split-based predictor maintenance. When the computing node splits, it is found that there is a pointer in the bucket slot but the corresponding prediction slot is empty, then it is considered that the memory crawler has not had time to update the corresponding prediction slot; at this time, the computing node reads the corresponding key-value pair from the remote end for hash calculation and writes the intercepted part of the hash value into the prediction slot through the remote comparison and exchange CAS primitive to complete the prediction slot update.

5. The write-optimized hash index construction method based on separate memory according to claim 4 is characterized in that: The S104 specifically includes: S1041, the computing node combines multiple remote CAS primitives in the modified directory entry into one request through the doorbell batch processing technology and sends the request; S1042, in the insert and update operations, the computing node merges the read bucket and write key value into one request through the doorbell batch processing technology, and pushes the signal request to the front of the read bucket operation. That is, when the read bucket operation is completed, the remote end will send a completion signal to the computing node to notify the computing node that the operation is completed.

6. A write-optimized hash index construction system based on separate memory, characterized in that: include: The node index and directory cache module is used to establish a hash index structure including directories, segments and buckets in the memory node based on the separated memory design, and to establish a directory tree structure including leaf items and non-leaf items, and cache the corresponding directory in the computing node; wherein each bucket has preset slots; the directory item in the leaf item points to the corresponding segment; A split module, used for maintaining a corresponding predictor for each slot, and indicating the position where the slot should be placed during splitting by reading the corresponding bit in the predictor during splitting; The memory node crawler module is used to start a single-threaded memory crawler in the memory node to periodically search for unupdated predictors when the hash index introduces splits; The RDMA network optimization module is used to merge multiple remote CAS primitives in the modified directory entry into one request and send it through the doorbell batch processing technology after the corresponding computing node successfully obtains the segment lock during the split; in the insertion and update operations, the computing node merges the reading bucket and the writing key value into one request through the doorbell batch processing technology.

Images