Hybrid memory based numa-aware key-value store system and operating method

Abstract

The application discloses a hybrid memory-based NUMA-aware key-value storage system and an operating method, and belongs to the field of key-value storage. The system comprises a plurality of NUMA nodes; in the PM of each NUMA node, key-value pair data is written in the form of log items into a log file; a first storage area in the DRAM of each NUMA node constitutes a volatile index layer and is used for storing the address information of the log item where the key and the latest value are located; a second storage area in the DRAM of each NUMA node constitutes a RAF cache and is used for caching key-value pair data; the higher the access frequency and the higher the delay of the NUMA node initiating an access request to access the PM of the NUMA node where the key-value pair data is located, the greater the corresponding cache benefit; when the RAF cache is replaced, the entry with the smallest cache benefit is selected for replacement. The application can optimize the PM access performance across NUMA nodes and provide efficient key-value storage services.

Description

Technical Field

[0001] This invention belongs to the field of key-value storage, and more specifically, relates to a NUMA-aware key-value storage system and its operation method based on hybrid memory. Background Technology

[0002] Key-value stores are database management systems that store data based on key-value pairs. They identify each data item with a unique key and store the actual data content using the value associated with the key. Their data model is simple and flexible, highly scalable, and can achieve low-latency and high-throughput data access. Therefore, they are the core components of many storage systems or large-scale distributed systems, including system caches, data storage engines, and metadata management.

[0003] With the development of processor technology and the emergence of multi-core processors, multiprocessor systems have become a common way to improve computing performance. In traditional symmetric multiprocessor (SMP) systems, memory access latency has become a major factor limiting system performance and scalability. Non-uniform memory access (NUMA) architecture reduces memory contention in multiprocessor systems by providing local memory for each processor, thus providing stronger horizontal scalability. Compared to Dynamic Random Access Memory (DRAM), Persistent Memory (PM) has advantages such as persistence, byte addressability, low power consumption, and low cost. Furthermore, it can provide sub-microsecond access latency through a directly connected memory bus and load / store instructions. This has prompted many researchers to actively explore the possibilities of hybrid memory composed of PM and DRAM in key-value storage.

[0004] In NUMA architecture, all computing and storage resources are organized across different NUMA nodes. These nodes communicate internally via an integrated memory controller bus and inter-node via high-speed inter-node links (e.g., UPI). Compared to a single CPU, NUMA architecture offers significantly more CPU computing and memory capacity. However, the performance of a CPU accessing a memory module (PM) in a remote NUMA node is far worse than accessing a PM in a local NUMA node. Due to the additional access latency to remote node memory in NUMA architecture, cross-NUMA node PM access becomes a bottleneck for the overall performance and system scalability of existing PM-based key-value stores. Therefore, researching a DRAM-PM hybrid key-value store suitable for NUMA architecture is of great significance.

[0005] Currently, key-value stores based on inconsistent hybrid memory are still in the early stages of research. Existing NUMA architecture solutions have proposed multi-replica techniques, space allocation techniques, hot data optimization techniques, multi-head log techniques, and write buffer techniques, but many problems still exist. For example, in multi-replica techniques, a single write operation often requires multiple updates, which is very unfriendly to PMs with low write performance and the need to consider data consistency. Space allocation techniques, such as PACTree, prioritize the allocation of local NUMA node resources, but their effect on reducing remote access is limited under complex user operation loads. Hot data optimization techniques maintain a cache of hot data across NUMA nodes in DRAM and flush it to the PM every time the hot dataset switches, but this not only makes it difficult to completely eliminate remote access, but also introduces additional hot dataset migration overhead. In addition, it is not effective for workloads where hotspots are not obvious. Multi-head log schemes, such as nibble, allocate local log space for each CPU core to avoid remote writes, but garbage collection requires frequent access to the upper-level hash index, interfering with foreground operations and not optimizing remote reads. Write buffer schemes, such as F3-tree, maintain a doubly linked list as a write buffer for each thread in each NUMA PM. The write request is first written to the local write buffer, and then flushed to the remote by a background thread. However, a large number of remote accesses are still generated during the background flushing process, and the use of a doubly linked list to organize the data in the write buffer consumes a lot of PM space.

[0006] In summary, key-value stores based on inconsistent hybrid memory are currently limited by the high overhead of remote access in NUMA architectures, and existing solutions suffer from write amplification and resource waste. Summary of the Invention

[0007] To address the shortcomings and improvement needs of existing technologies, this invention provides a NUMA-aware key-value storage system and operation method based on hybrid memory. Its purpose is to optimize PM access performance across NUMA nodes and solve the problem that efficient key-value storage services cannot be provided in DRAM-PM hybrid memory under NUMA architecture.

[0008] To achieve the above objectives, according to one aspect of the present invention, a NUMA-aware key-value storage system based on hybrid memory is provided, comprising: multiple NUMA nodes; each NUMA node includes computing resources and memory resources, the memory resources including DRAM and PM; the PM of each NUMA node stores a log file for recording log entries in append form; each log entry includes key-value pair data and corresponding write operations; the write operations include insert operations, update operations, and delete operations.

[0009] The key-value storage system maintains a volatile index layer and a RAF cache shared by all NUMA nodes;

[0010] The volatile index layer consists of the first memory region in the DRAM of each NUMA node, which is used to store the index data of each key-value pair; the index data includes the key and the address information of the log entry where the latest value is located;

[0011] The RAF cache consists of a second memory region in the DRAM of each NUMA node. Each cache entry is used to cache a key-value pair of data. The higher the access frequency of the cached key-value pair of data and the higher the latency of the NUMA node that initiates the access request to access the PM of the NUMA node where the key-value pair of data is located, the higher the cache benefit of the corresponding cache entry. Furthermore, when the RAF cache is replaced, the cache entry with the lowest cache benefit will be selected for replacement.

[0012] Furthermore, the RAF cache also maintains a delay set L. i ={l i-x ,1≤x,i≤n∩x,i∈N}, where n represents the total number of MUMA nodes, l i-x This represents the latency of accessing the PM in the x-th NUMA node from the i-th NUMA node;

[0013] Furthermore, for cache entry C in the RAF cache, when key-value pair data kv is written to cache entry C, its cache benefit will be initially set to... j and k represent the NUMA node number that requests access to kv and the NUMA node number that stores kv, respectively.

[0014] Each time a request to access key-value pair data (kv) hits cache entry C, the cache benefit of cache entry C will be updated to... TF old and TF new These represent the cache gains before and after the update, respectively.

[0015] Furthermore, the RAF cache is a hash table, and each entry is a hash bucket consisting of a fixed number of slots. Each slot is used to store a cache entry. The information recorded in the cache entry includes: the key, value, cache reward (TF) of the key-value pair data, the id of the NUMA node that stores the key-value pair data, and the lock. The hash bucket where the cached key-value pair data is located is addressed by the first hash value of the key.

[0016] Furthermore, the log files stored in the PM of the NUMA node are divided into multiple hash groups that correspond one-to-one with the second hash value of the key, and only one log file exists in each hash group as the active file;

[0017] When a log entry is written to PM, it is appended to the active file in the corresponding hash group according to the second hash value of the key; once the active file is full, it will be archived and cannot be written to or modified, and a new active file will be assigned to the hash group.

[0018] The key-value pairs within each hash group share a sequentially incrementing version number variable. Each time a log entry is appended to a hash group, the version number variable within that hash group is incremented.

[0019] Furthermore, the address information of the log entry includes: the length of the log entry, denoted as l_len; the ID of the NUMA node of the log entry, denoted as numa_id; the ID of the log file where the log entry is located within the hash group, denoted as log_file_id; and the offset of the log entry within the log file, denoted as offset;

[0020] Furthermore, for key-value pairs where both the key and value lengths remain unchanged, the log entry specifically includes: the operation type, denoted as Op; the key and value of the key-value pair, denoted as Key and Value respectively; the version number of the value, denoted as Version; and the address information of the log entry containing the previous version of the value corresponding to the key, denoted as h_addr.

[0021] For key-value pairs where the key length is constant but the value length is variable, the log entry specifically includes: the operation type, denoted as Op; the key and value of the key-value pair, denoted as Key and Value respectively; the version number of the value, denoted as Version; the length of the value, denoted as V_len; and the address information of the log entry containing the previous version of the value corresponding to the key, denoted as h_addr.

[0022] For key-value pairs where both the key and value lengths are variable, the log entry contains: the operation type, denoted as Op; the key and value of the key-value pair, denoted as Key and Value respectively; the version number of the value, denoted as Version; the length of the key, denoted as K_len; the length of the value, denoted as V_len; and the address information of the log entry containing the previous version corresponding to the key, denoted as h_addr.

[0023] According to another aspect of the present invention, a key-value operation method is provided, which is executed in the above-described NUMA-aware key-value storage system based on hybrid memory provided by the present invention;

[0024] Key-value operation methods include: receiving operation requests from the current load, parsing the operation type, and then executing the corresponding operation;

[0025] Operation types include: single-point read operation, range read operation, insert operation, update operation, and delete operation;

[0026] The execution of a single-point read operation includes:

[0027] Step R1: If RAF caching is enabled, proceed to step R2; otherwise, proceed to step R4.

[0028] Step R2: Based on the key in the operation request r Query the RAF cache; if a match is found, return the corresponding value directly. r The corresponding cache reward is updated, and the single-point read operation ends; if a cache miss occurs, proceed to step R3.

[0029] Step R3: Locate the volatile index layer to obtain the corresponding value. r The address information of the log entry is retrieved, and the log entry is read to obtain the value. r And return, and set the key. r Sum of values r The key-value pairs are cached in the RAF cache, and the single-point read operation ends.

[0030] Step R4: Locate the volatile index layer to obtain the corresponding value. r The address information of the log entry is retrieved, and the log entry is read to obtain the value. r And return, the single-point read operation ends;

[0031] The execution of a range read operation includes:

[0032] Step S1: Query all keys within the target key range in the volatile index layer and obtain the corresponding address information;

[0033] Step S2: Parse the NUMA node where each key-value pair data is located from the address information read in step S1, and assign the key-value pair data reading task to the corresponding NUMA node;

[0034] Step S3: Create multiple local background read threads on each NUMA node. These local background read threads execute key-value pair data reading tasks in parallel within the corresponding NUMA node and return the read values.

[0035] The execution of an insertion operation includes:

[0036] Step I1: Calculate the key-value pair data kv to be inserted. i middle key i The second hash value is used to determine the key-value pair data kv. i The target hash group to be inserted;

[0037] Step I2: Create a key-value pair data kv file in the active file of the target hash group of the target NUMA node for the insertion operation. i Allocate space for log entries;

[0038] Step I3: Construct key-value pair data (kv) i The corresponding log entry; in the constructed log entry, h_addr is set to invalid, and Op is the opcode corresponding to the insertion operation;

[0039] Step I4: Append the constructed log entries to the activity file;

[0040] The execution of the update operation includes:

[0041] Step U1: For the key-value pair data (kv) to be updated u Query key-value pair data (kv) in the volatile index layer. u key u The corresponding address information v_addr u ;

[0042] Step U2: Calculate the key u The second hash value is used to determine the key-value pair data kv. u The corresponding hash group, and the key-value pair data kv in the active file of that hash group. u Allocate space for log entries;

[0043] Step U3: Construct key-value pair data (kv) u The corresponding log entry; in the constructed log entry, h_addr is set to v_addr. u Op is the opcode corresponding to the update operation;

[0044] Step U4: Append the constructed log entry to the activity file and update the address information of the log entry to the volatile index layer;

[0045] The deletion operation includes:

[0046] Step D1: Query the target key in the volatile index layer. d The corresponding address information v_addr d ;

[0047] Step D2: Calculate the target key d The corresponding second hash value is used to determine the hash group where the key-value pair data to be deleted is located, and log entry space is allocated for the key-value pair data to be deleted in the active file of that hash group;

[0048] Step D3: Construct the log entry corresponding to the key-value pair data to be deleted; in the constructed log entry, h_addr is set to v_addr. d Op is the corresponding opcode for deletion;

[0049] Step D4: Append the constructed log entries to the activity file and add the target key from the volatile index layer. d The corresponding index data.

[0050] Furthermore, the key-value operation method provided by the present invention further includes: determining the type of the current load while parsing the operation type;

[0051] If the current load type is read-uniform load or write-intensive load, and RAF caching is enabled, then disable RAF caching;

[0052] If the current load type is neither a read-uniform load nor a write-intensive load, and RAF caching is disabled, then enable RAF caching.

[0053] Furthermore, the key-value operation method provided by the present invention further includes: periodically performing log item marking operations within each NUMA node; the execution of the log item marking operation includes:

[0054] Step L1: Within the current NUMA node, randomly select a hash group and traverse the log entries within it. For the currently traversed log entry E, if it is a valid log entry, proceed to step L2; otherwise, mark the current log entry as finished.

[0055] Step L2: If the operation type of the current log item E is an update operation or a delete operation, proceed to step L3; if the operation type of the current log item E is an insert operation, proceed to step L4.

[0056] Step L3: Search the historical operations of log entry E based on h_addr until an invalid log entry with h_addr is encountered, or an invalid log entry, or a log entry pointed to by h_addr has been reclaimed. Mark all the searched log entries as invalid.

[0057] Step L4: If the current log entry E is not referenced by the volatile index layer, then mark the current log entry E as invalid;

[0058] The key-value operation method also includes: when the number of invalid log entries in a log file exceeds a preset threshold, the valid log entries in the log file are appended to the active file in the same hash group, and the log file is deleted.

[0059] Furthermore, the key-value operation method provided by the present invention further includes: before shutting down the key-value storage system, storing the index data in the volatile index layer in a preset location in the PM, and persisting a flag bit in the PM to indicate whether the normal key-value storage system has been shut down normally.

[0060] Furthermore, the key-value operation method provided by the present invention also includes: reading index data and flag bit from PM when the key-value storage system starts up;

[0061] If the flag indicates that the key-value storage system has been shut down normally, the read index data will be loaded into the volatile index layer.

[0062] If the flag indicates that the key-value pair storage system has not been shut down properly, then within each NUMA node, a scan thread is allocated for each hash group to scan the log files within the corresponding hash group in order to rebuild the volatile index layer shown.

[0063] When the scanning thread scans the log file, for log entries with the same key, it only retains the log entry with the latest version number. After the scan is completed, it builds corresponding index data based on the retained log entries.

[0064] In summary, the above-described technical solutions conceived in this invention can achieve the following beneficial effects:

[0065] (1) This invention establishes a cross-NUMA node RAF cache in DRAM above the volatile index layer to cache key-value pair data. Accordingly, it proposes a method for calculating cache benefit, that is, the higher the access frequency of the cached key-value pair data and the higher the latency of the NUMA node initiating the access request to access the PM of the NUMA node where the key-value pair data is located, the higher the cache benefit of the corresponding cache entry will be. When performing cache replacement, the cache entry with the smallest cache benefit will be selected for replacement first. This caching mechanism and cache replacement strategy make key-value pair data that need to be accessed remotely will be cached first, realizing NUMA-aware key-value storage. Thus, the corresponding access request can be completed directly by accessing the cache without remote access, which effectively improves the access performance of the PM across NUMA nodes and solves the problem that DRAM-PM hybrid memory under NUMA architecture cannot provide efficient key-value storage services.

[0066] (2) In a preferred embodiment of the present invention, each time a cache hit occurs, specifically according to... Update the cache benefit of the hit cache entry, where TF old and TF new These represent the cache gains before and after the update, respectively. k-k This indicates local access latency. By maintaining the caching benefits of each cached entry in this way, it is possible to accurately identify key-value pairs that are frequently accessed and require remote access with low complexity, and then cache them.

[0067] (3) In a preferred embodiment of the present invention, a hash table is used to implement the RAF cache. Each table entry is a hash bucket consisting of a fixed number of slots. Each slot is used to store a cache entry, thereby enabling direct access to any position in the cache. Furthermore, the corresponding hash bucket can be located by calculating the hash value of the key, which greatly improves the read and write efficiency of the cache.

[0068] (4) In a preferred embodiment of the present invention, log entries are divided into different hash groups according to the hash value of the key, and each hash group has an independent active file. This allows the active files in multiple hash groups to be accessed concurrently by multiple threads. Since there are no duplicate keys between hash groups, write competition only occurs within the hash group, which effectively improves the concurrency of write operations.

[0069] (5) In a preferred embodiment of the present invention, a concise format is set for the address information of log entries and various types of log entries, thereby enabling efficient completion of subsequent key-value operations.

[0070] (6) The key-value operation method provided by this invention, based on the NUMA-aware key-value storage system based on hybrid memory provided by this invention, can efficiently perform various key-value operations. Specifically, when performing a single-point read operation, the RAF cache is queried first. Only when the RAF cache is disabled or a cache miss occurs is the volatile index layer queried, thereby effectively reducing remote access and improving the efficiency of single-point read operations. When performing a range read operation, this invention obtains the address of the log item containing all the key-value pairs to be read by accessing the volatile index layer, and then allocates the corresponding single-point read operation task to the corresponding NUMA node. On the NUMA node, multiple local background read threads execute the corresponding key-value pair read task in parallel, thereby effectively improving the efficiency of range read operations.

[0071] (7) When using RAF caching to cache key-value pairs, additional information needs to be maintained and calculated, which will generate certain overhead. In a uniform read load, each data is read evenly, and due to the limited cache space, cache replacement will occur frequently, which will cause more additional overhead. In a write-intensive load, there are fewer read operations, and RAF caching will not bring significant benefits. Based on these considerations, the key-value operation method provided by the present invention, in its preferred embodiment, determines the type of load while processing the load, and disables RAF caching when the load is determined to be a uniform read load or a write-intensive load, which can avoid the advantages of RAF caching from failing to offset the additional overhead and adapt to different load environments.

[0072] (8) The key-value operation method provided by the present invention will periodically mark invalid log entries in the log files within the node, and when there are many invalid log entries in a certain log file, the valid log entries will be migrated out and the log file will be deleted, thereby effectively improving the utilization rate of PM storage space in NUMA nodes.

[0073] (9) The key-value operation method provided by the present invention persists the index data in the volatile index layer to PM before the system is shut down, and persists the corresponding flag bit to indicate whether the key-value storage system is shut down normally. When the system is started, the volatile index layer is loaded or rebuilt according to the flag bit, thereby effectively ensuring the consistency of memory in the system. Attached Figure Description

[0074] Figure 1 This is a schematic diagram of the NUMA-aware key-value storage system structure based on hybrid memory provided in Embodiment 1 of the present invention;

[0075] Figure 2 This is a schematic diagram of the RAF cache structure provided in Embodiment 1 of the present invention;

[0076] Figure 3 This is a schematic diagram of the address and log data layout provided in Embodiment 1 of the present invention;

[0077] Figure 4 This is a schematic diagram of the log item location process provided in Embodiment 1 of the present invention;

[0078] Figure 5 This is a schematic diagram of the read request processing method provided in Embodiment 2 of the present invention;

[0079] Figure 6 This is a schematic diagram of the write request processing method provided in Embodiment 2 of the present invention;

[0080] Figure 7 This is a schematic diagram of the invalid log entry recovery process provided in Embodiment 2 of the present invention;

[0081] Figure 8 This is a schematic diagram of the multi-threaded recovery process provided in Embodiment 2 of the present invention. Detailed Implementation

[0082] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0083] In this invention, the terms "first," "second," etc. (if present) in the invention and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0084] To address the limitations of existing key-value stores in NUMA architectures due to high remote access overhead, which hinders efficient key-value storage services, this invention provides a NUMA-aware key-value storage system and its operation method based on hybrid memory. The overall idea is to leverage the characteristic that in a NUMA architecture, the latency of PM access across NUMA nodes is much greater than the latency of local PM access within a NUMA node. Building upon the existing key-value index, a NUMA-aware RAF cache is introduced. This RAF cache caches frequently accessed data while simultaneously caching key-value pairs that require cross-NUMA node access, thereby reducing the number of PM accesses across NUMA nodes and improving the key-value storage performance of the key-value storage system.

[0085] The following is an example.

[0086] A NUMA-aware key-value store system based on hybrid memory, such as Figure 1 As shown, it includes: multiple NUMA nodes; NUMA nodes include computing resources and memory resources, and memory resources include DRAM and PM; the DRAM in all NUMA nodes constitutes the volatile memory of the key-value storage system, and the PM in all NUMA nodes constitutes the persistent memory of the storage system.

[0087] The PM of a NUMA node stores log files for recording log entries in append-only format. Log entries include key-value pairs and corresponding write operations. Write operations include insert, update, and delete operations.

[0088] The key-value storage system maintains a volatile index layer and a RAF cache shared by all NUMA nodes;

[0089] The volatile index layer consists of the first memory region in the DRAM of each NUMA node, which is used to store the index data of each key-value pair; the index data includes the key and the address information of the log entry where the latest value is located;

[0090] The RAF cache consists of a second memory region in the DRAM of each NUMA node. Each cache entry is used to cache a key-value pair of data. The higher the access frequency of the cached key-value pair of data and the higher the latency of the NUMA node that initiates the access request to access the PM of the NUMA node where the key-value pair of data is located, the higher the cache benefit of the corresponding cache entry. Furthermore, when the RAF cache is replaced, the cache entry with the lowest cache benefit will be selected for replacement.

[0091] like Figure 1 As shown, the key-value storage system provided in this embodiment is logically divided into three layers: a top-level RAF cache, a middle-level volatile index layer, and a bottom-level log storage structure Nlog. The following provides a detailed description of each layer.

[0092] In this embodiment, the RAF cache is located in DRAM. Its function is to cache key-value pairs that are accessed frequently but have high access latency, thereby reducing the number of times the PM is accessed across NUMA nodes. To this end, this embodiment proposes a caching mechanism and cache replacement strategy suitable for NUMA architecture. Specifically, the RAF cache also maintains a latency set L. i ={l i-x ,1≤x,i≤n∩x,i∈N}, where n represents the total number of MUMA nodes, l i-x This represents the latency of accessing the PM in the x-th NUMA node from the i-th NUMA node;

[0093] Furthermore, for cache entry C in the RAF cache, when key-value pair data kv is written to cache entry C, its cache benefit will be initially set to... j and k represent the NUMA node number that requests access to kv and the NUMA node number that stores kv, respectively.

[0094] Each time a request to access key-value pair data (kv) hits cache entry C, the cache benefit of cache entry C will be updated to... TF old and TF new These represent the cache gains before and after the update, respectively.

[0095] It's easy to understand that when j = k, access to the key-value pair data kv is a local access; when j ≠ k, access to the key-value pair data kv is a remote access. The latency of remote access is significantly greater than that of local access. Therefore, when a cached entry is accessed remotely, its caching benefit becomes greater. Furthermore, the corresponding caching benefit is updated each time a cached entry is hit. Therefore, the caching benefit updated in this way also includes the access frequency information of the key-value pair data. In summary, this caching benefit update method can accurately identify and cache frequently accessed key-value pairs that require remote access with low complexity.

[0096] As an optional implementation method, in this embodiment, such as Figure 2As shown, the RAF cache is a hash table, and each entry is a hash bucket consisting of a fixed number of slots. Each slot is used to store a cache entry. The information recorded in the cache entry includes: the key, value, cache reward (TF) of the key-value pair data, the id (denoted as numa_id) of the NUMA node that stores the key-value pair data, and the lock. The hash bucket where the cached key-value pair data is located is addressed by the first hash value of the key.

[0097] The lock in the cache entry is used for concurrency control. When modifying the corresponding slot, the lock must be acquired first, and then the lock can be released after the modification is completed.

[0098] This embodiment also includes: when a cache miss occurs, if there are empty slots in the corresponding hash bucket, the corresponding key-value pair data is cached; when a cache miss occurs and there are no empty slots in the corresponding hash bucket, the key-value pair data with the lowest cache yield (TF) in the corresponding hash bucket is selected to replace the requested key-value pair data and then cached.

[0099] When the TF value of a certain slot reaches the upper limit, in order to prevent overflow, the TF value of all data in the same hash bucket is halved. In order to reduce the impact of too frequent TF value updates on the performance of front-end operations, this embodiment also adopts a periodic sampling strategy, that is, after a fixed number of lookup operations (32 times by default), the TF value of all cache entries in the entire RAF cache is reduced proportionally.

[0100] Since using RAF to cache data requires maintaining and calculating additional information, under certain loads, the advantages of RAF caching may not outweigh the overhead of maintaining and calculating additional information. To avoid this, in this embodiment, RAF caching can be disabled or enabled based on the actual load characteristics. Specifically, RAF caching will be disabled when one of the following conditions is met:

[0101] (1) If the number of read operations received by the RAF cache is less than a certain percentage (e.g., 10%) of all read operations, it means that the data in the load will be read evenly. This load is a read-uniform load. In this case, using the RAF cache for data caching will result in a low hit rate.

[0102] (2) All cached data in the same hash bucket have similar TF values. At this time, data of certain specific keys in the load is read evenly, which is a read-uniform load.

[0103] (3) Read operations account for a small proportion in the RAF cache. For example, the number of read operations in the RAF cache is less than 10% of all operations, which indicates that the load at this time is write-intensive. In order to ensure the persistent storage of data, most data needs to be directly persisted to PM. The RAF cache has no significant effect on improving the performance of key-value storage systems.

[0104] In this embodiment, the volatile index layer is located in DRAM. In practical applications, concurrent range indexes such as tree-based or learning model-based indexes can be used. Optionally, in this embodiment, considering that learning model-based indexes usually have good performance and small space usage, the learning model-based index is selected as the default index, which usually has good performance and small space usage.

[0105] Considering that remote DRAM access is faster than local PM access, the volatile index spans multiple NUMA nodes.

[0106] In this embodiment, the underlying log structure resides in the PM (Process Management Unit). Specifically, each NUMA node creates a local log file (denoted as the Nlog file), and strictly requires that all write operations can only be appended to the latest local Nlog file. To prevent a single operation log from becoming a bottleneck in a multi-threaded concurrent environment, the Nlog log structure uses a hash function to calculate the hash value of the key and groups the data according to the hash value of the key. All log entries with the same hash value are aggregated into a hash group, and each hash group of each NUMA node has an independent active file. Once the active file is full, it is archived and cannot be written to or modified, and a new active file is created. In this embodiment, the default size of the Nlog file is 8MB. Since there are no duplicate keys between hash groups, write contention only occurs within the hash group, thereby improving the concurrency of write operations.

[0107] In addition, key-value pairs within each hash group share a sequentially incrementing version number variable. Each time a log entry is appended to a hash group, the version number variable within that hash group is incremented.

[0108] Specifically, the address information of the log entries is as follows: Figure 3 As shown, the address information of a log entry includes: the length of the log entry, denoted as l_len; the ID of the NUMA node of the log entry, denoted as numa_id; the ID of the log file containing the log entry within the hash group, denoted as log_file_id; and the offset of the log entry within the log file, denoted as offset.

[0109] In this implementation, the address information of the log entry with the latest value stored in the volatile index layer is denoted as v_addr; optionally, in v_addr, l_len, numa_id, log_file_id, and offset occupy 2 bytes, 3 bytes, 4 bytes, and 8 bytes respectively, from high to low. Figure 4As shown, by using the hash group number calculated from the hash value of the key (numa_id, log_file_id, and key), the indexing system can accurately locate a specific log file, and then accurately locate the log entry based on the offset and l_len.

[0110] To accommodate different types of workloads, this embodiment designs three log item structures, such as... Figure 3 As shown, the log item structures corresponding to various key-value pair data are as follows:

[0111] For key-value pairs where both the key and value lengths remain unchanged, the log entry contains: the operation type, denoted as Op; the key and value of the key-value pair, denoted as Key and Value respectively; the version number of the value, denoted as Version; and the address information of the log entry containing the previous version of the value corresponding to the key, denoted as h_addr.

[0112] For key-value pairs where the key length is constant but the value length is variable, the log entry specifically includes: the operation type, denoted as Op; the key and value of the key-value pair, denoted as Key and Value respectively; the version number of the value, denoted as Version; the length of the value, denoted as V_len; and the address information of the log entry containing the previous version of the value corresponding to the key, denoted as h_addr.

[0113] For key-value pairs where both the key and value lengths are variable, the log entry contains: the operation type, denoted as Op; the key and value of the key-value pair, denoted as Key and Value respectively; the version number of the value, denoted as Version; the length of the key, denoted as K_len; the length of the value, denoted as V_len; and the address information of the log entry containing the previous version corresponding to the key, denoted as h_addr.

[0114] Optionally, in this embodiment, the Op values ​​corresponding to the insert (ins), update (upd), and delete (del) operations are represented by 11, 10, and 01, respectively. Simultaneously, the corresponding log entries are marked as invalid log entries by setting Op to 00. In the log entry structure, h_addr records the address of the log entry corresponding to the previous operation on the same key. Therefore, historical operations on the same key can be searched based on h_addr. It is easy to understand that for key-value pairs inserted via insert operations, there are no historical operations. Optionally, in this embodiment, h_addr in the log entry corresponding to the insert operation is set to invalid to mark the end point of the historical operation search.

[0115] Overall, this embodiment establishes a cross-NUMA node RAF cache in DRAM above the volatile index layer, and proposes a caching mechanism and cache replacement strategy suitable for NUMA architecture, realizing NUMA node-aware data caching and effectively improving the read performance of key-value pairs; and through improvements to the underlying log storage mechanism, it realizes a NUMA node-aware log storage structure, which can effectively improve the concurrency of key-value pairs in PM and improve write performance.

[0116] Example 2:

[0117] A key-value manipulation method. The key-value manipulation method provided in this embodiment is executed on the NUMA-aware key-value storage system based on hybrid memory provided in Embodiment 1 above.

[0118] The key-value operation method provided in this embodiment specifically includes: receiving an operation request from the current load, parsing the operation type, and then performing the corresponding operation;

[0119] Operation types include: single-point read operation, range read operation, insert operation, update operation, and delete operation;

[0120] like Figure 5 As shown, the execution of a single-point read operation includes:

[0121] Step R1: If RAF caching is enabled, proceed to step R2; otherwise, proceed to step R4.

[0122] Step R2 is based on the key in the operation request. r Query the RAF cache; if a match is found, return the corresponding value directly. r The corresponding cache reward is updated, and the single-point read operation ends; if a cache miss occurs, proceed to step R3.

[0123] Step R3: Locate the volatile index layer to obtain the corresponding value. r The address information of the log entry is retrieved, and the log entry is read to obtain the value. r And return, and set the key. r Sum of values r The key-value pairs are cached in the RAF cache, and the single-point read operation ends.

[0124] After obtaining the address information of the log entry from the volatile index layer, the length of the log entry, the ID of the NUMA node of the log entry, the ID of the log file containing the log entry within the hash group, and the offset of the log entry within the log file can be parsed. Figure 4 The illustrated locating process enables the location of log entries;

[0125] It is easy to understand that when the RAF cache misses, if there are no empty slots in the corresponding hash bucket when caching key-value data, the cache entry with the minimum cache benefit will be selected from the hash bucket for replacement.

[0126] Step R4: Locate the volatile index layer to obtain the corresponding value. r The address information of the log entry is retrieved, and the log entry is read to obtain the value. r And return, the single-point read operation ends;

[0127] Because log files are written strictly in an append-only manner, all valid log entries will not be modified. Therefore, Nlog supports lock-free reads, while the volatile index layer uses node locks to control the concurrency of read and write operations.

[0128] like Figure 5 As shown, the execution of a range read operation includes:

[0129] Step S1: Query all keys within the target key range in the volatile index layer and obtain the corresponding address information;

[0130] Step S2: Parse the NUMA node where each key-value pair data is located from the address information read in step S1, and assign the key-value pair data reading task to the corresponding NUMA node;

[0131] Step S3: Create multiple local background read threads on each NUMA node. The local background read threads execute the key-value pair data reading tasks in the corresponding NUMA node in parallel and return the read values.

[0132] like Figure 6 As shown, the insertion operation includes:

[0133] Step I1: Calculate the key-value pair data kv to be inserted. i middle key i The second hash value is used to determine the key-value pair data kv. i The target hash group to be inserted;

[0134] Step I2: Create a key-value pair data kv file in the active file of the target hash group of the target NUMA node for the insertion operation. i Allocate space for log entries;

[0135] Step I3: Construct key-value pair data (kv) i The corresponding log entry; in the constructed log entry, h_addr is set to invalid, and Op is the opcode corresponding to the insertion operation;

[0136] Step I4: Append the constructed log entries to the activity file.

[0137] like Figure 6 As shown, the execution of the update operation includes:

[0138] Step U1: For the key-value pair data (kv) to be updated u Query key-value pair data (kv) in the volatile index layer. u key u The corresponding address information v_addr u ;

[0139] Step U2: Calculate the key u The second hash value is used to determine the key-value pair data kv. u The corresponding hash group, and the key-value pair data kv in the active file of that hash group. u Allocate space for log entries;

[0140] Step U3: Construct key-value pair data (kv) u The corresponding log entry; in the constructed log entry, h_addr is set to v_addr. u Op is the opcode corresponding to the update operation;

[0141] Step U4: Append the constructed log entry to the activity file and update the address information of the log entry to the volatile index layer.

[0142] The execution of a delete operation is similar to that of an update operation, including:

[0143] Step D1: Query the target key in the volatile index layer. d The corresponding address information v_addr d ;

[0144] Step D2: Calculate the target key d The corresponding second hash value is used to determine the hash group where the key-value pair data to be deleted is located, and log entry space is allocated for the key-value pair data to be deleted in the active file of that hash group;

[0145] Step D3: Construct the log entry corresponding to the key-value pair data to be deleted; in the constructed log entry, h_addr is set to v_addr. d Op is the corresponding opcode for deletion;

[0146] It is easy to understand that when allocating log entries, all content is initialized to 0. For deleted key-value pairs, their values ​​are no longer used. Therefore, when constructing the log entry corresponding to the key-value pairs to be deleted, there is no need to populate the value field.

[0147] Step D4: Append the constructed log entries to the activity file and add the target key from the volatile index layer. d The corresponding index data.

[0148] To avoid the advantages of RAF caching not outweighing the overhead of maintaining and computing additional information, this embodiment also includes: determining the type of the current load while parsing the operation type;

[0149] If the current load type is read-uniform load or write-intensive load, and RAF caching is enabled, then disable RAF caching;

[0150] If the current load type is neither a read-uniform load nor a write-intensive load, and RAF caching is disabled, then enable RAF caching.

[0151] In this embodiment, the method for determining the uniform read load and the intensive write load can be referred to the description in the above embodiment 1. When condition (1) or (2) is met, the load type is determined to be a uniform read load. When condition (3) is met, the load type is determined to be a intensive write load.

[0152] In this embodiment, a write lock located in DRAM is used for each hash group to support concurrent writes to the active file. When an active file is appended to, the lock must be acquired first, the append operation performed, and then the lock released. Furthermore, once a write thread acquires the allocated log entry space, the lock is released without waiting for the data in the log entry to be fully written. In the volatile index layer, each node uses a node lock to control concurrent writes. When a node undergoes operations such as merging, splitting, or updating, the lock must be acquired first, the corresponding operation performed, and then the lock released.

[0153] As the key-value store system is used continuously, invalid log entries will gradually accumulate in the log file. To avoid invalid log entries occupying too much PM storage space, it is necessary to reclaim the storage space occupied by invalid log entries in the log file, i.e., perform garbage collection. In this embodiment, due to the existence of h_addr in the log entries, garbage collection can be performed without a query process, minimizing interference with foreground operations; at the same time, since the entries marked as invalid are all outdated, modifying them will not affect the foreground thread.

[0154] To effectively achieve garbage collection, this embodiment also includes: periodically performing log item marking operations within each NUMA node; such as... Figure 7 As shown, the execution of the log item marking operation includes:

[0155] Step L1: Within the current NUMA node, randomly select a hash group and traverse the log entries within it. For the currently traversed log entry E, if it is a valid log entry, proceed to step L2; otherwise, mark the current log entry as finished.

[0156] Step L2: If the operation type of the current log item E is an update operation or a delete operation, proceed to step L3; if the operation type of the current log item E is an insert operation, proceed to step L4.

[0157] Step L3: Search the historical operations of log entry E based on h_addr until an invalid log entry with h_addr is encountered, or an invalid log entry, or a log entry pointed to by h_addr has been reclaimed. Mark all the searched log entries as invalid.

[0158] Step L4: If the current log entry E is not referenced by the volatile index layer, then mark the current log entry E as invalid;

[0159] The key-value operation method also includes: when the number of invalid log entries in a log file exceeds a preset threshold, the valid log entries in the log file are appended to the active file in the same hash group, and the log file is deleted.

[0160] In this embodiment, each NUMA node has a garbage collection thread for the local log item traversal process of the NUMA node, which can reduce cross-node PM access and different garbage collection threads do not need to communicate with each other.

[0161] In this embodiment, the write thread always writes the `op` field last when appending log entries. Each active file is initialized to 0, so new log entries are invalid until `op` is updated to 01 (delete operation), 10 (update operation), or 11 (insert operation). After a system crash, the recovery thread ignores invalid log entries. Therefore, crash consistency of write operations is guaranteed.

[0162] To ensure data consistency in memory, such as Figure 8 As shown, the key-value operation method provided in this embodiment further includes: before shutting down the key-value storage system, storing the index data in the volatile index layer in a preset location in the PM, and persisting a flag bit in the PM to indicate whether the normal key-value storage system has been shut down normally;

[0163] When the key-value store system starts up, it reads the index data and the flag from the PM;

[0164] If the flag indicates that the key-value storage system has been shut down normally, the read index data will be loaded into the volatile index layer.

[0165] If the flag indicates that the key-value pair storage system has not been shut down properly, then within each NUMA node, a scan thread is allocated for each hash group to scan the log files within the corresponding hash group in order to rebuild the volatile index layer shown.

[0166] When the scanning thread scans the log file, for log entries with the same key, it only retains the log entry with the latest version number. After the scan is completed, it builds corresponding index data based on the retained log entries.

[0167] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A hybrid memory based NUMA aware key-value store system, characterized in that, The application relates to a key-value storage system comprising: a plurality of NUMA nodes; each NUMA node comprises computing resources and memory resources, the memory resources comprising DRAM and PM; the PM of each NUMA node stores a log file for recording log items in an appended form; each log item comprises key-value pair data and a corresponding write operation; the write operation comprises an insertion operation, an update operation and a deletion operation; a volatile index layer and a RAF cache shared by all NUMA nodes are maintained in the key-value storage system; the volatile index layer is composed of a first memory region in the DRAM of each NUMA node and is used for storing index data of each key-value pair; the index data comprises a key and address information of a log item in which the latest value is located; the RAF cache is composed of a second memory region in the DRAM of each NUMA node, each cache entry in the RAF cache is used for caching a key-value pair data; the higher the access frequency of the cached key-value pair data and the higher the delay of the NUMA node accessing the PM of the NUMA node in which the key-value pair data is located, the higher the cache benefit of the corresponding cache entry; and when cache replacement occurs in the RAF cache, the cache entry with the lowest cache benefit is selected for replacement; The RAFCache also maintains a set of latencies , n denotes the total number of MUMA nodes, denotes the latency to access the PM in the i th NUMA node from the x th NUMA node. And, for a cache entry C in the RAF cache, when a key-value pair data kv is written to the cache entry C, its cache benefit will be initialized as , j and k respectively represent the NUMA node number of the request to access kv and the NUMA node number where kv is stored; Access requests to key-value pair data kv The cache benefit of cache entry C will be updated as , and represent the cache benefits before and after the update, respectively.

2. The hybrid memory based NUMA aware key-value store system as recited in claim 1, wherein, the RAF cache is a hash table, each table entry is a hash bucket composed of a fixed number of slots, and each slot is used for storing a cache entry; the information recorded by the cache entry comprises a key key of a key-value pair data, a value value, a cache benefit TF, an id of a NUMA node in which the key-value pair data is stored, and a lock lock; the hash bucket in which the cached key-value pair data is located is addressed by a first hash value of the key.

3. The hybrid memory based NUMA aware key-value store system of claim 1 or 2, wherein, The log file stored in the PM of each NUMA node is divided into a plurality of hash groups corresponding to a second hash value of the key, and only one log file exists in each hash group as an active file; when a log item is written into the PM, the active file in the corresponding hash group is appended according to the second hash value of the key; after the active file is full, the active file cannot be written or modified, and meanwhile, the hash group is allocated a new active file; a sequentially increasing version number variable is shared by the key-value pair data in each hash group, and the version number variable in the hash group is increased each time a log item is appended and written into the hash group.

4. The hybrid memory based NUMA aware key-value store system as recited in claim 3, wherein, The address information of the log item comprises: the length of the log item, denoted as l_len; the id of the NUMA node in which the log item is located, denoted as numa_id; the id of the log file in the hash group, denoted as log_file_id; and the offset of the log item in the log file, denoted as offset; and for the key-value pair data with unchanged lengths of the key and the value, the log item in which the key-value pair data is located comprises: an operation type, denoted as Op; the key and the value of the key-value pair data, denoted as Key and Value respectively; the version number of the value, denoted as Version; and the address information of the log item in which the previous version of the value corresponding to the key is located, denoted as h_addr. For the key-value pair data with the fixed length of key and the variable length of value, the log item where the key-value pair data is located specifically includes: an operation type, denoted as Op; the key and the value of the key-value pair data, denoted as Key and Value respectively; the version number of the value, denoted as Version; the length of the value, denoted as V_len; and the address information of the log item where the value of the previous version corresponding to the key is located, denoted as h_addr; For the key-value pair data with the variable length of key and value, the log item where the key-value pair data is located specifically includes: an operation type, denoted as Op; the key and the value of the key-value pair data, denoted as Key and Value respectively; the version number of the value, denoted as Version; the length of the key, denoted as K_len; and the length of the value, denoted as V_len; and the address information of the log item where the previous version corresponding to the key is located, denoted as h_addr.

5. A key-value operation method, comprising: The hybrid memory based NUMA-aware key-value storage system of claim 4 is executed; The key-value operation method comprises: receiving an operation request from a current load, parsing an operation type, and then performing a corresponding operation. The operation type comprises: a single-point read operation, a range read operation, an insertion operation, an update operation, and a deletion operation. The execution of the single-point read operation comprises: Step R1: if the RAF cache is enabled, go to step R2; otherwise, go to step R4. Step R2: based on the key key in the operation request r Query the RAF cache, if hit, directly return the corresponding value value r , and update the corresponding cache benefit, single-point read operation ends; if not hit, go to step R3; Step R3: look up the volatile index layer to obtain the corresponding value value r the address information of the log item where the key key is located, and read the log item to obtain the value value r and return, look up the volatile index layer to obtain the corresponding value value r the address information of the log item where the key key is located, and read the log item to obtain the value value r and return, and cache the key-value pair data composed of the key key r and the value value r to the RAF cache, and the single-point read operation ends; Step R4: look up the volatile index layer to get the corresponding value value r the address information of the log item, and read the log item to get the value value r and return, the single-point read operation ends; The execution of the range read operation comprises: Step S1: querying all keys located in a target key range in the volatile index layer and obtaining corresponding address information; Step S2: parsing the NUMA node where each key-value pair data is located from the address information read in step S1, and distributing the key-value pair data reading task to the corresponding NUMA node; Step S3: creating a plurality of local background reading threads in each NUMA node, respectively, and performing the key-value pair data reading task in the corresponding NUMA node by the local background reading thread in parallel, and returning the read value. The execution of the insertion operation comprises: Step I1: calculating the key value pair data kv i the second hash value of the middle key key i to determine the target hash group to which the key value pair data kv i is to be inserted; Step I2: Inserting the key-value pair data kv into the active file of the target hash group of the target NUMA node of the insertion operation i Allocating log item space; Step I3: constructing key-value pair data kv i The corresponding log item; in the constructed log item, h_addr is set to invalid, and Op is the operation code corresponding to the insert operation; Step I4: appending the constructed log item to the active file; The execution of the update operation comprises: Step U1: for the key-value pair data kv to be updated u , querying the key-value pair data kv in the volatile index layer u with the key key u corresponding to the address information v_addr u ; Step U2: Calculate the second hash value of the key key u to determine the corresponding hash group of the key-value pair data kv u , and allocate log item space for the key-value pair data kv u in the active file of the hash group. Step U3: Constructing the key-value pair data kv u The corresponding log item; in the constructed log item, h_addr is set as v_addr u Op is the operation code corresponding to the update operation; Step U4: appending the constructed log item to the active file, and updating the address information of the log item to the volatile index layer; The execution of the deletion operation comprises: Step D1: query the target key key in the volatile index layer d corresponding address information v_addr d ; Step D2: Calculate the target key key d corresponding second hash value to determine the hash group where the key-value pair data to be deleted is located, and allocate log item space for the key-value pair data to be deleted in the active file of the hash group; Step D3: constructing a log item corresponding to the data pair of the key value pair to be deleted; in the constructed log item, h_addr is set as v_addr, and Op is a corresponding operation code of more deletion d . Step D4: Append write the constructed log entry to the active file and update the target key key d corresponding index data.

6. The key-value operation method of claim 5, wherein, Further comprising: While parsing the operation type, judging the type of the current load; If the type of the current load is a read uniform load or a write intensive load, and the RAF cache is enabled, then the RAF cache is disabled; If the type of the current load is neither a read uniform load nor a write intensive load, and the RAF cache is disabled, then the RAF cache is enabled. 7.The key-value operation method of claim 5 or 6, wherein, Further comprising: Periodically performing a log item marking operation in each NUMA node; The execution of the log item marking operation comprises: Step L1: in the current NUMA node, randomly selecting a hash group, traversing the log items in the hash group, and for the current traversed log item E, if it is a valid log item, go to step L2; otherwise, the current log item marking ends. Step L2: If the operation type of the current log item E is an update operation or a delete operation, go to step L3; if the operation type of the current log item E is an insert operation, go to step L4; Step L3: Search the history operation according to h_addr in the log item E until an invalid log item is encountered, or an invalid log item, or a log item pointed by h_addr has been recycled, and mark all the searched log items as invalid; Step L4: If the current log item E is not referenced by the volatile index layer, mark the current log item E as invalid; The key-value operation method further comprises: when the number of invalid log items in a log file exceeds a preset threshold, appending the valid log items in the log file to the active file in the same hash group, and deleting the log file. 8.The key-value operation method of claim 5 or 6, wherein, Further comprising: before the key-value storage system is closed, storing the index data in the volatile index layer to a preset location in the PM, and persisting a flag bit flag in the PM, which is used to indicate whether the key-value storage system is normally closed.

9. The key-value operation method of claim 8, wherein, Further comprising: When the key-value storage system is started, reading the index data and the flag bit flag from the PM; If the flag bit flag indicates that the key-value storage system is normally shut down, loading the read index data to the volatile index layer; If the flag bit flag indicates that the key-value storage system is not normally shut down, allocating a scan thread to each hash group in each NUMA node, and scanning the log files in the corresponding hash group to rebuild the volatile index layer; When the scan thread scans the log files, for log items with the same key, only the log item with the latest version number is retained, and after the scanning is completed, the corresponding index data is established according to the retained log items.

Images