A key-value storage system and method based on a cache gradient cold and hot data layering mechanism

Abstract

The application discloses a key value storage system and method based on a cache gradient cold and hot data layering mechanism, the system comprising a memory component, a device component and a disk component, the method being: a filter screens Key to shunt cold and hot data; hot data enters a hot cache module, and cold data enters a cold cache module; the hot cache module eliminates obsolete data to enter the cold cache module, and cold data reaches a quantity to enter a GPU sort component to be sorted while being backed up to a GPU cache area component; cold data is stored to the disk component after being sorted by the GPU sort component, and meanwhile, files with a disk file heat value sum less than a predetermined value are migrated to a lower layer of the system, and files with a disk file heat value sum greater than the predetermined value are migrated to an upper layer of the system. The application optimizes the memory component, increases a gradient grading cold and hot data mechanism, replaces a memory data structure, optimizes a read-write mechanism of memory cache data, sets a "hot data upsurge" mechanism, and can optimize reading and writing with a better and more accurate cache strategy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data storage, and in particular to a key-value storage system and method based on a cache gradient cold and hot data layering mechanism. BACKGROUND

[0002] NoSQL is a non-relational database born with the demand for super large-scale concurrency, and the key-value pair (Key-Value) storage database is an efficient management scheme that can cope with distributed large-scale data. Since the disk I / O overhead is one of the efficiency bottlenecks of the database, many schemes have been proposed to reduce disk I / O. LSM tree is one of the schemes to solve the problem of frequent disk read and write. The Log-Structured Merge Tree (LSM-tree) is a key-value storage system and method based on a cache gradient cold and hot data layering mechanism. The distributed key-value pair storage database based on the LSM-tree structure has become the mainstream database scheme in the industry, such as the classic KV database LevelDB, the multi-threaded database RocksDB with multiple optimizations, the WiscKey based on key-value separation, and the PebblesDB based on the partitioning idea. The essence of LSM-Tree is based on the out-place update strategy. The data update operation directly appends the new value to the end of the file, converting random write to sequential write, which improves the write performance while causing redundant data and write amplification. At the same time, the design principle of LSM-tree requires memory first and then disk. When data is written, LSM does not directly write to the disk, but keeps it in memory for a period of time and uses the Write-Ahead Logging (WAL) to ensure data is not lost.

[0003] The key-value separation strategy well controls the write amplification of LSM-tree, but at the same time, it increases the requirement for reading and does not optimize the cache structure of the memory, so that in the typical WPI workload, point lookup read accounts for 75% and write accounts for 25% (75% point lookups, 25% writes). The cache data is not utilized well to speed up reading. The existing memory component structure mostly uses B / B+ tree structure to replace the memory skip list. Since B / B+ tree is a balanced structure for reading and writing, the introduction of B / B+ tree will bring the overhead of memory space. For some hot data, B / B+ tree will also cause skewing and other problems.

[0004] The key-value storage database is the preferred solution for many distributed systems to manage large-scale data. There are many improvements and optimization schemes for LSM-Tree structure, but there are limited improvements for the cache. Moreover, the existing memory database and LSM-tree database are not well combined, so it is necessary to optimize the existing key-value storage database memory component.

[0005] There is a need for a better and more accurate cache hot data strategy to improve read performance, and to build a system that can optimize read and write at the same time. SUMMARY

[0006] The purpose of the present application is to provide a key-value storage system and method based on cache gradient cold and hot data layering mechanism, which can optimize cache components, increase gradient classification of cold and hot data mechanism, replace memory data structure, optimize read and write mechanism of memory cache data, and can optimize read and write with a better and more accurate cache strategy.

[0007] The purpose of the present application can be achieved by the following technical solutions: a key-value storage system based on cache gradient cold and hot data layering mechanism, comprising an ordered layered Key-Value data memory component, a device component and a disk component, the memory component comprising a cascaded BloomFilter module, a cold HashTable cache module and a hot HashTable cache module, the device component comprising a GPU sort component, a GPU cache area component and a driving module;

[0008] The cascaded BloomFilter module is used for accurate screening of Key in Key-Value, and shunting hot data and cold data;

[0009] The cold HashTable cache module and the hot HashTable cache module are respectively used for caching hot data and cold data;

[0010] The GPU sort component is used for sorting cold data entering the GPU sort component;

[0011] The GPU cache area component is used for backing up cold data entering the GPU sort component for sorting;

[0012] The driving module is used for coordinating the memory component to call the GPU sort component.

[0013] A key-value storage method based on cache gradient cold and hot data layering mechanism, the method is applied to a key-value storage system based on cache gradient cold and hot data layering mechanism, and the method comprises the following working steps:

[0014] S1, cascaded circulating BloomFilter accurately screens Key in Key-Value, and shunts hot data and cold data;

[0015] S2, the hot data enters the hot HashTable cache module, and the cold data enters the cold HashTable cache module;

[0016] S3, the cold data in the cold HashTable cache module reaches a set number and enters a GPU sort component to sort, and the cold data reaches the set number and is simultaneously backed up to a GPU cache area component;

[0017] S4, the cold data is stored to a disk component after being sorted by the GPU sort component.

[0018] As a further scheme of the application: the cascade circulation BloomFilter is a three-layer cascade circulation.

[0019] As a further scheme of the application: the three-layer cascade circulation BloomFilter accurate Key screening step includes:

[0020] S11, the first layer BloomFilter screens the Key, if the current Key does not exist, then the Key is added, and the Key hotness value is set to 0, if the Key exists;

[0021] S12, the second layer BloomFilter is searched, if the second layer BloomFilter does not exist the Key, then the Key is directly inserted, and the Key hotness value is set to 1, if the Key exists;

[0022] S13, the third layer BloomFilter is searched, if the third layer BloomFilter does not exist the Key, then the Key is inserted, and the Key hotness value is set to 3, if the Key still exists;

[0023] S14, the Key hotness value is directly set to the maximum value 128.

[0024] As a further scheme of the application: the hot data Key hotness value is 1, 3 and 128; the cold data Key hotness value is 0.

[0025] As a further scheme of the application: the GPU cache area component is used for hot data read optimization operation, and the steps of the GPU cache area component used for read optimization operation include:

[0026] S31, it is judged whether the searched data is hot data, if yes;

[0027] S32, according to the query, the memory component is searched, if searched, then the result is directly returned;

[0028] S33, if not searched, the GPU cache area component is searched in parallel, if searched, then the result is returned;

[0029] S34, if not searched, the disk component is searched.

[0030] As a further scheme of the present application: the hot HashTable cache module and the cold HashTable cache module record Key offset, Value offset and the hotness value of Key in HashTable.

[0031] As a further scheme of the present application: the disk component adopts the structure of hot layer and cold layer to manage the metadata disk SSTable file of Key, the hot layer adopts hash function management, and the cold layer adopts ordered management.

[0032] As a further scheme of the present application: the hash function management of the hot layer comprises the following steps:

[0033] S41, record the hotness value sum in the SSTable file written in the disk, so as to distinguish the cold file and the hot file;

[0034] S42, the hot layer of the disk SSTable file adopts Hash function partition, and the Key data with similar prefixes are divided into one partition;

[0035] S43, the data with low hotness value sum is stored in the slow storage system logical lower layer, and the data with high hotness value sum is stored in the fast storage system logical upper layer.

[0036] As a further scheme of the present application: the Value file in the Key-Value adopts the strategy of Value file in-place update, and the strategy of Value file in-place update comprises the following steps:

[0037] S21, invalid space Key tracking is performed on the stored Value file after key value separation;

[0038] S22, if the Key is directly inserted without update, the append method is adopted to insert the Value file;

[0039] S23, if the Key is updated, the Value offset is found, the position of the invalid Value file is located, and the new Value file is inserted;

[0040] S24, if the original address Value file space is not enough to insert the current Value file, the append method is continued to insert.

[0041] The present application has the following beneficial effects:

[0042] 1、The application discloses a key value storage system and method based on a cache gradient cold and hot data layering mechanism, which adopts a cold and hot data discrimination mechanism with accurate cold and hot gradient layering, and processes different cold and hot data in different ways, so that data reading and writing are more efficient, GPUsort and HashTable are used to replace SkipList in the original memory component, HashTable is used to improve the reading and writing performance, unordered data can be sorted on the GPU by using the super high computing power of the GPU, the cache component is optimized, the gradient grading cold and hot data mechanism is increased, the memory data structure is replaced, the reading and writing mechanism of the memory cache data is optimized, the reading and writing is optimized by using a better and more accurate cache strategy, hot data is stored in the memory as much as possible for users to use, and cold data is sunk to the disk, so that the reading is accelerated, the disk I / O is reduced, the reading performance is improved by optimizing the reading and writing and better and more accurate caching of hot data.

[0043] 2、Increase the disk file cold and hot data layering active sensing technology, perform cold and hot file layering active sensing on the disk, store hot layer files in the logical upper layer of the storage system as much as possible, manage SSTable files in the disk component through the cold and hot layering structure, record the total heat value, distinguish cold and hot files, divide similar Key prefixes into a partition, realize disorder in the partition and order between partitions, support cold data sinking and hot data rising by using the heat value, perform hierarchical storage of data by using the different speeds of storage media, optimize the reading and writing by performing ordered layering of data, and improve the data reading performance.

[0044] 3、For the Value file, the in-place updating strategy is adopted, invalid space tracking is performed on the storage Value file after key value separation, the in-place updating strategy can avoid user operation blocking caused by one-time large GC, reduce space amplification, and avoid instantaneous computing power competition and high space amplification caused by one-time GC. DETAILED DESCRIPTION

[0045] Figure 1 A work flow chart of the key value storage system and method based on the cache gradient cold and hot data layering mechanism;

[0046] Figure 2 A work flow chart of the cascaded circulation BloomFilter;

[0047] Figure 3 A work flow chart of the disk component adopting the hot layer and cold layer structure to manage the metadata disk SSTable file of Key;

[0048] Figure 4 A work flow chart of the in-place updating strategy of the Value file. Detailed Implementation

[0049] Embodiments of the present invention are described in detail below. Examples of these embodiments are illustrated in the accompanying drawings, wherein the same or similar symbols denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0050] To facilitate a precise understanding of this invention, the terms and their English abbreviations involved in this invention are explained first: BloomFilter is a Bloom filter, generally used to check whether an element is in a set; Key-Value is a key-value pair, including a key and a value, which is a data structure of a database storage system; HashTable is a hash table, a data structure that can be accessed directly based on key-value pairs; Hash is a hash function, which is a function in HashTable; SkipList is a skip list; GPUsort is GPU sorting; O(1) is the time complexity of the program; SSTable file is a disk-sorted string table, and the key-value pairs in the SSTable file are stored in order; append method is the traditional append writing method; GC is garbage collection.

[0051] like Figures 1-4 As shown, a key-value storage system based on a cache gradient cold and hot data tiering mechanism is disclosed. The system includes a memory component, a device component, and a disk component for ordered tiered key-value data. The memory component includes a cascaded Bloom Filter module, a cold HashTable cache module, and a hot HashTable cache module. The device component includes a GPUsort component, a GPU cache component, and a driver module.

[0052] The cascaded BloomFilter module is used to accurately filter the Key in the Key-Value pair, separating hot data from cold data;

[0053] The cold HashTable cache module and the hot HashTable cache module are used to cache hot data and cold data, respectively.

[0054] The GPUsort component is used to sort cold data entering the GPUsort component;

[0055] The GPU cache component is used to back up cold data that enters the GPUsort component for sorting;

[0056] The driver module is used to coordinate the memory component's call to the GPUsort component.

[0057] A key-value storage method based on a cache gradient hot and cold data hierarchical mechanism, the method is applied to a key-value storage system based on a cache gradient hot and cold data hierarchical mechanism, the method comprises the following working steps:

[0058] S1, cascade circulating BloomFilter accurately screens Key in Key-Value, and shunts hot data and cold data;

[0059] S2, the hot data enters the hot HashTable cache module, and the cold data enters the cold HashTable cache module;

[0060] S3, the cold data in the cold HashTable cache module reaches a set amount and enters the GPU sort component for sorting, and the cold data reaches a set amount and is simultaneously backed up to the GPU cache area component;

[0061] S4, the cold data is stored to the disk component after being sorted by the GPU sort component.

[0062] Cascade circulating BloomFilter undertakes the most important hot and cold data identification of the whole system, and other components rely on the Key hot and cold value to determine what operation to perform. After cascade circulating BloomFilter marks the hot and cold value for a Key, the hot and cold value is used to judge whether the Key data is hot data or cold data, and whether it should fall into the hot HashTable cache module or the cold HashTable cache module. The hot HashTable cache module will eliminate obsolete data into the cold HashTable cache module. When the cold HashTable cache module reaches a set value, the system will call GPU sort to perform data sorting and caching functions. GPU sort uses super large computing power and super large video memory to sort and cache data. When sorted, the CPU assembles data and writes it to the disk component.

[0063] The flow chart of the system work is as shown in Figure 1 The memory component is divided into cascade BloomFilter module, cold HashTable cache module and hot HashTable cache module. A user transmits a request through a put interface. When a Key-Value key value pair reaches the memory component, the Key first passes through the cascade circulating BloomFilter module. Cascade circulating BloomFilter is divided into three layers. Three cascade circulating BloomFilters form a circulating filter queue. The head of the queue is marked by a current pointer. The weight from the head to the tail of the queue is set by the system and is not equal. Therefore, the frequency of different Keys can be filtered out.

[0064] The precise screening Key step of the three-level cascading cyclic BloomFilter includes: setting a certain size of Key quantity limit for each cascading cyclic BloomFilter, first verifying whether the quantity of the first layer BloomFilter pointed to by the current pointer has exceeded the limit, if the limit is exceeded, clearing the content of the current BloomFilter, and pointing the current pointer to the second layer BloomFilter. Figure 2 As shown in FIG. 8, when a Key arrives, it needs to be given an accurate hot value, we will first check whether the Key exists in the first layer BloomFilter, if not, directly assign the hot value to 0, if exists, then find the Key in the second layer BloomFilter, if not exists in the second layer BloomFilter, then assign the hot value to 1, wherein the weight of the second layer BloomFilter is 1, and the weight of the first layer BloomFilter is 0, if not exists in the second layer BloomFilter, then the weight is 0+1=1, so the hot value of the Key is assigned to 1, if exists, then continue to find in the third layer BloomFilter, if not exists, then assign the hot value to 3, wherein the weight of the third layer BloomFilter is 2, so the weight after finding in the current layer is the sum of the weights of the three layer BloomFilters, so the weight is 0+1+2=3, so the hot value of the Key is assigned to 3, if the Key still exists in the third layer BloomFilter, it means that the hotness of the Key is very hot, so that the three layer BloomFilter still cannot give its hotness layer, then we can directly assign its hot value to the maximum hot value Max, that is, 128.

[0065] The cascading cyclic BloomFilter undertakes the role of precise shunting of cold and hot data, and provides guidance for the cold HashTable cache module and the hot HashTable cache module, the BloomFilter filters out different hotness Keys layer by layer, and caches the hotter Key data in the hash table, the hot HashTable cache module will eliminate obsolete data into the cold HashTable cache module, and the cold data will be accumulated to a certain quantity and sent to the device end GPU for sorting, and the hot data will be left in the memory component to continue to provide read acceleration.

[0066] Further, the hot HashTable cache module and the cold HashTable cache module optimize reading and writing at the same time, and the reading and writing speed of the hot HashTable cache module and the cold HashTable cache module is approximately O(1). The hot HashTable cache module and the cold HashTable cache module optimize reading and writing at the same time, and the reading and writing speed is approximately O(1). The hot HashTable cache module and the cold HashTable cache module record the Key offset, the Value offset and the heat value of the Key in the HashTable, which facilitates subsequent searching for the Value according to the offset and provides a basis for subsequent adaptive processing of cold data and hot data.

[0067] The HashTable is divided into two areas, a hot data area and a cold data area. The hot data area stores the current high heat value Key, and the rest of the Key is stored in the cold data area. After distinguishing the cold data and the hot data, we store the Key with a heat value of 1, 3 and 128 into the hot HashTable, and store the Key with a heat value of 0 into the cold HashTable, and meanwhile, we eliminate the obsolete hot data into the cold data area. When the cold data area is full, the data is sent to the GPU sort for sorting, and the Value is also written into the Value file through the append write mode at the same time.

[0068] Further, the device component includes a cache area component recording the complete Key-Value key-value pair, and a driver module. When the cold data in the cold data area accumulates to a certain extent, the GPU and the CPU are two independent processors, which have their own independent computing resources and memory. Therefore, our data wants to help the GPU sort to sort, and needs to use the driver module to make a system call to complete operations such as GPU video memory space application, data bidirectional movement copy, synchronous and asynchronous mode control, error information processing, etc. The driver module can help us complete these operations. The driver module sends all the cold data in the cold HashTable to the GPU sort for sorting, and meanwhile, leaves a backup to the GPU cache area. The GPU can provide huge computing power to speed up the sorting. A new GPU sorting byte stream algorithm is used to realize the sorting. Compared with the CPU, the parallel computing resources can be used to achieve the purpose of sorting in a shorter time. Since the GPU has a huge cache space, we can use a part of the video memory to cache the Key-Value key-value pair to realize aggressive read optimization operation.

[0069] When searching, only hot data is searched in this way, because hot data is more likely to exist in the GPU buffer, and cold data does not have to be searched in this way. If it is a point query search, first search in the memory component, if searched, directly return, if not searched, search in the GPU cache, at this time the search is parallel search, the parallel capability of the GPU provides support for parallel search, after parallel search, if searched, return the result, if not, search in the disk component.

[0070] The memory structure using Hash is difficult for range query, but the range query can be converted into point query. After the GPU sorts, the sorted results are transmitted back to the CPU memory space by the driver module, and the CPU controls the data to be flushed into the disk component.

[0071] Further, the cold and hot layers are used to manage the metadata SSTable of the Key of the disk component. The hot layer uses hash management, and the cold layer uses ordered management. The ordered management of the cold layer is to store the SSTable in the storage area in a sequential and non-overlapping manner as much as possible. If the data overlaps, active data merging is needed to achieve the necessary condition of non-overlap. The SSTable file of the disk is partitioned by hash. Similar Key prefixes are divided into a partition to realize the order in the partition and the order between partitions.

[0072] At the same time, the hotness value sum of the record written into the SSTable of the disk is used to distinguish cold files and hot files, and the hotness value sum supports the operation of cold data sinking and hot data rising. Cold data sinking uses the different speeds of storage media to perform hierarchical storage. According to the hotness value sum of the record, this operation can be supported. The files on the disk will be operated according to the hotness value sum of the contained Key. The files with a hotness value sum less than a predetermined value will be migrated to the lower layer of the system, and the files with a hotness value sum greater than a predetermined value will be migrated to the upper layer of the system, to realize hierarchical storage of cold data and hot data. Hot data rising can put files with a high hotness value sum into a higher level, so that the search for Key can be faster and more efficient.

[0073] When flushing each SSTable, record the total hotness value in SSTable, which is saved in metadata. The total hotness value supports "hot data upwelling, cold data downwelling". If the total hotness value is less than a threshold, it is a cold SSTable, and if the total hotness value is greater than the threshold, it is a hot SSTable. For example, level 0 is a hot layer, and other levels are cold layers. The threshold is set to 50. If the total hotness value of a SSTable in level 0 is 10, the SSTable is defined as a cold SSTable. Since it is in the hot layer of level 0, it needs to be actively transferred to a lower level, which is "cold data downwelling". If a SSTable in a level greater than 0 has a total hotness value of 80, it is marked as a hot SSTable, and needs to be actively transferred to level 0, which is "hot data upwelling".

[0074] The flowchart is shown in Figure 3 The system first actively queries the metadata of the file in memory. According to the principle that "hot data should be placed in the hot layer and cold data should be placed in the cold layer", the total hotness value of the file is used to determine whether all metadata files match the above principle. If the hotness of the file does not match the coldness of the level, the hot file in the cold layer will be "upwelled" to the hot layer, and the cold file in the hot layer will be "downwelled" to the cold layer. After the operation is completed, this part of the process will be dormant for a period of time, and then actively enter this process again.

[0075] Further, for Value files, the in-place update strategy is adopted. For Value files stored after key-value separation, invalid space tracking is performed, that is, the Key is judged. If the Key is directly inserted, the traditional key-value separation append method is used for Value file insertion. If the Key is updated, the Value file offset is found to locate the position of the invalid Value file, and the new Value file is inserted. If the space of the original Value file is not enough to insert the current Value file, append insertion is continued. The above method uses a new strategy for Value file update, that is, the in-place update strategy. If a Key-Value is updated, the size of the Value is the same as the original Value. The address of the original Value is found, and the new value is overwritten. If it is not enough to write, then the file is appended to the end of the file.

[0076] The in-place update strategy flowchart is shown in Figure 4As shown: when the user request arrives, firstly, it is judged whether the current write operation can be updated in place, if yes, it can be inserted directly, if not, the request is written in the form of appending, then it is judged whether the invalid space needs to be actively recycled, if yes, the active GC is triggered, at this time, the GC amount is reduced, because the original invalid space is updated, if not, the process is ended.

[0077] This in-place update strategy can avoid the blocking of user-side operation caused by a large amount of GC at one time, and can reduce the space amplification. At the same time, this strategy can avoid the instantaneous computing power competition and high space amplification caused by one-time GC.

[0078] The application discriminates the user request through cold and hot data, accurately keeps the hot data in the memory and stores the cold data into the disk, through the gradient grading cold and hot data flag, the cold and hot data in the memory can be moved, the cold and hot files on the disk can be actively perceived and actively moved, so that faster reading and less I / O effect are achieved, in addition, the space amplification is reduced by the in-place GC strategy, the invalid space is better utilized, and the burden of active GC is reduced.

[0079] From the perspective of memory usage management, HashTable is used to replace the skip list in the memory, HashTable can solve the problem of data skew, HashTable is fast and the data is evenly distributed, in addition, HashTable has excellent performance in the case of massive data, through HashTable, a memory read-write component with O(1) read-write performance can be obtained. The existing cache component manages data, lacks accurate cold and hot mechanism, through the cascade circulation BloomFilter, different weights are given to hot data with different frequencies, fine management of hot value is realized, through fine cold and hot value marking of cold and hot data, the hot value in the cache can be cached as much as possible, the limited cache space is used efficiently, and the cold and hot value can be brought to the disk for cold and hot data medium storage; at the same time, GPUsort is used, which solves the problem of large computing power consumption of data sorting in HashTable before writing to the disk, because GPU contains a large amount of idle computing power, the sorting speed is much faster than CPU in the case of large data volume.

[0080] From the disk point of view, the cold and hot values can be brought into the file, and the file is dumped according to the cold and hot degree of the file, that is, the cold data is transferred to the lower layer of the slow storage system logic for storage, and the hot data is stored in the upper layer of the fast storage system logic for faster reading and writing. The key-value separation strategy can cause a large invalid space, and the application executes the in-place update strategy on the basis of the key-value separation, records the space of the updated invalid Key-Value, manages the invalid space for reuse, has a special Log recording the starting position and size of the invalid space, and finds the suitable space for in-place update insertion when the next Key comes, so as to achieve the purpose of reusing the invalid space.

[0081] The above merely describes the preferred embodiments of the present application, but the protection scope of the present application is not limited thereto, and any skilled person in the art can make equivalent replacements or changes within the technical range disclosed by the present application according to the technical solution and the inventive concept of the present application, which should be covered within the protection scope of the present application.

Claims

1. A key-value storage system based on a cache gradient cold and hot data tiering mechanism, characterized in that: The system includes a memory component, a device component, and a disk component for ordered hierarchical key-value data. The memory component includes a cascaded bloom filter module, a cold hashtable cache module, and a hot hashtable cache module. The device component includes a GPUsort component, a GPU cache component, and a driver module. The cascaded bloomfilter module is used to precisely filter the key in the key-value pair, and to separate hot data from cold data; Specifically, it includes: The cascaded loop BloomFilter is a three-level cascaded loop. The cold hashtable caching module and the hot hashtable caching module are used to cache hot data and cold data, respectively. The first-level Bloom filter filters keys. If the key does not exist, it is added with a key popularity value of 0. If the key exists, the second-level Bloom filter is searched. If the key does not exist in the second-level Bloom filter, it is directly inserted with a key popularity value of 1. If the key exists, the third-level Bloom filter is searched. If the key does not exist in the third-level Bloom filter, it is inserted with a key popularity value of 3. If the key still exists, its popularity value is directly set to the maximum value of 128. The GPUsort component is used to sort cold data entering the GPUsort component; The GPU cache component is used to back up cold data that enters the GPUsort component for sorting; The driver module is used to coordinate the memory component's call to the GPUsort component; The disk component uses a hot layer and a cold layer structure to manage the key metadata disk SSTable file. The hot layer is managed using a hash function, and the cold layer is managed in an ordered manner.

2. A key-value storage method based on a cache gradient cold and hot data hierarchical mechanism, characterized in that: The method is applied to the key-value storage system based on the cache gradient cold and hot data tiering mechanism as described in claim 1, and the method includes the following steps: S1, Cascaded loop Bloom Filter precisely filters the key in the key-value pair, separating hot data from cold data; S2. The hot data enters the hot hashtable cache module, and the cold data enters the cold hashtable cache module; S3. When the cold data accumulation in the cold hashtable cache module reaches a set number, it enters the GPUsort component for sorting. When the cold data accumulation reaches a set number, it is also backed up to the GPU cache component. S4. The cold data is sorted by the GPUsort component and then stored in the disk component.

3. The key-value storage method based on a cache gradient cold and hot data hierarchical mechanism according to claim 2, characterized in that: The hot data key has a popularity value of 1, 3, or 128; the cold data key has a popularity value of 0.

4. The key-value storage method based on a cache gradient cold and hot data hierarchical mechanism according to claim 2, characterized in that: The GPU cache component is used for hot data read optimization operations, and the steps of the GPU cache component for read optimization operations include: S31. Determine if the search data is hot data; if it is hot data; S32. Search the memory component according to the query. If found, return directly. S33. If no result is found, perform a parallel search within the GPU cache component. If a result is found, return the result. S34. If not found, search the disk components.

5. A key-value storage method based on a cache gradient cold and hot data hierarchical mechanism according to claim 2, characterized in that: The hashtable in the hot HashTable module and the cold HashTable module records the key offset, value offset, and key popularity value.

6. The key-value storage method based on a cache gradient cold and hot data hierarchical mechanism according to claim 2, characterized in that: The hot layer is managed using a hash function, which includes the following steps: S41. Record the total heat value written to the SSTable file on disk to distinguish between cold and hot files; S42. The hot layer in the disk SSTable file is partitioned using a hash function, and key data with similar prefixes are grouped into one partition; S43. Store data with low total popularity values ​​in a slower storage medium partition, and store data with high total popularity values ​​in a faster storage medium partition.

7. A key-value storage system and method based on a cache gradient cold and hot data tiering mechanism according to claim 2, characterized in that: The value file in the key-value pair adopts an in-situ update strategy, which includes the following steps: S21. Track invalid keys in the stored Value file after key-value separation; S22. If the key is directly inserted without updates, use the append method to insert the value into the file. S23. If the key is updated, find the value offset, locate the position of the invalid value file, and insert the new value file. S24. If the space of the Value file at the original address is insufficient to insert the current Value file, continue to insert using the append method.

Images