Data processing method and apparatus, and device and storage medium

Abstract

The embodiments of the present application relate to the technical field of computers. Provided are a data processing method and apparatus, and an electronic device and a computer-readable storage medium. The method comprises: using a first hash function and a second hash function to encode, into a first hash value and a second hash value, each key in key value data to be stored; traversing first hash values and second hash values of all keys in said key value data, and determining parameter information of a perfect hash corresponding to said key value data, wherein the parameter information of the perfect hash comprises: an index of a slot corresponding to each key; and storing said key value data according to the index of the slot corresponding to each key. The solutions of the embodiments of the present application solve the problems of IO overheads of disks being high and the query efficiency being low during data query.

Description

Data processing method, device, equipment and storage medium Technical Field

[0001] The present application relates to the field of computer technology, and more specifically, to a data processing method, device, electronic device, and computer-readable storage medium. Background Art

[0002] One of the most important functions of a database is data retrieval. Especially when the data volume is large and complex, how to efficiently search for data is the main problem currently facing databases.

[0003] Currently, the vast majority of data storage formats are based on an ordered approach. Data is sorted into a specified order and then grouped into pages or blocks for storage on a storage medium (e.g., HDD or SSD). The page size generally matches the storage system's page size. During the query phase, range indexing is used to locate the data page or block, and then the specified content is retrieved from that data block. This type of algorithm generally achieves a search efficiency of O(logN). When the data volume is very large and the index cannot be fully cached, this results in increased disk I / O overhead, significantly reducing query efficiency.

[0004] Summary of the Invention

[0005] The embodiments of the present application provide a data processing method, device, electronic device and computer-readable storage medium, which can solve the problems of high disk IO overhead and low query efficiency during data query.

[0006] According to a first aspect of an embodiment of the present application, a data processing method is provided, the method comprising:

[0007] Using a first hash function and a second hash function, encoding each key in the key-value data to be stored into a first hash value and a second hash value;

[0008] Traversing the first hash values ​​and the second hash values ​​of all keys in the key-value data to be stored, and determining parameter information of a perfect hash corresponding to the key-value data to be stored, wherein the parameter information of the perfect hash includes: an index of a storage unit slot corresponding to each key;

[0009] The key-value data to be stored is stored according to the index of the slot corresponding to each key.

[0010] According to a second aspect of an embodiment of the present application, a data processing device is provided, the device comprising:

[0011] A processing unit is configured to encode each key in the key-value data to be stored into a first hash value and a second hash value using a first hash function and a second hash function; and traverse the first hash values ​​and the second hash values ​​of all keys in the key-value data to be stored to determine parameter information of a perfect hash corresponding to the key-value data to be stored, wherein the parameter information of the perfect hash includes: an index of a storage unit slot corresponding to each key;

[0012] The storage unit is used to store the key-value data to be stored according to the index of the slot corresponding to each key.

[0013] According to a third aspect of an embodiment of the present application, an electronic device is provided, comprising: a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the data processing method described in the first aspect above.

[0014] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the data processing method described in the first aspect are implemented.

[0015] The beneficial effects of the technical solution provided by the embodiments of the present application are:

[0016] By traversing the first hash value and second hash value of all keys in the key-value data to be stored, determining the index of the slot corresponding to each key, and storing the key-value data to be stored according to the index of the slot corresponding to each key, the size of the data index can be reduced, and the disk IO overhead can be reduced when performing data queries, thereby improving query efficiency, thereby solving the problem of high disk IO overhead and low query efficiency during data queries. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for describing the embodiments of the present application.

[0018] FIG1 is a schematic diagram of the structure of LSM-Tree in the related art;

[0019] FIG2 is a schematic diagram of the structure of an LSM-Tree for implementing the data processing method provided in an embodiment of the present application;

[0020] FIG3 is a schematic diagram of a data storage format table based on a perfect hash algorithm provided in an embodiment of the present application;

[0021] FIG4 is a flow chart of a data processing method provided in an embodiment of the present application;

[0022] FIG5 is another schematic diagram of a data storage format provided by an embodiment of the present application;

[0023] FIG6 is a schematic structural diagram of a data processing device provided in an embodiment of the present application;

[0024] FIG7 is a schematic structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] The following describes the embodiments of the present application in conjunction with the accompanying drawings. It should be understood that the embodiments described below in conjunction with the accompanying drawings are exemplary descriptions for explaining the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions of the embodiments of the present application.

[0026] Those skilled in the art will understand that, unless otherwise stated, the singular forms "a," "an," "said," and "the" used herein may also include plural forms. It should be further understood that the terms "including" and "comprising" used in the embodiments of the present application mean that the corresponding features can be implemented as the presented features, information, data, steps, operations, elements, and / or components, but do not exclude implementation as other features, information, data, steps, operations, elements, components, and / or combinations thereof supported by the present technical field. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, the element can be directly connected or coupled to the other element, or it can refer to the element and the other element establishing a connection relationship through an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein indicates at least one of the items defined by the term, for example, "A and / or B" indicates implementation as "A," or implementation as "A," or implementation as "A and B."

[0027] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0028] First, several terms involved in this application are introduced and explained:

[0029] The perfect hash algorithm is used to map a given set of K keys to N different values ​​(hash(key_i) is not equal to hash(key_j)_{i is not equal to j}), that is, there is no hash collision.

[0030] The Log-Structured-Merge-Tree (LSM-tree) is a data structure used for key-value storage. Unlike B+ trees or red-black trees, it lacks a strict tree-like data structure. Instead, it employs a multi-layered storage structure that spans memory and disk, storing increasing amounts of data from top to bottom. LSM-trees utilize sequential writes to improve write performance. While this layered design slightly reduces read performance, the trade-off in read performance for high write performance has made LSM-trees a very popular storage structure.

[0031] Figure 1 is a schematic diagram of the structure of LSM-Tree in related technologies. As shown in Figure 1, LSM-Tree consists of three important components: MemTable, Immutable MemTable and SSTable.

[0032] MemTable: (memory table) is the data structure of LSM-Tree in memory, used to store the latest data and organize this data in order according to the key.

[0033] When a MemTable reaches a certain size, it is converted to an Immutable MemTable. An Immutable MemTable is an intermediate state between a MemTable and an on-disk SSTable. During this conversion process, write operations are handled by the new MemTable, without blocking data updates.

[0034] SSTable: (Sorted String Table) is the data structure of LSM-Tree on disk. SSTable is a format for storing key-value pairs, sorted by key. It is an ordered structure and can also be called a sorted string table. An SSTable will consist of multiple sorted files called segments. These segments are immutable once written to disk. Each Segment contains a series of Block blocks of configurable size. The index of these Blocks will be stored at the end of the SSTable to help quickly find specific Blocks. When an SSTable is opened, the index table will be loaded into memory, and then a binary search will be performed in the memory index based on the key. After finding the disk offset corresponding to the key, the corresponding block data will be read from the disk.

[0035] Flush operation: When the memory in the LSM-tree is full, the data in the memory needs to be written to the disk and organized into an SSTable structure. This process is called a flush operation.

[0036] Compaction operation: In order to maintain the tree structure of the LSM-tree, when the number of SSTable files in the Li layer exceeds the preset value, the SSTables in the Li layer and the SSTables in the Li+1 layer with overlapping key values ​​need to be merged and sorted while discarding some obsolete key values. This process is called a compaction operation.

[0037] One of the most important functions of a database is data retrieval. Especially when the data volume is large and complex, how to efficiently search for data is the main problem currently facing databases.

[0038] The relevant solutions include the following:

[0039] Solution 1: Build a hash index on the SSTable data format. SST is a table format that stores data sequentially and is widely used in various key-value databases. When building a table, the data is divided according to a certain block size (usually 4K), and the data in the block is arranged in the order of the key. Then, the maximum key of each block and the file location information of the corresponding block are used as new key-value pairs and stored in sequence in the metadata index block. When searching, first read the index block information, and then use the key to index to the specific block and file location in a binary search. Then, after reading the content of the block, locate the corresponding data through a binary search.

[0040] This solution improves the performance of point queries by adding another layer of hash index to each data block based on the SSTable. When constructing a data block, each key is hashed and its location information is recorded in the hash table. If there is a hash conflict, that is, a key already exists at that location, the location is marked as unavailable. When querying, the hash index is first queried. If the location is valid, the key location can be directly located and the data can be read. If the location is invalid, it will fall back to the binary search algorithm.

[0041] Solution 2: Cuckoo Hash table format. The Cuckoo Hash algorithm is a hash algorithm that effectively resolves hash conflicts. Its query time complexity is theoretically constant and has good algorithmic characteristics. When using Cuckoo Hash to build a table, first select n groups of hash algorithms, that is, n hash tables. When inserting a key, start the calculation from the first hash function. If the position is not occupied, the insertion is successful. If there is already data at that position, the data at that position is removed and the new data is inserted. The removed data is then reinserted into the next hash table. If the last hash table still has data that has been kicked out, the calculation continues from the first hash, and so on. If the number of attempts is too large, you may need to reselect the hash function or expand the size of the hash table. When querying data, similar to the insertion process, start searching from the first hash table. If there is no match, continue searching the next hash table until the last hash table.

[0042] Solution 3: Hash + tree full memory index solution. This solution builds the storage location information of all data into the memory index.

[0043] When building a data index, data is first mapped to different partitions using a hash algorithm to prevent data skew. Data within the same partition is indexed using a tree structure, such as a red-black tree or B-Tree. The data index contains metadata, including the data's storage location and timestamp. This metadata is typically stored in a fixed-size space. When querying data, the partition is first located using the hash table. The data's storage location is then retrieved from the tree-based index structure, directly retrieving the data from a specific location on the file or disk.

[0044] In order to achieve efficient query performance, most existing technologies improve data retrieval efficiency by optimizing indexes. Among them, the hash algorithm is the preferred choice for most optimization solutions because of its good algorithmic characteristics. However, the existing implementation solutions using the hash algorithm still have many shortcomings. Specifically:

[0045] Solution 1, described above, improves index search efficiency by adding a hash index to the range index. However, when a hash conflict occurs, it falls back to a binary search algorithm. One drawback of binary search and similar algorithms is that their time complexity is higher than the constant level, at O(logN). Furthermore, during actual algorithm execution, CPU cache misses are very high, resulting in significant CPU overhead. This solution fails to effectively address the impact of hash conflicts, which presents challenges for scenarios with high real-time requirements.

[0046] Solution 2 above uses the well-characterized Cuckoo Hash algorithm to resolve hash conflicts. Theoretically, this algorithm has a constant time complexity for query and insertion, which is the number of hash tables. Choosing too few hash tables can lead to more conflicts and rehashing during the construction process. Excessive hash tables means more query I / O. In actual database applications, some databases load entire tables into memory for data lookups to reduce I / O overhead, which also results in greater memory consumption.

[0047] Moreover, in the actual application of the above-mentioned solutions 1 and 2, both the S index and the data block need to be loaded into the memory first, and then the data is located according to the data index information. This type of algorithm only speeds up the data indexing speed in the memory, and does not significantly reduce the disk IO.

[0048] Solution 3 above only requires loading the data index into memory and then locating the data based on the memory information. Data does not need to be pre-loaded into memory. However, the memory solution in Solution 3 is constructed using a hash + tree approach, resulting in a query complexity of O(logn). In actual applications, the index information for a single key can reach tens of bytes, resulting in significant memory overhead.

[0049] The data processing method, device, electronic device, computer-readable storage medium, and computer program product provided in this application are intended to solve the above technical problems in the prior art.

[0050] The following describes several exemplary embodiments to illustrate the technical solutions of the embodiments of the present application and the technical effects produced by the technical solutions of the present application. It should be noted that the following embodiments can refer to, draw on, or combine with each other, and the same terms, similar features, and similar implementation steps in different embodiments will not be repeated.

[0051] FIG2 is a schematic diagram of the structure of the LSM-Tree for implementing the data processing method provided in the embodiment of the present application. The LSM-Tree shown in FIG2 includes: MemTable, Immutable MemTable and PH Table (Perfect Hash Table).

[0052] MemTable: (memory table) is the data structure of LSM-Tree in memory, used to store the latest data and organize the data in order according to the index order of the slot corresponding to the key.

[0053] When the MemTable reaches a certain size, it will be converted into an Immutable MemTable. Immutable MemTable is an intermediate state between converting the MemTable into a PHTable on disk.

[0054] PHTable (Perfect Hash Table, Sorted String Table) is the on-disk data structure of an LSM-Tree. A PHTable is a data storage format based on a perfect hashing implementation. It is an ordered structure, sorted by the index of the slot corresponding to the key. As shown in Figure 3, a PHTable can be composed of multiple segments. Each segment contains a series of blocks, each block contains a series of slots, and each slot is used to store a key-value pair.

[0055] In the embodiment of the present application, a Key-Value database is constructed based on LSM-Tree, providing high-performance reading and writing and lower cost advantages.

[0056] Using the LSM-Tree structure, the database first places all written data into a data structure with memory as the storage medium and records the write data log (WAL) to recover data from abnormal conditions and ensure high reliability. When the cached data in the memory reaches a certain amount, it is flushed to the persistent storage medium in the format of a perfect hash table and first placed in the L0 layer. When the data in the L0 layer reaches a certain amount, it is merged with the data in the lower layer to generate an updated data table file, and so on. When executing a query operation, the search starts with the data in the memory. If the key to be searched does not exist, the search is carried out from L0 to the last layer of the data table in sequence until the search is successful or the traversal is completed.

[0057] This application proposes a data storage format table based on a perfect hash algorithm, which greatly reduces the size of the data index and ensures a maximum of one disk IO overhead during data retrieval. It can still provide lower and more stable query latency even in the case of massive data.

[0058] The following is a detailed description of the data storage solution, data update (merging) solution, and data reading solution involved in a data processing method proposed in this application.

[0059] First, the implementation process of the data storage solution is described in detail.

[0060] An embodiment of the present application provides a data processing method, as shown in FIG4 , comprising:

[0061] S11, using a first hash function and a second hash function, encoding each key in the key-value data to be stored into a first hash value and a second hash value;

[0062] S12. Traverse the first hash values ​​and the second hash values ​​of all keys in the key-value data to be stored, and determine parameter information of a perfect hash corresponding to the key-value data to be stored, wherein the parameter information of the perfect hash includes: an index of a storage unit slot corresponding to each key;

[0063] S13. Store the key-value data to be stored according to the index of the slot corresponding to each key.

[0064] Specifically, in this embodiment, in step S11, two hash functions (i.e., a first hash function and a second hash function) can be used to map the string of each key in the key-value data to be stored into two 32-bit integers, recorded as hash1 (first hash value) and hash2 (second hash value). Among them, the two hash functions can be Djb2Hash and CityHash. In the embodiment of the present application, the purpose of selecting two independent hash functions is to increase the dispersion of the hash hash and facilitate subsequent calculations.

[0065] In this embodiment, by traversing the first hash value and the second hash value of all keys in the key-value data to be stored, the index of the slot corresponding to each key is determined, and the key-value data to be stored is stored according to the index of the slot corresponding to each key. This can reduce the size of the data index, reduce the disk IO overhead when performing data query, and improve query efficiency.

[0066] In some embodiments, S12 may include:

[0067] S121, dividing all keys in the key-value data to be stored into multiple hash buckets;

[0068] S122. Sort the multiple hash buckets from largest to smallest according to the number of keys in each bucket;

[0069] S123: Traverse the multiple hash buckets according to the order of the multiple hash buckets, and determine parameter information of the perfect hash corresponding to each hash bucket.

[0070] Specifically, in this embodiment, we first need to determine the hash bucket load level c, that is, the expected number of keys in each bucket. A larger c results in fewer buckets, a higher computational load (number of retries), and a smaller hash function size (number of parameters). A smaller c results in more buckets, a relatively lower computational load, and a larger hash function size. Therefore, based on practical considerations, c = 5 is generally chosen to balance computational load and memory consumption. The desired number of buckets, m = N / c, is then calculated.

[0071] According to the perfect hashing algorithm, all keys are divided into two sets S1 and S2. The data in S1 accounts for 60% of the total data, and the number of corresponding buckets is p1 = 0.3 * m. The number of buckets corresponding to S2 is the remaining part m - p1. The bucket to which each key belongs is calculated according to the following formula 1:

[0072] Sort the buckets from largest to smallest based on the number of keys they occupy.

[0073] Starting from the first bucket in the sort order, calculate the parameter information for the perfect hash.

[0074] In some embodiments, the parameter information of the perfect hash also includes: the first parameter and the second parameter of the perfect hash corresponding to each hash bucket, and S122 may specifically include: determining the first parameter and the second parameter of the perfect hash corresponding to the current hash bucket, and the index of the slot corresponding to each key in the current hash bucket; wherein, the first parameter and the second parameter of the perfect hash corresponding to the current hash bucket ensure that the slot corresponding to each key in the current hash bucket is not occupied.

[0075] In this embodiment, the size of the initial perfect hash table is determined to be n=γ scale *N, where γ scale The initial value of can be chosen to be 1.1, that is, the size of the hash table is slightly larger than the amount of data, which can reduce the probability of hash collisions and reduce the calculation time under the premise of increasing a certain amount of memory space.

[0076] For the current bucket j If the previous bucket j-1 α has been calculated j-1 (which can correspond to the first parameter above) and β j-1 (corresponding to the second parameter above), then reuse the previous bucket j-1 The parameters are: α j =α j-1 and β j =βj-1 , and calculate the bucket according to the following formula 2 j Each key in i Corresponding slot i : slot i =(β j +α j *hash1(key i )+hash2(key i )mod n formula 2

[0077] If key i Corresponding slot i Already occupied, you need to reselect α j and β j Calculation is performed until there is no conflict or the maximum number of attempts is reached. j All keys corresponding to the slot i If all are not occupied, then determine the current α j and β j .

[0078] If all buckets are successfully found to meet the conditions of α and β, the parameter information of the perfect hash is determined, and the key-value data to be stored can be stored according to the index of the slot corresponding to each key. If all buckets are not found to meet the conditions of α and β, then expand γ scale =γ scale +step, where step can be selected as 0.05, thereby expanding the size of the hash table and reducing the probability of collision. Repeat the above steps S121-S123 until the parameter information of the perfect hash is determined.

[0079] In one embodiment, if the current hash bucket is the first-ranked hash bucket among the multiple hash buckets, S122 may specifically include:

[0080] S1a. For each key in the current hash bucket, determine the index of the slot corresponding to the key based on the first hash value and the second hash value of the key, the initial perfect hash table, and the randomly selected first initial parameter and the second initial parameter;

[0081] S2a. If the slot corresponding to each key in the current hash bucket is not occupied, determine that the first initial parameter is the first parameter of the current hash bucket, and the second initial parameter is the second parameter of the current hash bucket;

[0082] S3a. If the slot corresponding to the first key in the current hash bucket is occupied, repeat the following steps:

[0083] S31a, reselecting the first initial parameter and the second initial parameter;

[0084] S32a, determining the index of the slot corresponding to each key according to the first hash value and the second hash value of each key in the current hash bucket, the reselected first initial parameter and the second initial parameter, and the initial perfect hash table;

[0085] S33a, determine whether the slot corresponding to each key is occupied;

[0086] S34a: If the slot corresponding to each key is not occupied, determine the reselected first initial parameter as the first parameter of the current hash bucket, and determine the reselected second initial parameter as the second parameter of the current hash bucket.

[0087] For example, assuming there are 100 keys and 5 hash buckets, the hash buckets are sorted from large to small according to the number of keys in each hash bucket: hash bucket 2, hash bucket 3, hash bucket 1, hash bucket 5, and hash bucket 4.

[0088] For each key in the first-order hash bucket 2, based on hash1 and hash2 of each key, the initial perfect hash table (size n=110, i.e., mapping 100 keys to 100 slots out of 110 slots), and randomly selected parameters α1 and β1, the index of the slot corresponding to each key is determined by the above formula 2. If the slot corresponding to each key in hash bucket 2 is not occupied, α1 is determined as the first parameter of hash bucket 2, and β1 is determined as the second parameter of hash bucket 2. If the slot corresponding to at least one key (e.g., key2) in hash bucket 2 is occupied, it is necessary to repeat the "process of reselecting parameters and calculating based on formula 2" until the slot corresponding to each key in hash bucket 2 is unoccupied, or until the "process of reselecting parameters and calculating based on formula 2" is repeated a preset number of times and the parameters of hash bucket 2 are still not determined.

[0089] In another embodiment, if the current hash bucket is a non-first-order hash bucket among the multiple hash buckets, S122 may specifically include:

[0090] S1b. For each key in the current hash bucket, determine the index of the slot corresponding to the key based on the first hash value and the second hash value of the key, the initial perfect hash table, and the first parameter and the second parameter corresponding to the previous hash bucket;

[0091] S2b. If the slot corresponding to each key in the current hash bucket is not occupied, determine the first parameter corresponding to the previous hash bucket as the first parameter of the current hash bucket, and the second parameter corresponding to the previous hash bucket as the second parameter of the current hash bucket;

[0092] S3b. If the slot corresponding to the second key in the current hash bucket is occupied, repeat the following steps:

[0093] S31b, reselecting the first parameter and the second parameter;

[0094] S32b, determining the index of the slot corresponding to each key according to the first hash value and the second hash value of each key in the current hash bucket, the reselected first parameter and the second parameter, and the initial perfect hash table;

[0095] S33b, determine whether the slot corresponding to each key is occupied;

[0096] S34b: If the slot corresponding to each key is not occupied, determine that the reselected first parameter is the first parameter of the current hash bucket, and the reselected second parameter is the second parameter of the current hash bucket.

[0097] In the above embodiment, if the step of repeatedly executing has been performed a preset number of times and at least one slot corresponding to a key in the current hash bucket is still occupied, it is determined that the first parameter and the second parameter corresponding to the current hash bucket are not determined.

[0098] Continuing with the above example, if the current hash bucket is hash bucket 1 and the parameters of hash bucket 3 have been determined to be α2 and β2, then for each key in hash bucket 1, based on each key's hash1 and hash2, the initial perfect hash table (size n = 110), and the parameters α2 and β2 of hash bucket 3, the index of the slot corresponding to each key is determined using the above formula 2. If the slot corresponding to each key in hash bucket 1 is unoccupied, α2 is determined as the first parameter of hash bucket 1, and β2 is determined as the second parameter of hash bucket 1. If the slot corresponding to at least one key (for example, key28) in hash bucket 1 is occupied, the "reselect parameters and calculate based on formula 2" process needs to be repeated until the slot corresponding to each key in hash bucket 1 is unoccupied, or until the "reselect parameters and calculate based on formula 2" process is repeated a preset number of times and the parameters of hash bucket 1 are still not determined.

[0099] In some other embodiments, if the first parameter and the second parameter corresponding to at least one hash bucket among the plurality of hash buckets are not determined, the method further includes:

[0100] S14 (not shown in the figure): Repeat the following steps until the first parameter and the second parameter corresponding to each hash bucket in the multiple hash buckets are determined:

[0101] S1c, expanding the initial perfect hash table according to a preset step size to obtain a perfect hash table;

[0102] S2c. For each key in the first hash bucket, determine the index of the slot corresponding to each key according to the first hash value and the second hash value of each key, the obtained first parameter and the second parameter, and the perfect hash table;

[0103] S3c, determine whether the slot corresponding to each key is occupied;

[0104] S4c. If the slot corresponding to each key is not occupied, determine that the obtained first parameter is the first parameter of the first hash bucket, and the obtained second parameter is the second parameter of the first hash bucket;

[0105] S5c: If the slot corresponding to the third key in the first hash bucket is occupied, repeat the following steps:

[0106] S51c, reacquiring the first parameter and the second parameter;

[0107] S52c, determining the index of the slot corresponding to each key according to the first hash value and the second hash value of each key, the first parameter and the second parameter that were retrieved, and the perfect hash table;

[0108] S53c, determining whether the slot corresponding to each key is occupied;

[0109] S54c: If the slot corresponding to each key is not occupied, determine that the re-acquired first parameter is the first parameter of the first hash bucket, and the re-acquired second parameter is the second parameter of the first hash bucket.

[0110] Specifically, in this embodiment, if the first hash bucket is the hash bucket that is ranked first among multiple hash buckets, the first parameter and second parameter obtained are randomly selected first parameters and second parameters; if the first hash bucket is a hash bucket that is not ranked first among multiple hash buckets (for example, the hash bucket that is ranked second), the first parameter and second parameter obtained are the first parameter and second parameter of the previous hash bucket (for example, the hash bucket that is ranked first).

[0111] In other words, if steps S31a-S34a and / or steps S31b-S34b in the above embodiment have been repeated a predetermined number of times, for example, 100 times, and one or more slots corresponding to keys in the first hash bucket are still occupied, then the first and second parameters that are satisfied by all hash buckets have not been successfully found. In this case, the perfect hash parameter information can be recalculated by expanding the perfect hash table.

[0112] Continuing with the above example, if steps S31b-S34b have been repeated a preset number of times for hash bucket 5 and the corresponding parameters have not yet been determined, i.e., there is a conflict when mapping 100 keys to 100 of the 110 slots, then the initial perfect hash table (size n=110) is expanded to a perfect hash table (size n=115, i.e., mapping 100 keys to 100 of the 115 slots), and the perfect hash parameter information is recalculated starting from hash bucket 2 to hash bucket 4. The calculation process for hash bucket 2 can refer to steps S1a-S34a, and the calculation process for each of hash buckets 3, hash bucket 1, hash bucket 5, and hash bucket 4 can refer to steps S1b-S34b. For the sake of brevity, these steps are not repeated here.

[0113] In some embodiments, step S13 in the above embodiment may specifically include:

[0114] Writing the key-value data to be stored into at least one storage block according to the index of the slot corresponding to each key, where the number of slots stored in each storage block is determined based on the size of the storage unit of the storage device;

[0115] The at least one storage block is compressed and then written into the storage device.

[0116] In this embodiment, the length information of each slot can be written into the head position of the corresponding storage block according to the index of the slot corresponding to each key, and the key value data in each slot can be written into the corresponding storage block in sequence. For details, please refer to Figure 5.

[0117] It should be noted that the size of the storage block (the number of storage slots) can be chosen to be consistent with the size of the disk or memory page.

[0118] For example, a storage block can store 20 slots, so 100 slots are written into 20 storage blocks in the order of their respective indexes. For example, slots with indexes 1 to 20 are written into storage block 1. Assuming that slots with indexes 21, 38, and 40 are empty, slots with indexes 22 to 43 are written into storage block 2, and so on.

[0119] It should be noted that, in this embodiment, the slot index and the order of the key may correspond (for example, key2 is stored in slot2), or may not correspond (for example, key10 is stored in slot5).

[0120] In order to further reduce the size of the data file, a file compression method can be used to compress the data in each storage block before writing it to the storage device.

[0121] In the above embodiment, the following may also be included:

[0122] Record the file offset position information of each storage block to obtain offset data;

[0123] Determine a portion of the first hash value or the second hash value of each key as signature information of each key to obtain signature data;

[0124] The step of compressing the at least one storage block and then writing the compressed data into the storage device comprises:

[0125] Generate a data table file based on the compressed at least one storage block, the offset data, the signature data, the parameter information of the perfect hash, and the attribute information of the data table;

[0126] The data table file is written into the storage device using a log structure merge tree LSM-Tree structure.

[0127] In this embodiment, each storage block is written to the file while recording the offset position information of the file where the block is located. The signature information of the key in each slot can also be calculated, for example, the last 8 bits of the second hash value of each key are used as the signature information of the corresponding key: signature(key i )=hash2(key i )&0XFF Formula 3

[0128] The signature information of all keys in the slots forms a signature array (corresponding to the signature data mentioned above). Its function is to filter out non-existent data during the query process. Because any key can be mapped to a valid slot based on Formula 2, if a key does not match the signature information of the corresponding slot, it means that the key does not exist in the slot and can be directly ignored, reducing file reading overhead.

[0129] When all blocks are written, the offset information, signature array, perfect hash parameter information, and data table attribute information of each block are written into the footer of the file to generate a data table file.

[0130] For example, the specific format of the data table file in this embodiment can be seen in Figure 3. The attribute information of the data table may include one or more of the following information: data table version information, the number of keys in the data table, the average size of keys and values ​​in the data table, the compression algorithm type, etc.

[0131] It should be noted that, in this embodiment, the last 8 bits or the first 8 bits of the hash value of each key can also be used as the signature information of the corresponding key, and this embodiment of the present application does not limit this.

[0132] The above data table files are generated based on static data. For real-time update scenarios, data will continue to change over time, new key-value pairs will be inserted, and old key-value pairs will be deleted or updated. Therefore, the data table also needs to be dynamically updated to provide queries for new data.

[0133] Therefore, the embodiment of the present application also provides a method for data updating (or data merging). Combined with the structure of LSM-Tree, each time new data is written (inserting new data or updating old data), it will first be written into the memory table. When the size of the memory table reaches a certain threshold, the memory table is marked as non-updatable, and a new memory table is regenerated for subsequent writing. The non-updatable data table then becomes a static data set. The data in the data set is written into the L0 layer (top layer) in the format of a perfect hash table. When the L0 layer data reaches a certain threshold, the L0 data is merged with the data in the lower layer, and so on.

[0134] That is, if the amount of key-value data written into the first memory table reaches a second preset threshold, the first memory table is marked as a non-updatable state, and the key-value data in the first memory table is determined as the key-value data to be stored.

[0135] Traditional LSM-Trees store data in ascending order of key, so file merges are typically performed using a merge sort. However, the data in this application is stored in ascending order of slot index corresponding to the key. Therefore, traditional LSM-Tree file merges are not suitable for this application. Therefore, this application adopts the solution described in the following embodiment.

[0136] Writing the data table file to the storage device using an LSM-Tree structure includes performing at least one of the following operations:

[0137] Write the data table file into the L0 layer of the LSM-Tree structure;

[0138] If the first data file stored in the Li layer reaches a first preset threshold, the first data file is merged into the Li+1 layer, where i ranges from 0 to n and n is a positive integer greater than 1.

[0139] In one embodiment, merging the first data file into layer Li+1 includes:

[0140] If the L i+1 layer does not store data, the first data file is directly written into the L i+1 layer;

[0141] If the second data file is already stored in layer L i+1, the first data file and the second data file are divided into S data segments according to the hash value of the key, and the following operations are performed on the j-th data segment belonging to the first data file and the j-th data segment belonging to the second data file:

[0142] Obtaining first hash values ​​and second hash values ​​of all keys in the two j-th data segments;

[0143] Determine the index of the slot corresponding to each key based on the first hash value and the second hash value of each key;

[0144] Determine the corresponding operation based on the key value data in each slot;

[0145] Wherein, S is a preset value greater than 1, and j ranges from 1 to S.

[0146] In the above embodiment, the corresponding operation is determined according to the key value data in each slot, including:

[0147] If the current slot is empty, no operation will be performed;

[0148] If the key-value data in the current slot is the key-value data in the first data file, or the key-value data in the second data file, determining to execute the step of writing the key-value data in the current slot into the storage device according to the index of the current slot;

[0149] If the key-value data in the current slot includes: the key-value data in the first data file and the key-value data in the second data file, determining to write the new key-value data into the current slot, and executing the step of writing the new key-value data in the current slot into the storage device according to the index of the current slot,

[0150] The new key-value data is obtained based on the key-value data in the first data file and the key-value data in the second data file.

[0151] Specifically, in this embodiment, the new data file (which may correspond to the first data file mentioned above) is identified as delta, and the old data file (which may correspond to the second data file mentioned above) is identified as base. In order to reduce memory usage and improve concurrency, the data file will be divided into S segments according to the hash value of the key: segment (key i )=(hash1(key i )+hash2(keyi )) mod S

[0152] In this way, the data in each segment is independent of each other and can be processed in parallel.

[0153] For the data in the j-th segment in the delta data and the j-th segment in the base data, perform the following operations until S segments are traversed:

[0154] 1. Get all the keys in the j-th segment in the delta data and the j-th segment in the base data, and calculate hash1 and hash2 for each key.

[0155] In order to reduce the amount of calculation, hash1 and hash2 can also be persisted to the data file during the file generation stage and can be read directly to reduce the calculation overhead.

[0156] 2. For all keys obtained, the slot index is calculated according to the data storage process in the above embodiment, and the delta data and base data belonging to the same slot are aggregated together.

[0157] For each slot, it is divided into the following situations:

[0158] A.slot is empty;

[0159] B. slot contains only delta data;

[0160] C. slot contains only base data;

[0161] D. slot includes both delta data and base data.

[0162] For case A, this can be ignored. For case B (newly added data) or case C (unmodified data), no merge operation is required and the data can be written directly to the new file. For case D (updating old data), an update operation is required. The final updated data is stored in the corresponding slot. The data is then organized and written to the new data file with reference to the content and related descriptions shown in Figures 3 and 5 to complete the data file merge process.

[0163] In this embodiment, the specific update operation can be a customized operation based on the actual needs of the user. For example:

[0164] Function 1: Data is directly overwritten. In this case, the base data in D is overwritten by the delta data, and the data in the slot in the new file is the delta data.

[0165] Function 2: Base data is accumulated by delta data. If base data is a numeric value and delta is an accumulation operation, the data in the slot of the new file is the result of adding the base data and delta data. For example, for a bank deposit and withdrawal business scenario, if the base data is "Deposit - 1 million" and the delta data is "Withdrawal - 100,000", the updated data will be "Deposit - 900,000".

[0166] In an embodiment of the present application, by using a perfect hash algorithm as a data table for data index, a query method with at most only one IO overhead can be provided.

[0167] In addition, the data table merging method uses the characteristics of perfect hashing to merge new data and old data belonging to the same location by grouping, and optimizes the merging process based on the presence or absence of new data.

[0168] The data index table using the perfect hash algorithm is combined with LSM-Tree to form a complete key-value DB database.

[0169] In addition, the embodiment of the present application further provides a data processing method for realizing data reading (also referred to as data query). Based on the above embodiment, the above method may further include:

[0170] S15 (not shown in the figure): in response to the received read instruction, read the corresponding key-value data from the data table file of the storage device, wherein the read instruction is used to instruct to read the key-value data of a specific key.

[0171] In this embodiment, the process of reading the key-value data of a specific key may include:

[0172] If the specific key is found in the memory table, the key value data corresponding to the specific key is read;

[0173] If the key is not found in the memory table, the data table file of the L0 layer is traversed to search for the specific key;

[0174] If it is not found in the data table file of the Li layer, it is searched in the specific data table file of the Li+1 layer until the Ln layer is found, or the specific key is found and the key value data corresponding to the specific key is read, wherein the specific data table file is the data table file corresponding to the key value range containing the specific key.

[0175] The process of reading the key-value data of the specific key from any data table file includes:

[0176] Read the offset data, signature data, and perfect hash parameter information in the data table file, wherein the perfect hash parameter information includes the first parameter and the second parameter of the perfect hash corresponding to all keys in the data table file;

[0177] determining a first hash value and a second hash value for the specific key;

[0178] Determine the index of the slot corresponding to the specific key based on the first hash value and the second hash value of the specific key and parameter information of the perfect hash;

[0179] Determining signature information corresponding to the specific key based on the first hash value or the second hash value of the specific key;

[0180] Search the signature data based on the signature information corresponding to the specific key. If no matching signature information is found, determine that the specific key does not exist in the data table file, and return a message indicating that the key value data does not exist.

[0181] If matching signature information is found, the key value data in the slot in the corresponding storage block is read according to the slot index and the offset data;

[0182] If the specific key is the same as the key in the read key-value data, the read key-value data is returned; otherwise, information indicating that the key-value data does not exist is returned.

[0183] Specifically, according to the structural characteristics of the LSM-Tree and the data merging process in the above embodiment, the data at the upper layer is more recent, so the query process is also performed from top to bottom:

[0184] First, search the in-memory table. Because in-memory tables are typically used for dynamic updates, data structures such as SkipList, Vector, and HashTable are often used. For the specified key (i.e., the key to be queried), search using their respective search algorithms. If the search is successful, the result is returned. Otherwise, the search must be performed in the persistent data table file.

[0185] When searching from a data table file, the search proceeds from the top layer to the bottom layer. If no matching key is found in layer L_i, the search continues in the next layer L_(i+1) until a match is found or the last layer is reached. This completes the entire data search process.

[0186] First, the search begins at the L0 layer. Because data files in L0 are not arranged in order, it is necessary to traverse all data files in L0 from the newest to the oldest. For non-L0 data, data table files are arranged in monotonically increasing order by key. Simply select the data files that contain the specified key range to search.

[0187] When searching for a specified key in any data table file, the index information is loaded from the footer of the data file into the memory, including: parameter information of the perfect hash index, block offset data and signature data. Then, the hash_1 and hash_2 of the specified key are calculated, and the slot index corresponding to the specified key is calculated by the above formula 2. The corresponding signature information is calculated by the above formula 3. The signature information corresponding to the slot is compared. If there is no match, it is determined that the value of the specified key must not exist in the current data table. If the match is successful, according to the block position corresponding to the slot, the block is read from the data table file and the content of the corresponding slot is parsed. If the parsed key is equal to the specified key, the search is successful, otherwise it is returned as not existing. At this point, the search for the data file is completed.

[0188] The solution provided in this application offers efficient point-to-point query performance, requiring at most one disk IO operation per query, making data query latency predictable. This offers significant benefits for systems requiring low and stable query latency. Compared to existing solutions using range indexes, which have a query complexity of O(logn), higher query latency, and lack predictable query latency, the variance in query latency for different keys can be significant.

[0189] Although some solutions can theoretically reduce the overhead of a single disk IO query by storing the entire data storage location index in memory, such solutions incur significant memory overhead compared to the solution provided by this application. Because the solution provided by this application uses an in-memory index that only retains the parameters of the perfect hash function, each key only occupies approximately 2 bytes. This allows for more data to be stored than existing solutions under the same memory environment.

[0190] The embodiment of the present application provides a data processing device. As shown in FIG6 , the data processing device 1000 may include: a processing unit 1001 and a storage unit 1002 , wherein:

[0191] The processing unit 1001 is configured to encode each key in the key-value data to be stored into a first hash value and a second hash value using a first hash function and a second hash function; and traverse the first hash values ​​and the second hash values ​​of all keys in the key-value data to be stored to determine parameter information of a perfect hash corresponding to the key-value data to be stored, wherein the parameter information of the perfect hash includes: an index of a storage unit slot corresponding to each key;

[0192] The storage unit 1002 is used to store the key-value data to be stored according to the index of the slot corresponding to each key.

[0193] Furthermore, when determining the parameter information of the perfect hash corresponding to the key-value data to be stored, the processing unit 1001 is specifically configured to:

[0194] Divide all keys in the key-value data to be stored into multiple hash buckets;

[0195] Sort the multiple hash buckets from largest to smallest according to the number of keys in each bucket;

[0196] The multiple hash buckets are traversed according to the order of the multiple hash buckets to determine parameter information of the perfect hash corresponding to each hash bucket.

[0197] Furthermore, the parameter information of the perfect hash also includes: a first parameter and a second parameter of the perfect hash corresponding to each hash bucket;

[0198] When determining the parameter information of the perfect hash corresponding to each hash bucket, the processing unit 1001 is specifically configured to: determine a first parameter and a second parameter of the perfect hash corresponding to the current hash bucket, and an index of a slot corresponding to each key in the current hash bucket;

[0199] The first parameter and the second parameter of the perfect hash corresponding to the current hash bucket ensure that the slot corresponding to each key in the current hash bucket is not occupied.

[0200] Furthermore, if the current hash bucket is the first-order hash bucket among the multiple hash buckets, the processing unit 1001, when determining the parameter information of the perfect hash corresponding to the current hash bucket, is specifically configured to:

[0201] For each key in the current hash bucket, determine the index of the slot corresponding to the key based on the first hash value and the second hash value of the key, the initial perfect hash table, and the randomly selected first initial parameter and the second initial parameter;

[0202] If the slot corresponding to each key in the current hash bucket is not occupied, determining that the first initial parameter is the first parameter of the current hash bucket, and the second initial parameter is the second parameter of the current hash bucket;

[0203] If the slot corresponding to the first key in the current hash bucket is occupied, repeat the following steps:

[0204] Reselect the first initial parameter and the second initial parameter;

[0205] Determine the index of the slot corresponding to each key based on the first hash value and the second hash value of each key in the current hash bucket, the reselected first initial parameter and the second initial parameter, and the initial perfect hash table;

[0206] Determine whether the slot corresponding to each key is occupied;

[0207] If the slot corresponding to each key is not occupied, the reselected first initial parameter is determined to be the first parameter of the current hash bucket, and the reselected second initial parameter is determined to be the second parameter of the current hash bucket.

[0208] Furthermore, if the current hash bucket is not the first hash bucket in the order of the multiple hash buckets, the processing unit 1001 is specifically configured to:

[0209] For each key in the current hash bucket, determine the index of the slot corresponding to the key based on the first hash value and the second hash value of the key, the initial perfect hash table, and the first parameter and the second parameter corresponding to the previous hash bucket;

[0210] If the slot corresponding to each key in the current hash bucket is not occupied, determine the first parameter corresponding to the previous hash bucket as the first parameter of the current hash bucket, and the second parameter corresponding to the previous hash bucket as the second parameter of the current hash bucket;

[0211] If the slot corresponding to the second key in the current hash bucket is occupied, repeat the following steps:

[0212] Reselect the first parameter and the second parameter;

[0213] Determine the index of the slot corresponding to each key based on the first hash value and the second hash value of each key in the current hash bucket, the newly selected first parameter and the second parameter, and the initial perfect hash table;

[0214] Determine whether the slot corresponding to each key is occupied;

[0215] If the slot corresponding to each key is not occupied, the reselected first parameter is determined to be the first parameter of the current hash bucket, and the reselected second parameter is determined to be the second parameter of the current hash bucket.

[0216] Furthermore, if the step of repeatedly executing has been executed a preset number of times and a slot corresponding to at least one key in the current hash bucket is still occupied, it is determined that the first parameter and the second parameter corresponding to the current hash bucket are not determined.

[0217] Furthermore, if the first parameter and the second parameter corresponding to at least one hash bucket among the multiple hash buckets are not determined, the processing unit 1001 is further configured to:

[0218] Repeat the following steps until the first parameter and the second parameter corresponding to each hash bucket in the plurality of hash buckets are determined:

[0219] Expanding the initial perfect hash table according to a preset step size to obtain a perfect hash table;

[0220] For each key in the first hash bucket, determine the index of the slot corresponding to each key according to the first hash value and the second hash value of each key, the obtained first parameter and the second parameter, and the perfect hash table;

[0221] Determine whether the slot corresponding to each key is occupied;

[0222] If the slot corresponding to each key is not occupied, determine that the obtained first parameter is the first parameter of the first hash bucket, and the obtained second parameter is the second parameter of the first hash bucket;

[0223] If the slot corresponding to the third key in the first hash bucket is occupied, repeat the following steps:

[0224] Re-obtain the first parameter and the second parameter;

[0225] Determine the index of the slot corresponding to each key based on the first hash value and the second hash value of each key, the retrieved first parameter and the second parameter, and the perfect hash table;

[0226] Determine whether the slot corresponding to each key is occupied;

[0227] If the slot corresponding to each key is not occupied, the re-acquired first parameter is determined to be the first parameter of the first hash bucket, and the re-acquired second parameter is determined to be the second parameter of the first hash bucket.

[0228] Furthermore, the storage unit 1002 is specifically used to: write the key-value data to be stored into at least one storage block according to the index of the slot corresponding to each key, and the number of slots stored in each storage block is determined based on the size of the storage unit of the storage device; compress the at least one storage block and write it into the storage device.

[0229] Furthermore, the storage unit 1002 is specifically configured to: write the length information of each slot into the head position of the corresponding storage block according to the index of the slot corresponding to each key, and write the key value data in each slot into the corresponding storage block in sequence.

[0230] Furthermore, the processing unit 1001 is further configured to: record the file offset position information of each storage block to obtain offset data; determine a portion of the first hash value or the second hash value of each key as signature information of each key to obtain signature data;

[0231] The storage unit 1002 is specifically used to: generate a data table file based on the compressed at least one storage block, the offset data, the signature data, the parameter information of the perfect hash and the attribute information of the data table; and write the data table file to the storage device using a log structure merge tree LSM-Tree structure.

[0232] Furthermore, when the storage unit 1002 adopts the LSM-Tree structure to write the data table file into the storage device, it is specifically configured to:

[0233] Write the data table file into the L0 layer of the LSM-Tree structure;

[0234] If the first data file stored in the Li layer reaches a first preset threshold, the first data file is merged into the Li+1 layer, where i ranges from 0 to n and n is a positive integer greater than 1.

[0235] Furthermore, merging the first data file into the Li+1 layer includes:

[0236] If the L i+1 layer does not store data, the first data file is directly written into the L i+1 layer;

[0237] If the second data file is already stored in layer L i+1, the first data file and the second data file are divided into S data segments according to the hash value of the key, and the following operations are performed on the j-th data segment belonging to the first data file and the j-th data segment belonging to the second data file:

[0238] Obtaining first hash values ​​and second hash values ​​of all keys in the two j-th data segments;

[0239] Determine the index of the slot corresponding to each key based on the first hash value and the second hash value of each key;

[0240] Determine the corresponding operation based on the key value data in each slot;

[0241] Wherein, S is a preset value greater than 1, and j ranges from 1 to S.

[0242] Furthermore, determining the corresponding operation based on the key value data in each slot includes:

[0243] If the current slot is empty, no operation will be performed;

[0244] If the key-value data in the current slot is the key-value data in the first data file, or the key-value data in the second data file, determining to execute the step of writing the key-value data in the current slot into the storage device according to the index of the current slot;

[0245] If the key-value data in the current slot includes: the key-value data in the first data file and the key-value data in the second data file, determining to write the new key-value data into the current slot, and executing the step of writing the new key-value data in the current slot into the storage device according to the index of the current slot,

[0246] The new key-value data is obtained based on the key-value data in the first data file and the key-value data in the second data file.

[0247] Furthermore, the processing unit 1001 is also used to: if the amount of key-value data written into the first memory table reaches a second preset threshold, mark the first memory table as a non-updatable state, and determine the key-value data in the first memory table as the key-value data to be stored.

[0248] Furthermore, it also includes: a reading unit 1003, which is used to read corresponding key-value data from the data table file of the storage device in response to a received read instruction, wherein the read instruction is used to instruct to read the key-value data of a specific key.

[0249] Furthermore, the reading unit 1003 is specifically configured to:

[0250] If the specific key is found in the memory table, the key value data corresponding to the specific key is read;

[0251] If the key is not found in the memory table, the data table file of the L0 layer is traversed to search for the specific key;

[0252] If it is not found in the data table file of the Li layer, it is searched in the specific data table file of the Li+1 layer until the Ln layer is found, or the specific key is found and the key value data corresponding to the specific key is read, wherein the specific data table file is the data table file corresponding to the key value range containing the specific key.

[0253] Furthermore, when the reading unit 1003 reads the key value data of the specific key from any data table file, it is specifically used to:

[0254] Read the offset data, signature data, and perfect hash parameter information in the data table file, wherein the perfect hash parameter information includes the first parameter and the second parameter of the perfect hash corresponding to all keys in the data table file;

[0255] determining a first hash value and a second hash value for the specific key;

[0256] Determine the index of the slot corresponding to the specific key based on the first hash value and the second hash value of the specific key and parameter information of the perfect hash;

[0257] Determining signature information corresponding to the specific key based on the first hash value or the second hash value of the specific key;

[0258] Search the signature data based on the signature information corresponding to the specific key. If no matching signature information is found, determine that the specific key does not exist in the data table file, and return a message indicating that the key value data does not exist.

[0259] If matching signature information is found, the key value data in the slot in the corresponding storage block is read according to the slot index and the offset data;

[0260] If the specific key is the same as the key in the read key-value data, the read key-value data is returned; otherwise, information indicating that the key-value data does not exist is returned.

[0261] The device of the embodiment of the present application can execute the method provided by the embodiment of the present application, and its implementation principle is similar. The actions performed by each module in the device of each embodiment of the present application correspond to the steps in the method of each embodiment of the present application. For the detailed functional description of each module of the device, please refer to the description in the corresponding method shown in the previous text, and will not be repeated here.

[0262] An embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored on the memory. The processor executes the above-mentioned computer program to implement the steps of the data processing method. Compared with the prior art, it can achieve: by traversing the first hash value and the second hash value of all keys in the key-value data to be stored, determining the index of the slot corresponding to each key, and storing the key-value data to be stored according to the index of the slot corresponding to each key, which can reduce the size of the data index and reduce the disk IO overhead when performing data query, thereby improving query efficiency, thereby solving the problem of large disk IO overhead and low query efficiency during data query.

[0263] In an optional embodiment, an electronic device is provided, as shown in FIG7 , and the electronic device 4000 shown in FIG7 includes: a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, such as through a bus 4002. Optionally, the electronic device 4000 may further include a transceiver 4004, which may be used for data exchange between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the transceiver 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present application.

[0264] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like.

[0265] Bus 4002 may include a path for transmitting information between the aforementioned components. Bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, for example. Bus 4002 may be divided into an address bus, a data bus, a control bus, and so on. For ease of illustration, FIG7 shows only one thick line, but this does not mean that there is only one bus or only one type of bus.

[0266] The memory 4003 may be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disc storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium that can be used to carry or store computer programs and can be read by a computer, without limitation herein.

[0267] The memory 4003 is used to store the computer program for executing the embodiment of the present application, and the execution is controlled by the processor 4001. The processor 4001 is used to execute the computer program stored in the memory 4003 to implement the steps shown in the above method embodiment.

[0268] An embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps and corresponding contents of the aforementioned method embodiment can be implemented.

[0269] An embodiment of the present application also provides a computer program product, including a computer program, which can implement the steps and corresponding contents of the aforementioned method embodiment when executed by a processor.

[0270] It should be understood that, although each operation step is indicated by arrows in the flowchart of the embodiment of the present application, the order of implementation of these steps is not limited to the order indicated by the arrows. Unless otherwise clearly stated herein, in some implementation scenarios of the embodiment of the present application, the implementation steps in each flowchart can be performed in other orders according to demand. In addition, some or all of the steps in each flowchart can include multiple sub-steps or multiple stages based on actual implementation scenarios. Some or all of these sub-steps or stages can be executed at the same time, and each sub-step or stage in these sub-steps or stages can also be executed at different times respectively. Under different scenarios at the execution time, the execution order of these sub-steps or stages can be flexibly configured according to demand, and the embodiment of the present application does not limit this.

[0271] The above description is only an optional implementation method for some implementation scenarios of this application. It should be pointed out that for ordinary technicians in this technical field, without departing from the technical concept of the solution of this application, the use of other similar implementation methods based on the technical ideas of this application also falls within the protection scope of the embodiments of this application.

Claims

1. A data processing method, comprising: Using a first hash function and a second hash function, encoding each key in the key-value data to be stored into a first hash value and a second hash value; Traversing the first hash values ​​and the second hash values ​​of all keys in the key-value data to be stored, and determining parameter information of a perfect hash corresponding to the key-value data to be stored, wherein the parameter information of the perfect hash includes: an index of a storage unit slot corresponding to each key; The key-value data to be stored is stored according to the index of the slot corresponding to each key.

2. The method according to claim 1, wherein: The determining of parameter information of a perfect hash corresponding to the key-value data to be stored includes: Divide all keys in the key-value data to be stored into multiple hash buckets; Sort the multiple hash buckets from large to small according to the number of keys in each bucket; The multiple hash buckets are traversed according to the order of the multiple hash buckets to determine parameter information of the perfect hash corresponding to each hash bucket.

3. The method according to claim 2, wherein: The parameter information of the perfect hash also includes: a first parameter and a second parameter of the perfect hash corresponding to each hash bucket; The determining of parameter information of the perfect hash corresponding to each hash bucket includes: Determine the first and second parameters of the perfect hash corresponding to the current hash bucket, as well as the index of the slot corresponding to each key in the current hash bucket; The first parameter and the second parameter of the perfect hash corresponding to the current hash bucket ensure that the slot corresponding to each key in the current hash bucket is not occupied.

4. The method according to claim 3, wherein: If the current hash bucket is the first-ranked hash bucket among the multiple hash buckets, determining parameter information of the perfect hash corresponding to the current hash bucket includes: For each key in the current hash bucket, determine the index of the slot corresponding to the key according to the first hash value and the second hash value of the key, the initial perfect hash table, and the randomly selected first initial parameter and the second initial parameter; If the slot corresponding to each key in the current hash bucket is not occupied, determine that the first initial parameter is the first parameter of the current hash bucket, and the second initial parameter is the second parameter of the current hash bucket; If the slot corresponding to the first key in the current hash bucket is occupied, repeat the following steps: Reselect the first initial parameter and the second initial parameter; Determine the index of the slot corresponding to each key according to the first hash value and the second hash value of each key in the current hash bucket, the reselected first initial parameter and the second initial parameter, and the initial perfect hash table; Determine whether the slot corresponding to each key is occupied; If the slot corresponding to each key is not occupied, determine that the reselected first initial parameter is the first parameter of the current hash bucket, and the reselected second initial parameter is the second parameter of the current hash bucket.

5. The method according to claim 3, wherein: If the current hash bucket is a non-first-order hash bucket among the multiple hash buckets, determining parameter information of the perfect hash corresponding to the current hash bucket includes: For each key in the current hash bucket, determine the index of the slot corresponding to the key according to the first hash value and the second hash value of the key, the initial perfect hash table, and the first parameter and the second parameter corresponding to the previous hash bucket; If the slot corresponding to each key in the current hash bucket is not occupied, determine that the first parameter corresponding to the previous hash bucket is the first parameter of the current hash bucket, and the second parameter corresponding to the previous hash bucket is the second parameter of the current hash bucket; If the slot corresponding to the second key in the current hash bucket is occupied, repeat the following steps: Reselect the first parameter and the second parameter; reselecting the first parameter and the second parameter according to the first hash value and the second hash value of each key in the current hash bucket, And the initial perfect hash table, determine the index of the slot corresponding to each key; Determine whether the slot corresponding to each key is occupied; If the slot corresponding to each key is not occupied, it is determined that the reselected first parameter is the first parameter of the current hash bucket, and the reselected second parameter is the second parameter of the current hash bucket.

6. The method according to claim 4 or 5, wherein: If the repeatedly executed step has been executed for a preset number of times and there is still at least one slot corresponding to a key in the current hash bucket occupied, it is determined that the first parameter and the second parameter corresponding to the current hash bucket have not been determined.

7. The method according to claim 6, wherein: If the first parameter and the second parameter corresponding to at least one hash bucket among the plurality of hash buckets are not determined, the method further includes: Repeat the following steps until the first parameter and the second parameter corresponding to each hash bucket in the plurality of hash buckets are determined: Expanding the initial perfect hash table according to a preset step size to obtain a perfect hash table; For each key in the first hash bucket, determine the index of the slot corresponding to each key according to the first hash value and the second hash value of each key, the obtained first parameter and the second parameter, and the perfect hash table; Determine whether the slot corresponding to each key is occupied; If the slot corresponding to each key is not occupied, determine that the first parameter obtained is the first parameter of the first hash bucket, and the second parameter obtained is the second parameter of the first hash bucket; If the slot corresponding to the third key in the first hash bucket is occupied, repeat the following steps: Re-obtain the first parameter and the second parameter; Determine the index of the slot corresponding to each key according to the first hash value and the second hash value of each key, the first parameter and the second parameter retrieved, and the perfect hash table; Determine whether the slot corresponding to each key is occupied; If the slot corresponding to each key is not occupied, it is determined that the re-acquired first parameter is the first parameter of the first hash bucket, and the re-acquired second parameter is the second parameter of the first hash bucket.

8. The method according to any one of claims 1 to 5 and 7, wherein: The storing the key-value data to be stored according to the index of the slot corresponding to each key includes: Writing the key-value data to be stored into at least one storage block according to the index of the slot corresponding to each key, wherein the number of slots stored in each storage block is determined based on the size of the storage unit of the storage device; The at least one storage block is compressed and then written into the storage device.

9. The method according to claim 8, wherein: The step of writing the key-value data to be stored into at least one storage block according to the index of the slot corresponding to each key includes: According to the index of the slot corresponding to each key, the length information of each slot is written into the head position of the corresponding storage block, and the key value data in each slot is written into the corresponding storage block in sequence.

10. The method according to claim 9, further comprising: Record the file offset position information of each storage block to obtain the offset data; Determine a portion of the first hash value or the second hash value of each key as signature information of each key to obtain signature data; The step of compressing the at least one storage block and then writing the compressed at least one storage block into the storage device comprises: Generate a data table file based on the compressed at least one storage block, the offset data, the signature data, the parameter information of the perfect hash, and the attribute information of the data table; The data table file is written into the storage device using a log structure merge tree LSM-Tree structure.

11. The method according to claim 10, wherein: The adopting the LSM-Tree structure to write the data table file into the storage device includes performing at least one of the following operations: Writing the data table file into the L0 layer of the LSM-Tree structure; If the first data file stored in the Li layer reaches a first preset threshold, the first data file is merged into the Li+1 layer, where i is 0 to n, and n is a positive integer greater than 1.

12. The method according to claim 11, wherein: The merging of the first data file into the Li+1 layer includes: If the Li+1 layer does not store data, the first data file is directly written into the Li+1 layer; If the second data file has been stored in the Li+1 layer, the first data file and the second data file are divided into S data segments according to the hash value of the key, and the following operations are performed for the j-th data segment belonging to the first data file and the j-th data segment belonging to the second data file: Obtaining first hash values ​​and second hash values ​​of all keys in the two j-th data fragments; Determine the index of the slot corresponding to each key according to the first hash value and the second hash value of each key; Determine the corresponding operation based on the key value data in each slot; Among them, S is a preset value greater than 1, and j ranges from 1 to S.

13. The method according to claim 12, wherein: Determining the corresponding operation according to the key value data in each slot includes: If the current slot is empty, no operation is performed; If the key value data in the current slot is the key value data in the first data file, or the key value data in the second data file, determine to execute the step of writing the key value data in the current slot into the storage device according to the index of the current slot; If the key value data in the current slot includes: the key value data in the first data file and the key value data in the second data file, determine to write the new key value data into the current slot, and execute the step of writing the new key value data in the current slot into the storage device according to the index of the current slot, The new key-value data is obtained based on the key-value data in the first data file and the key-value data in the second data file.

14. The method according to any one of claims 1-5, 7, 9-13, further comprising: If the amount of key-value data written into the first memory table reaches a second preset threshold, the first memory table is marked as a non-updatable state, and the key-value data in the first memory table is determined as the key-value data to be stored.

15. The method according to claim 11, further comprising: In response to the received read instruction, the corresponding key-value data is read from the data table file of the storage device, wherein the read instruction is used to instruct to read the key-value data of a specific key.

16. The method according to claim 15, wherein: The process of reading the key-value data for a specific key includes: If the specific key is found in the memory table, the key value data corresponding to the specific key is read; If the key is not found in the memory table, the data table file of the L0 layer is traversed to search for the specific key; If it is not found in the data table file of the Li layer, then search in the specific data table file of the Li+1 layer until the Ln layer is found, or, find the specific key and read the key value data corresponding to the specific key, wherein the specific data table file is a data table file corresponding to the key value range containing the specific key.

17. The method according to claim 16, wherein: The process of reading the key value data of the specific key from any data table file includes: Read the offset data, signature data and parameter information of the perfect hash in the data table file, wherein the parameter information of the perfect hash includes the first parameter and the second parameter of the perfect hash corresponding to all the keys in the data table file; determining a first hash value and a second hash value for the particular key; Determine the index of the slot corresponding to the specific key based on the first hash value and the second hash value of the specific key and the parameter information of the perfect hash; Determine signature information corresponding to the specific key based on the first hash value or the second hash value of the specific key; Searching the signature data based on the signature information corresponding to the specific key, if no matching signature information is found, determining that the specific key does not exist in the data table file, and returning information indicating that the key value data does not exist; If matching signature information is found, the key value data in the slot in the corresponding storage block is read according to the slot index and the offset data; If the specific key is the same as the key in the read key value data, the read key value data is returned; otherwise, the prompt key value is returned. Information that the data does not exist.

18. A data processing device, comprising: a processing unit, configured to encode each key in the key-value data to be stored into a first hash value and a second hash value by using a first hash function and a second hash function; And, traversing the first hash values ​​and the second hash values ​​of all keys in the key-value data to be stored, determining parameter information of a perfect hash corresponding to the key-value data to be stored, wherein the parameter information of the perfect hash includes: an index of a storage unit slot corresponding to each key; The storage unit is used to store the key-value data to be stored according to the index of the slot corresponding to each key.

19. An electronic device comprising: A memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the data processing method according to any one of claims 1 to 17.

20. A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the data processing method according to any one of claims 1 to 17.

Images