A method, apparatus, and storage medium for writing and reading KV data based on multi-level indexes.

Abstract

The present invention discloses a KV data writing and reading method, device and storage medium based on multi-level indexing, wherein the KV data writing and reading method based on multi-level indexing includes a KV data writing process, wherein the KV data writing process includes: performing a hash operation on the key of the KV data to obtain the hash value hash(key) of the key; selecting a corresponding B-family tree according to the hash value hash(key); judging whether the value of the KV data exceeds a preset threshold; if the judgment result is no, directly storing the KV data in the corresponding B-family tree; if the judgment result is yes, storing the KV data in a hash index bithash file. The present invention can achieve a time complexity of O(1) through a combination of a B-family tree and a hash index bithash file through multi-level indexing, greatly improving the read operation efficiency while ensuring that the write operation performance is not weakened, and providing a KV database storage engine with better performance, which is suitable for scenarios with a large number of read operations.

Description

Technical Field

[0001] The present invention relates to the field of database storage technology, and in particular to a KV data writing and reading method, device and storage medium based on multi-level indexing. Background Art

[0002] KV database refers to Key-value database, which is a kind of database that stores data in key-value pairs. In order to improve the query speed of the database, it is necessary to establish a database index for the database. There are three common database index data structures for KV database: hash table, ordered array and search tree.

[0003] The hash table stores key-value pairs. If you use a hash table to store indexes, you must first use a hash function to convert the key into a corresponding index position in the array, and then put the value at this position. If multiple keys get the same index after conversion, then add a linked list here to store key-value objects with the same hash value. If you are given a specified key and asked to query the corresponding value, you must first convert the index value of the array according to the hash function. If there are multiple values ​​at this position, then traverse the linked list. The disadvantages of the hash table are obvious. Since the hash table is unordered, the speed of interval query using the hash index is very slow.

[0004] The performance of ordered arrays in equal value queries and range queries is excellent. The key-value array is stored in order. At this time, if you want to find the value corresponding to the key, you can use binary search to quickly get it, and the time complexity is O(log(N)). Although the binary search can greatly improve the efficiency of queries, the time complexity of O(log(N)) is still inefficient for database queries. In addition, the key-value array is stored in order, which is troublesome when updating data. To insert a record in the middle, all the subsequent records must be moved, which is too costly.

[0005] Search tree, a data structure, it is a set of n (n>=0) finite nodes with a hierarchical relationship. It is called a "tree" because it looks like an upside-down tree, that is, its root is facing up and its leaves are facing down. Search tree query generally uses binary search, and the time complexity is O(log(N)).

[0006] In view of this, the technical problem to be solved by the present invention is: how to improve the performance of the KV database storage engine in the scenario of a large number of read operations, and it is necessary to improve the read operation efficiency under the premise of ensuring that the write operation performance is not weakened. Summary of the invention

[0007] In order to solve the above technical problems, the present invention proposes a KV data writing and reading method, device and storage medium based on multi-level indexing. The specific technical solutions are as follows:

[0008] A KV data writing and reading method based on a multi-level index includes a KV data writing process, wherein the KV data writing process includes:

[0009] Perform hash operation on the key of KV data to obtain the hash value of the key, hash(key);

[0010] Select the corresponding B-family tree according to the hash value hash(key);

[0011] Determine whether the value of the KV data exceeds the preset threshold;

[0012] If the judgment result is no, the KV data is directly stored in the corresponding B-family tree;

[0013] If the judgment result is yes, the KV data is stored in the hash index bithash file.

[0014] As an optional implementation of the present invention, in the KV data writing and reading method based on multi-level index, storing the KV data in a hash index bithash file includes:

[0015] The hash index bithash file includes a data area, a hash area and a bitmap area;

[0016] Write the KV data to the data area of ​​the hash index bithash file, and record the offset and size S[offset+size] in the data area;

[0017] Calculate the hash value hash(key) of the key in the KV data, and update the bit value of the hashID bit corresponding to the hash value hash(key) in the bitmap in the bitmap area;

[0018] The hashID, key, S[offset+size] are used as hash index records when key-value data is written, and the hash index records are written into the corresponding partition in the hash index bithash file.

[0019] As an optional implementation of the present invention, in the KV data writing and reading method based on multi-level indexing, the hash index bithash file includes:

[0020] Verification area: Use cyclic verification code to verify the data area;

[0021] Conflict area: used to store keys with the same hash value and their offset and data size in the data area;

[0022] And the footer area: fixed-length storage data area, check area, conflict area, hash area, and offset and size of the bitmap area.

[0023] As an optional implementation of the present invention, in the KV data writing and reading method based on multi-level indexing, the hash index bithash file presets a storage threshold. When the data stored in the hash index bithash file reaches the preset storage threshold, or the remaining storage space of the hash index bithash file is insufficient to store new KV data, a new hash index bithash file is created.

[0024] As an optional implementation of the present invention, in the KV data writing and reading method based on multi-level indexing, when it is determined that the value of the KV data exceeds a preset threshold, the storage location of the KV data on the B-family tree is determined according to the hash value hash(key), and the hash value hash(key) and the index information corresponding to the KV data stored in the hash index bithash file are stored at the storage location.

[0025] As an optional implementation of the present invention, in the KV data writing and reading method based on multi-level index, before performing hash operation on the key of the KV data, the method includes:

[0026] The write-ahead log WAL that records the KV data is written into the memtable. At this time, the data is stored in memory.

[0027] When the memtable is full, a hash operation is performed on the key of the KV data.

[0028] As an optional implementation manner of the present invention, the KV data writing and reading method based on multi-level indexing includes a KV data reading process, and the KV data reading process includes:

[0029] Calculate the hash value of the query key based on the query key hash(key)';

[0030] Find the corresponding B-family tree according to the hash value hash(key)' of the query key, and read the value data of the query key from the B-family tree;

[0031] If the read is successful, the value data stored in the read B-family tree is returned; if the read fails, the value data of the query key is read from the hash index bithash file;

[0032] If the read is successful, the value data in the read hash index bithash file is returned. If the read fails, the relevant information that the value data corresponding to the query key was not read is returned.

[0033] As an optional implementation of the present invention, in the KV data writing and reading method based on multi-level index, before calculating the hash value hash(key)' of the query key according to the query key, the method includes:

[0034] Read in the memory table memtable according to the query key;

[0035] If the read is successful, the value data read from the memory table memtable is directly returned. If the read fails, the hash value hash(key)' of the query key is calculated based on the query key.

[0036] The present invention also provides a KV data writing and reading device based on a multi-level index, including a KV data writing module, wherein the KV data writing module includes:

[0037] The hash operation unit performs hash operation on the key of the KV data to obtain the hash value of the key, hash(key);

[0038] A B-family tree storage unit selects a corresponding B-family tree according to the hash value hash(key);

[0039] The processing unit determines whether the value of the KV data exceeds a preset threshold;

[0040] If the judgment result is no, the KV data is directly stored in the corresponding B-family tree in the B-family tree storage unit;

[0041] The bithash file storage unit stores the KV data in the hash index bithash file of the hash index bithash file storage unit if the judgment result is yes.

[0042] The present invention also provides a storage medium storing a computer executable program, characterized in that when the computer executable program is executed, the KV data writing and reading method based on multi-level indexing as described in any one of the above items is implemented.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] The KV data writing and reading method based on multi-level indexing of the present invention can achieve O(1) time complexity through the combination of B-tree and hash index bithash file and multi-level indexing. Compared with the existing KV storage engine (such as embedded KV storage engine RocksDB), which requires O(log n) time complexity because of the need for binary search, it greatly improves the read operation efficiency while ensuring that the write operation performance is not weakened, and provides a KV database storage engine with better performance, which is suitable for scenarios with a large number of read operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 Data writing process of the KV data writing and reading method based on multi-level indexing in an embodiment of the present invention Figure 1 ;

[0046] Figure 2 Data writing process of the KV data writing and reading method based on multi-level indexing in an embodiment of the present invention Figure 2 ;

[0047] Figure 3 Process of storing KV data in a hash index bithash file according to an embodiment of the present invention Figure 1 ;

[0048] Figure 4 Schematic diagram of the hashID bit marking in the bitmap of the hash index bithash file in the embodiment of the present invention;

[0049] Figure 5 A schematic diagram of processing the same hashID when KV data is stored in a hash index bithash file according to an embodiment of the present invention;

[0050] Figure 6 The processing flow chart of the embodiment of the present invention in which KV data is stored in a hash index bithash file for the same hashID;

[0051] Figure 7 A hash index bithash file data structure according to an embodiment of the present invention;

[0052] Figure 8 A data reading flow chart of a KV data writing and reading method based on multi-level indexing according to an embodiment of the present invention;

[0053] Fig. 9 Flow chart of KV data reading process of KV data writing and reading method based on multi-level index in the embodiment of the present invention Figure 1 ;

[0054] Fig.10Flow chart of KV data reading process of KV data writing and reading method based on multi-level index in the embodiment of the present invention Figure 2 . DETAILED DESCRIPTION

[0055] To make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be described clearly and completely in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them.

[0056] Therefore, the following detailed description of the embodiments of the present invention is not intended to limit the scope of the invention claimed for protection, but merely represents some embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features and technical solutions in the embodiments may be combined with each other.

[0058] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0059] In the description of the present invention, it should be noted that the orientation or positional relationship indicated by the terms "upper", "lower", etc. is based on the orientation or positional relationship shown in the drawings, or the orientation or positional relationship in which the invention product is usually placed when in use, or the orientation or positional relationship commonly understood by those skilled in the art. Such terms are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0060] See also Figure 1 As shown, this embodiment provides a KV data writing and reading method based on a multi-level index, including a KV data writing process, and the KV data writing process includes:

[0061] Perform hash operation on the key of KV data to obtain the hash value of the key, hash(key);

[0062] Select the corresponding B-family tree according to the hash value hash(key);

[0063] Determine whether the value of the KV data exceeds the preset threshold;

[0064] If the judgment result is no, the KV data is directly stored in the corresponding B-family tree;

[0065] If the judgment result is yes, the KV data is stored in the hash index bithash file.

[0066] The B-tree described in this embodiment includes a binary search tree, a B-tree, a B+ tree, and a B* tree. Preferably, a B+ tree is used.

[0067] This embodiment uses a combination of B-tree and hash index bithash files to achieve O(1) time complexity through multi-level indexing. Compared with the existing KV storage engine (such as the embedded KV storage engine RocksDB), which requires O(log n) time complexity due to the need for binary search, it greatly improves the read operation efficiency while ensuring that the write operation performance is not weakened, and provides a KV database storage engine with better performance, which is suitable for scenarios with a large number of read operations.

[0068] As an optional implementation of this embodiment, see Figure 3 and Figure 4 As shown, in order to implement hash indexing for KV data, the embodiment of storing KV data in a hash index bithash file includes:

[0069] The hash index bithash file includes a data area, a hash area and a bitmap area;

[0070] Write the KV data to the data area of ​​the hash index bithash file, and record the offset and size S[offset+size] in the data area;

[0071] Calculate the hash value hash(key) of the key in the KV data, and update the bit value of the hashID bit corresponding to the hash value hash(key) in the bitmap in the bitmap area;

[0072] The hashID, key, S[offset+size] are used as hash index records when key-value data is written, and the hash index records are written into the corresponding partition in the hash index bithash file.

[0073] In this embodiment, hash(key), where key represents the key value of the element, the value of hash(key) represents the hash value calculated by the hash function.

[0074] The so-called bitmap uses a bit to mark the value corresponding to an element, and the key is the element. Since the data is stored in bits, the storage space can be greatly saved.

[0075] This embodiment stores KV data in a hash index bithash file, updates the bit value on the corresponding bitmap according to the hash value hash(key) of the key in the key-value data, and uses it as the hash index record when the key-value data is written; in this way, when performing a key query, the data existence is judged through the bitmap area. If the data does not exist, it can be returned with only one comparison calculation overhead. If the data exists, the key-value data is directly indexed to the data according to the hash index record. Therefore, the file data storage and reading method applied to database storage in this embodiment adopts a new file format through hash index when performing data storage, and the time complexity can reach O(1) when performing key value retrieval, which greatly improves the retrieval efficiency of key-value data.

[0076] As an optional implementation of this embodiment, recording the hash index record in a file in this embodiment includes:

[0077] Sort the hash index records by hashID, write the offset and size of all hashIDs into the hash area of ​​the file in a fixed-length manner, and record the offset and size H[offset+size] of the hash area;

[0078] Serialize the bitmap recorded by the hash index, write it into the file as the bitmap area, and record the offset and size B[offset+size] of the bitmap area;

[0079] The offset and size S[offset+size] of the data area, the offset and size H[offset+size] of the hash area, and the offset and size B[offset+size] of the bitmap area are stored as the footer area of ​​the file.

[0080] After sorting by hashID, this embodiment stores the offset and size S[offset+size] of the data area, the offset and size H[offset+size] of the hash area, and the offset and size B[offset+size] of the bitmap area as the footer area of ​​the file. When querying the key value, if the data existence in the bitmap area is judged as the key value, the data area can be gradually indexed to obtain the key-value data according to the offset and size S[offset+size] of the data area, the offset and size H[offset+size] of the hash area, and the offset and size B[offset+size] of the bitmap area.

[0081] An ideal hash function would achieve the effect that if key1≠key2, then hash(key1)≠hash(key2). However, in reality, it is almost impossible to find a hash function that has different hash values ​​for different keys. In fact, no matter how good a hash function is, hash collisions are unavoidable. For a hash function, no matter how large the storage area (n) is, when the data to be stored is greater than n, there will inevitably be situations where the hash values ​​are the same. This is called a hash collision.

[0082] Since this embodiment performs hash calculation on key-value data, in principle, assuming that the hash algorithm is uniform, the hash values ​​hash(key) obtained by hash calculation of key-value data are different from each other. However, if there are two or more key-value data that have the same hash value hash(key) after hash calculation, a group of identical hashIDs will appear, which will affect the hashID sorting. Therefore, see Figure 5 and Figure 6 As shown, as an optional implementation of this embodiment, this embodiment further includes, before sorting the hash index records by hashID:

[0083] According to the hash index record, the key with the same hashID and the offset and size S[offset+size] corresponding to the key in the data area are packaged into a conflict block, and written into the conflict area of ​​the file in sequence;

[0084] Save the offset and size C[offset+size] of the conflicting block of the key with the same hashID in the conflicting area, and write back C[offset+size] as the unique associated offset and size of the key with the same hashID.

[0085] This embodiment packages key-value data with the same hash value obtained through hash calculation into a conflict block, stores the conflict block in the conflict area of ​​the file, sorts the entire conflict block sharing the same hashID, and uses the offset and size C[offset+size] of the conflict block stored in the conflict area of ​​the file as the offset and size of the key-value data storage corresponding to the hashID.

[0086] This embodiment stores KV data in a bithash file with a hash index. Assuming that the hash algorithm is uniform, a large proportion of data will not fall into the conflict area. For data that does not fall into the conflict area, only one io in the hash area is needed to obtain the offset and size H[offset+size] of the hash area of ​​the data, and one io in the data area is needed to obtain the data through the offset and size S[offset+size] of the data area. Even if the queried key falls into the conflict area, a binary search for the key in the same conflict area only requires a very small io cost, and the overall average query times are constant.

[0087] See also Figure 6 As shown, as an optional implementation of the present embodiment, in the present embodiment, the KV data is stored in a hash index bithash file. In the process of writing the key-value data to the data area of ​​the file, the check code of the key-value data is calculated, the check code is stored in the check area of ​​the file, and the offset and size J[offset+size] of the check code of each key in the check area are recorded; the offset and size J[offset+size] in the check area are saved in the footer area of ​​the file.

[0088] The verification area of ​​this embodiment is used to perform data verification on the data area when opening a file. The footer area of ​​this embodiment can add custom data as a file format verification when reading, and the entire footer area is a fixed size.

[0089] See also Figure 7 As shown, the hash index bithash file described in this embodiment includes:

[0090] Data area: stores compressed key-value data;

[0091] Verification area: Use cyclic verification code to verify the data area;

[0092] Conflict zone: keys with the same hash value and their offset and data size in the data zone will be placed in the conflict zone;

[0093] Hash area: a fixed-length area that stores the offset and size of the data area where the key is located or the conflict area;

[0094] Bitmap area: Use bitmap to store whether the bit corresponding to the hash of the key has a value;

[0095] Footer area: fixed-length storage data area, check area, conflict area, hash area, offset and size of the bitmap area.

[0096] See also Figure 2 As shown, the KV data storage engine of this embodiment includes N B+ trees, and correspondingly includes M hash index bithash files. The hash index bithash file presets a storage threshold. When the data stored in the hash index bithash file reaches the preset storage threshold, or the remaining storage space of the hash index bithash file is insufficient to store new KV data, a new hash index bithash file is created. Therefore, the KV data on a B+ tree may exist in the same hash index bithash file, or there may be different hash index bithash files. Therefore, in order to facilitate index query, as an optional implementation of this embodiment, when it is determined that the value of the KV data exceeds the preset threshold, the storage location of the KV data on the B family tree is determined according to the hash value hash(key), and the hash value hash(key) and the index information corresponding to the KV data stored in the hash index bithash file are stored at the storage location.

[0097] This embodiment performs a threshold judgment on the value of the KV data. When the value exceeds the preset threshold, it is stored in the hash index bithash file. The index is used to facilitate key value query and read the corresponding value.

[0098] As an optional implementation of this embodiment, see Figure 1 As shown, in the KV data writing and reading method based on multi-level index, before performing hash operation on the key of KV data, the method includes:

[0099] The write-ahead log WAL that records the KV data is written into the memtable. At this time, the data is stored in memory.

[0100] When the memtable is full, a hash operation is performed on the key of the KV data.

[0101] Also, see Figure 8 As shown, the KV data writing and reading method based on multi-level indexing described in this embodiment includes a KV data reading process, and the KV data reading process includes:

[0102] Calculate the hash value of the query key based on the query key hash(key)';

[0103] Find the corresponding B-family tree according to the hash value hash(key)' of the query key, and read the value data of the query key from the B-family tree;

[0104] If the read is successful, the value data stored in the read B-family tree is returned; if the read fails, the value data of the query key is read from the hash index bithash file;

[0105] If the read is successful, the value data in the read hash index bithash file is returned. If the read fails, the relevant information that the value data corresponding to the query key was not read is returned.

[0106] In this embodiment, reading the value data of the query key from the hash index bithash file includes opening the hash index bithash file:

[0107] Open the file, read the content of the footer area according to the fixed size, and perform file format verification on the footer area;

[0108] Read the offset and size S[offset+size] of the data area in the footer area of ​​the file, the offset and size J[offset+size] of the check area, the offset and size H[offset+size] of the hash area, and the offset and size B[offset+size] of the bitmap area;

[0109] According to the offset and size B[offset+size] of the bitmap area, the data in the bitmap area is deserialized and stored in the memory;

[0110] According to the offset and size H[offset+size] of the hash area, all hashIDs and corresponding offset+size are loaded into the memory.

[0111] Further, see Fig. 9 and Fig.10 As shown, the KV data reading process of this embodiment includes:

[0112] Perform hash calculation on the query key to obtain hash(key) of the query key, and further obtain hashID=hash(key) of the query key;

[0113] Check whether the key exists in the bitmap area according to the hashID of the query key. If it does not exist, return an empty result and the query ends. If it does exist, proceed to the next step.

[0114] According to the rank of the query key’s hashID in the bitmap area, calculate the offset and size H[offset+size] of the hash area record;

[0115] According to the offset and size H[offset+size] of the hash area, the query data corresponding to the query key is read in the data area or the conflict area.

[0116] Further, in this embodiment, according to the offset and size H[offset+size] of the hash area, reading the query data corresponding to the query key in the data area or the conflict area includes:

[0117] Determine whether it falls into the conflict area according to the offset and size H[offset+size] of the hash area;

[0118] If it is determined that it does not fall in the conflict area, the data is read directly from the data area according to the offset and size S[offset+size]. If it is determined that it falls in the conflict area, the conflict block is determined according to the offset and size C[offset+size] of the corresponding conflict area, and the offset and size S[offset+size] of the query key in the data area are replaced by binary search in the conflict block, and the data is read from the data area.

[0119] As an optional implementation of this embodiment, see Fig. 9 As shown, in this embodiment, according to the sorting rank of the hashID of the query key in the bitmap area, the offset and size H[offset+size] of the hash area record are calculated, including:

[0120] The fixed length of the hash area when writing each H[offset+size] is width, and the writing is in order, so the offset and size H[offset+size] of the query key recorded in the hash area are calculated by rank*width.

[0121] As an optional implementation of this embodiment, the data verification area can adopt a variety of different verification algorithms; the hsah algorithm can select different specific numbers in the footer area and can select different values; the offset+size information of each area in the footer area can be stored separately in the meta area, and the footer area only stores the offset+size of the meta area.

[0122] As an optional implementation of this embodiment, see Figure 8 As shown, in the KV data writing and reading method based on multi-level indexes described in this embodiment, the step of calculating the hash value hash(key)' of the query key according to the query key includes:

[0123] Read in the memory table memtable according to the query key;

[0124] If the read is successful, the value data read from the memory table memtable is directly returned. If the read fails, the hash value hash(key)' of the query key is calculated based on the query key.

[0125] This embodiment also provides a KV data writing and reading device based on a multi-level index, including a KV data writing module, wherein the KV data writing module includes:

[0126] The hash operation unit performs hash operation on the key of the KV data to obtain the hash value of the key, hash(key);

[0127] A B-family tree storage unit selects a corresponding B-family tree according to the hash value hash(key);

[0128] The processing unit determines whether the value of the KV data exceeds a preset threshold;

[0129] If the judgment result is no, the KV data is directly stored in the corresponding B-family tree in the B-family tree storage unit;

[0130] The bithash file storage unit stores the KV data in the hash index bithash file of the hash index bithash file storage unit if the judgment result is yes.

[0131] The KV data writing and reading device based on multi-level indexing of this embodiment includes a B-family tree storage unit and a bithash file storage unit. Through the combination of the B-family tree and the hash index bithash file, and through multi-level indexing, a time complexity of O(1) can be achieved. Compared with the existing KV storage engine (such as the embedded KV storage engine RocksDB), because binary search is required, a time complexity of O(log n) is required. Under the premise of ensuring that the write operation performance is not weakened, the read operation efficiency is greatly improved, and a KV database storage engine with better performance is provided, which is suitable for scenarios with a large number of read operations.

[0132] As an optional implementation of this embodiment, in order to implement hash indexing for KV data, the embodiment described in storing KV data in a hash index bithash file of a hash index bithash file storage unit includes:

[0133] The hash index bithash file includes a data area, a hash area and a bitmap area;

[0134] Write the KV data to the data area of ​​the hash index bithash file, and record the offset and size S[offset+size] in the data area;

[0135] Calculate the hash value hash(key) of the key in the KV data, and update the bit value of the hashID bit corresponding to the hash value hash(key) in the bitmap in the bitmap area;

[0136] The hashID, key, S[offset+size] are used as hash index records when key-value data is written, and the hash index records are written into the corresponding partition in the hash index bithash file.

[0137] The KV data storage engine of this embodiment includes N B+ trees, and correspondingly includes M hash index bithash files. The hash index bithash file has a preset storage threshold. When the data stored in the hash index bithash file reaches the preset storage threshold, or the remaining storage space of the hash index bithash file is insufficient to store new KV data, a new hash index bithash file is created. Therefore, the KV data on a B+ tree may exist in the same hash index bithash file, or there may be different hash index bithash files. Therefore, in order to facilitate index query, as an optional implementation of this embodiment, when it is determined that the value of the KV data exceeds the preset threshold, the storage location of the KV data on the B family tree is determined according to the hash value hash(key), and the hash value hash(key) and the index information corresponding to the KV data stored in the hash index bithash file are stored at the storage location.

[0138] This embodiment performs a threshold judgment on the value of the KV data. When the value exceeds the preset threshold, it is stored in the hash index bithash file. The index is used to facilitate key value query and read the corresponding value.

[0139] As an optional implementation of this embodiment, the KV data writing and reading device based on multi-level indexing, the KV data writing module includes:

[0140] The memory storage unit records the write-ahead log WAL of the KV data and writes it into the memory table memtable. At this time, the data is stored in the memory. When the memory table memtable is full, a hash operation is performed on the key of the KV data.

[0141] At the same time, the KV data writing and reading device based on multi-level indexing described in this embodiment includes a KV data reading module, and the KV data reading module includes:

[0142] The hash operation unit calculates the hash value of the query key according to the query key, hash(key)';

[0143] A B-family tree query unit searches for a corresponding B-family tree according to the hash value hash(key)' of the query key, and reads the value data of the query key from the B-family tree;

[0144] The bithash file query unit returns the value data stored in the read B-family tree if the reading is successful, and reads the value data of the query key from the hash index bithash file if the reading fails;

[0145] If the read is successful, the value data in the read hash index bithash file is returned. If the read fails, the relevant information that the value data corresponding to the query key was not read is returned.

[0146] The KV data reading module of this embodiment includes a file opening unit, which opens the hash index bithash file:

[0147] Open the file, read the content of the footer area according to the fixed size, and perform file format verification on the footer area;

[0148] Read the offset and size S[offset+size] of the data area in the footer area of ​​the file, the offset and size J[offset+size] of the check area, the offset and size H[offset+size] of the hash area, and the offset and size B[offset+size] of the bitmap area;

[0149] According to the offset and size B[offset+size] of the bitmap area, the data in the bitmap area is deserialized and stored in the memory;

[0150] According to the offset and size H[offset+size] of the hash area, all hashIDs and corresponding offset+size are loaded into the memory.

[0151] Further, see Figure 6 and Figure 7 As shown, the KV data reading module of this embodiment performs data query according to the query key, including: performing hash calculation according to the query key to obtain hash(key) of the query key, and further obtaining hashID=hash(key) of the query key;

[0152] Check whether the key exists in the bitmap area according to the hashID of the query key. If it does not exist, return an empty result and the query ends. If it does exist, proceed to the next step.

[0153] According to the rank of the query key’s hashID in the bitmap area, calculate the offset and size H[offset+size] of the hash area record;

[0154] According to the offset and size H[offset+size] of the hash area, the query data corresponding to the query key is read in the data area or the conflict area.

[0155] Further, in this embodiment, according to the offset and size H[offset+size] of the hash area, reading the query data corresponding to the query key in the data area or the conflict area includes:

[0156] Determine whether it falls into the conflict area according to the offset and size H[offset+size] of the hash area;

[0157] If it is determined that it does not fall in the conflict area, the data is read directly from the data area according to the offset and size S[offset+size]. If it is determined that it falls in the conflict area, the conflict block is determined according to the offset and size C[offset+size] of the corresponding conflict area, and the offset and size S[offset+size] of the query key in the data area are replaced by binary search in the conflict block, and the data is read from the data area.

[0158] As an optional implementation of this embodiment, see Figure 7 As shown, in this embodiment, according to the sorting rank of the hashID of the query key in the bitmap area, the offset and size H[offset+size] of the hash area record are calculated, including:

[0159] The fixed length of the hash area when writing each H[offset+size] is width, and the writing is in order, so the offset and size H[offset+size] of the query key recorded in the hash area are calculated by rank*width.

[0160] As an optional implementation of this embodiment, the data verification area can adopt a variety of different verification algorithms; the hsah algorithm can select different specific numbers in the footer area and can select different values; the offset+size information of each area in the footer area can be stored separately in the meta area, and the footer area only stores the offset+size of the meta area.

[0161] As an optional implementation of this embodiment, the KV data writing and reading device based on multi-level indexing described in this embodiment, the KV data reading module includes:

[0162] A memory data query unit, reading data from the memory table memtable according to the query key;

[0163] If the read is successful, the value data read from the memory table memtable is directly returned. If the read fails, the hash value hash(key)' of the query key is calculated based on the query key.

[0164] This embodiment provides a storage medium storing a computer executable program. When the computer executable program is executed, any one of the multi-level index-based KV data writing and reading methods is implemented.

[0165] The storage medium described in the present embodiment may include a data signal propagated in a baseband or as a part of a carrier wave, wherein a readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by an instruction execution system, an apparatus, or a device or used in combination with it. The program code contained on the readable storage medium may be transmitted with any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

[0166] This embodiment also provides an electronic device, including a processor and a memory, wherein the memory is used to store a computer executable program, and when the computer program is executed by the processor, the processor executes the KV data writing and reading method based on multi-level indexing.

[0167] The electronic device is presented in the form of a general-purpose computing device. The processor may be one or more than one and work in coordination. The present invention does not exclude distributed processing, that is, the processor may be dispersed in different physical devices. The electronic device of the present invention is not limited to a single entity, but may also be the sum of multiple physical devices.

[0168] The memory stores a computer executable program, which is usually a machine-readable code. The computer-readable program can be executed by the processor to enable the electronic device to perform the method of the present invention, or at least part of the steps in the method.

[0169] The memory includes a volatile memory, such as a random access memory unit (RAM) and / or a cache memory unit, and may also be a non-volatile memory, such as a read-only memory unit (ROM).

[0170] It should be understood that the electronic device of the present invention may also include elements or components not shown in the above examples. For example, some electronic devices also include display units such as display screens, and some electronic devices also include human-computer interaction elements such as buttons, keyboards, etc. As long as the electronic device can execute the computer-readable program in the memory to implement the method of the present invention or at least part of the steps of the method, it can be considered as an electronic device covered by the present invention.

[0171] Through the above description of the implementation mode, it is easy for those skilled in the art to understand that the present invention can be implemented by hardware capable of executing a specific computer program, such as the system of the present invention, and the electronic processing unit, server, client, mobile phone, control unit, processor, etc. contained in the system. The present invention can also be implemented by computer software that executes the method of the present invention, such as control software executed by a microprocessor, an electronic control unit, a client, a server, etc. However, it should be noted that the computer software that executes the method of the present invention is not limited to being executed by one or a specific hardware entity, and it can also be implemented in a distributed manner by unspecified specific hardware. For computer software, the software product can be stored in a computer-readable storage medium (which can be a CD-ROM, a U disk, a mobile hard disk, etc.), and can also be distributed and stored on the network, as long as it can enable the electronic device to execute the method according to the present invention.

[0172] The above embodiments are only used to illustrate the present invention and are not intended to limit the technical solutions described in the present invention. Although the present invention has been described in detail with reference to the above embodiments, the present invention is not limited to the above specific implementation methods. Therefore, any modification or equivalent replacement of the present invention; and all technical solutions and improvements thereof that do not depart from the spirit and scope of the invention are included in the scope of the claims of the present invention.

Claims

1. A KV data writing and reading method based on multi-level indexing, characterized in that: Including a KV data writing process, the KV data writing process includes: Perform hash operation on the key of KV data to obtain the hash value of the key, hash(key); Select the corresponding B-family tree according to the hash value hash(key); Determine whether the value of the KV data exceeds the preset threshold; If the judgment result is no, the KV data is directly stored in the corresponding B-family tree; If the judgment result is yes, the KV data is stored in the hash index bithash file; The storing of KV data in the hash index bithash file includes: The hash index bithash file includes a data area, a hash area and a bitmap area; Write the KV data to the data area of ​​the hash index bithash file, and record the offset and size S[offset+size] in the data area; Calculate the hash value hash(key) of the key in the KV data, and update the bit value of the hashID bit corresponding to the hash value hash(key) in the bitmap in the bitmap area; The hashID, key, S[offset+size] are used as hash index records when key-value data is written, and the hash index records are written into the corresponding partition in the hash index bithash file; The hash index bithash file includes: Verification area: Use cyclic verification code to verify the data area; Conflict area: used to store keys with the same hash value and their offset and data size in the data area; And the footer area: fixed-length storage data area, check area, conflict area, hash area, offset and size of the bitmap area; Writing the hash index record into the corresponding partition in the hash index bithash file includes: Sort the hash index records by hashID, write the offset and size of all hashIDs into the hash area of ​​the file in a fixed-length manner, and record the offset and size H[offset+size] of the hash area; Serialize the bitmap recorded by the hash index, write it into the file as the bitmap area, and record the offset and size B[offset+size] of the bitmap area; The offset and size S[offset+size] of the data area, the offset and size H[offset+size] of the hash area, and the offset and size B[offset+size] of the bitmap area are stored as the footer area of ​​the file; Before sorting the hash index records by hashID, the following steps are also included: According to the hash index record, the key with the same hashID and the offset and size S[offset+size] corresponding to the key in the data area are packaged into a conflict block, and written into the conflict area of ​​the file in sequence; Save the offset and size C[offset+size] of the conflicting block of the key with the same hashID in the conflicting area, and write back C[offset+size] as the unique associated offset and size of the key with the same hashID; In the process of storing the KV data in the hash index bithash file and writing the key-value data to the data area of ​​the file, the check code of the key-value data is calculated, the check code is stored in the check area of ​​the file, and the offset and size J[offset+size] of the check code of each key in the check area are recorded; the offset and size J[offset+size] in the check area are saved to the footer area of ​​the file; When it is determined that the value of the KV data exceeds a preset threshold, the storage location of the KV data on the B-family tree is determined according to the hash value hash(key), and the hash value hash(key) and the index information corresponding to the KV data stored in the hash index bithash file are stored at the storage location.

2. The KV data writing and reading method based on multi-level indexing according to claim 1 is characterized in that: The hash index bithash file has a preset storage threshold. When the data stored in the hash index bithash file reaches the preset storage threshold, or the remaining storage space of the hash index bithash file is insufficient to store new KV data, a new hash index bithash file is created.

3. The KV data writing and reading method based on multi-level indexing according to claim 1 is characterized in that: Before performing the hash operation on the key of the KV data, the following steps are included: The write-ahead log WAL that records the KV data is written into the memtable. At this time, the data is stored in memory. When the memtable is full, a hash operation is performed on the key of the KV data.

4. The KV data writing and reading method based on multi-level indexing according to claim 1 is characterized in that: Including a KV data reading process, the KV data reading process includes: Calculate the hash value of the query key based on the query key hash(key)'; Find the corresponding B-family tree according to the hash value hash(key)' of the query key, and read the value data of the query key from the B-family tree; If the read is successful, the value data stored in the read B-family tree is returned; if the read fails, the value data of the query key is read from the hash index bithash file; If the read is successful, the value data in the read hash index bithash file is returned. If the read fails, the relevant information that the value data corresponding to the query key was not read is returned.

5. The KV data writing and reading method based on multi-level indexing according to claim 4 is characterized in that: The step of calculating the hash value hash(key)' of the query key according to the query key includes: Read from the memtable according to the query key; If the read is successful, the value data read from the memory table memtable is directly returned. If the read fails, the hash value of the query key hash(key)' is calculated based on the query key.

6. A KV data writing and reading device based on multi-level indexing, characterized in that: A KV data writing module is included, and the KV data writing module includes: The hash operation unit performs hash operation on the key of the KV data to obtain the hash value of the key, hash(key); A B-family tree storage unit selects a corresponding B-family tree according to the hash value hash(key); The processing unit determines whether the value of the KV data exceeds a preset threshold; If the judgment result is no, the KV data is directly stored in the corresponding B-family tree in the B-family tree storage unit; The bithash file storage unit stores the KV data in a hash index bithash file of the hash index bithash file storage unit if the judgment result is yes; The storing of KV data in the hash index bithash file includes: The hash index bithash file includes a data area, a hash area and a bitmap area; Write the KV data to the data area of ​​the hash index bithash file, and record the offset and size S[offset+size] in the data area; Calculate the hash value hash(key) of the key in the KV data, and update the bit value of the hashID bit corresponding to the hash value hash(key) in the bitmap in the bitmap area; The hashID, key, S[offset+size] are used as hash index records when key-value data is written, and the hash index records are written into the corresponding partition in the hash index bithash file; The hash index bithash file includes: Verification area: Use cyclic verification code to verify the data area; Conflict area: used to store keys with the same hash value and their offset and data size in the data area; And the footer area: fixed-length storage data area, check area, conflict area, hash area, offset and size of the bitmap area; Writing the hash index record into the corresponding partition in the hash index bithash file includes: Sort the hash index records by hashID, write the offset and size of all hashIDs into the hash area of ​​the file in a fixed-length manner, and record the offset and size H[offset+size] of the hash area; Serialize the bitmap recorded by the hash index, write it into the file as the bitmap area, and record the offset and size B[offset+size] of the bitmap area; The offset and size S[offset+size] of the data area, the offset and size H[offset+size] of the hash area, and the offset and size B[offset+size] of the bitmap area are stored as the footer area of ​​the file; Before sorting the hash index records by hashID, the following steps are also included: According to the hash index record, the key with the same hashID and the offset and size S[offset+size] corresponding to the key in the data area are packaged into a conflict block, and written into the conflict area of ​​the file in sequence; Save the offset and size C[offset+size] of the conflicting block of the key with the same hashID in the conflicting area, and write back C[offset+size] as the unique associated offset and size of the key with the same hashID; In the process of storing the KV data in the hash index bithash file and writing the key-value data to the data area of ​​the file, the check code of the key-value data is calculated, the check code is stored in the check area of ​​the file, and the offset and size J[offset+size] of the check code of each key in the check area are recorded; the offset and size J[offset+size] in the check area are saved to the footer area of ​​the file; When it is determined that the value of the KV data exceeds a preset threshold, the storage location of the KV data on the B-family tree is determined according to the hash value hash(key), and the hash value hash(key) and the index information corresponding to the KV data stored in the hash index bithash file are stored at the storage location.

7. A storage medium storing a computer executable program, characterized in that: When the computer executable program is executed, the KV data writing and reading method based on multi-level indexing as described in any one of claims 1 to 5 is implemented.

Images