Data storage and query method and device and electronic equipment

Abstract

The invention provides a data storage and query method and device and electronic equipment, and relates to the technical field of cloud computing, and the data storage method comprises the following steps: obtaining to-be-stored key value data; hash calculation is conducted on a key of the key value data to obtain a Hash value of the key, the Hash value comprises position information of the key value data in at least two index hierarchies in a target storage space, the at least two index hierarchies comprise the first index hierarchies and the second index hierarchies, and the second index hierarchies comprise the first index hierarchies and the second index hierarchies. The second index hierarchy is a next index hierarchy of the first index hierarchy; and storing the key value data to a target position corresponding to the second index hierarchy in the first index hierarchy according to the hash value of the key. According to the embodiment of the invention, the key value data is stored by utilizing the hash value of the key, so that when the key value data is queried, the position of the key value data in each index hierarchy can be positioned according to the hash value of the key, and the query efficiency of the key value can be improved.

Description

Technical Field

[0001] The present application relates to the field of cloud computing technology, and in particular to a data storage and query method, device and electronic device. Background Art

[0002] The key-value storage engine is a common storage structure in databases. Data is written to the memory in an append-only manner. When the memory reaches the threshold, it is sorted by key and then flushed to the disk file.

[0003] Currently, the query methods for general key values ​​are as follows Figure 1 As shown in the figure, when querying the value of a key, first read the sparse index in the file, use the sparse index to locate the data block range where the Key is located, then perform one or more disk read block operations, and finally use the binary search algorithm to find the value (Value) in the data block.

[0004] In order to speed up the query, when storing key values, complex indexes are designed for the data blocks in the data file to speed up the location of the data block where the key is located by reducing the amount of data read. However, this method of querying key values ​​is less efficient. Summary of the invention

[0005] The embodiments of the present application provide a data storage method, device and electronic device to solve the problem of low query efficiency of key values ​​when querying key values.

[0006] To solve the above technical problems, this application is implemented as follows:

[0007] In a first aspect, an embodiment of the present application provides a data storage method, the method comprising:

[0008] Get the key value data to be stored;

[0009] Performing a hash calculation on a key of the key-value data to obtain a hash value of the key, wherein the hash value includes location information of the key-value data in at least two index levels in the target storage space, wherein the at least two index levels include a first index level and a second index level, and the second index level is an index level below the first index level;

[0010] According to the hash value of the key, the key value data is stored in a target location corresponding to the second index level in the first index level.

[0011] Optionally, the first index level is a data fragment, the second index level is a page, the at least two index levels further include a third index level, the third index level is a bucket, wherein each of the data fragments includes at least two pages, and each of the pages includes at least two buckets;

[0012] The step of performing a hash calculation on the key of the key-value data to obtain a hash value of the key includes:

[0013] Performing hash calculation on the key of the key-value data to obtain data fragment location information corresponding to the key-value data in the target data fragment, page location information of the target page in the target data fragment, bucket location information of the target bucket in the target page, and key value information in the target bucket;

[0014] The storing, according to the hash value of the key, the key value data to a target location corresponding to the second index level in the first index level includes:

[0015] According to the hash value of the key, the key value data is stored in a target location, where the target location corresponds to the page location information of the target page in the data fragment, the bucket location information of the target bucket in the target page, and the key value information in the target bucket.

[0016] Optionally, before storing the key value data in a target location corresponding to the second index level in the first index level according to the hash value of the key, the method further includes:

[0017] Determine, according to the attribute characteristics of the key of the key-value data, a target fragment group to which the key-value data belongs, wherein the target fragment group is a fragment group in a first fragment group and a second fragment group into which data fragments are divided according to the attributes, and each fragment group includes at least two data fragments;

[0018] The step of performing a hash calculation on the key of the key-value data to obtain a hash value of the key includes:

[0019] In the target shard group, a hash calculation is performed on the key of the key-value data to obtain a hash value of the key.

[0020] Optionally, the method further comprises:

[0021] Constructing metadata information of the at least two index levels, the metadata information including attribute information of the at least two index levels;

[0022] The metadata information is stored in the target storage space.

[0023] Optionally, the metadata information includes at least one of the following:

[0024] The offset of the shard group in the target storage space, the offset of the data shard in the shard group, the size of the shard group, the size of the data shard, the number of pages contained in each of the data shards, the number of buckets contained in each of the pages, and a Bloom filter.

[0025] In a second aspect, an embodiment of the present application provides a data query method, the method comprising:

[0026] Obtain the key of the key-value data in the target storage space to be queried;

[0027] Performing a hash calculation on a key of the key-value data to obtain a hash value of the key, wherein the hash value includes location information of the key-value data in at least two index levels in the target storage space, wherein the at least two index levels include a first index level and a second index level, and the second index level is an index level below the first index level;

[0028] Determine, according to the hash value, a target position of the key-value data at the second index level in the first index level.

[0029] Optionally, the method further comprises:

[0030] Acquire metadata information of the at least two index levels, wherein the metadata information includes attribute information of the at least two index levels;

[0031] The determining, according to the hash value, a target position of the key-value data corresponding to the second index level in the first index level includes:

[0032] Determine a target position of the key-value data at the second index level in the first index level according to the hash value and the metadata information.

[0033] Optionally, the first index level is a data fragment, the target storage space includes at least two data fragments; the metadata information includes an offset of the data fragment in the target storage space and a size of the at least two data fragments, and the metadata information further includes a Bloom filter;

[0034] Before performing a hash calculation on the key of the key-value data to obtain the hash value of the key, the method further includes:

[0035] Determine a target data fragment among the at least two data fragments where the key-value data is located, using an offset of the data fragment in the target storage space and the Bloom filter;

[0036] The step of performing a hash calculation on the key of the key-value data to obtain the hash value of the key includes:

[0037] Performing hash calculation on the key of the key-value data to obtain data fragment location information corresponding to the key-value data in the target data fragment;

[0038] The determining, according to the hash value and the metadata information, a target position of the key-value data corresponding to the second index level in the first index level includes:

[0039] A target position of the key-value data at the second index level in the target data fragment is determined according to the data fragment position information and the size of the target data fragment.

[0040] Optionally, the second index level is a page, the at least two index levels further include a third index level, the third index level is a bucket, and each page includes at least two buckets; the metadata information further includes the number of pages included in the target data fragment and the number of buckets included in each page;

[0041] The step of performing a hash calculation on the key of the key-value data to obtain data fragment location information corresponding to the key-value data in the target data fragment includes:

[0042] Performing hash calculation on the key of the key-value data to obtain page location information of a target page in the target data fragment corresponding to the key-value data, bucket location information of a target bucket in the target page, and key value information of the key-value data in the target bucket;

[0043] The determining, according to the data fragment location information and the offset of the data fragment in the storage space, a target location of the key-value data in the second index level in the target data fragment comprises:

[0044] The target page of the key-value data in the target data fragment, the target bucket in the target page, and the target position in the target bucket are determined based on the page location information, the bucket location information and the key value information, as well as the offset of the data fragment in the storage space, the size of the target data fragment, the number of pages contained in the target data fragment and the number of buckets contained in each page.

[0045] Optionally, determining the target page of the key-value data in the target data fragment, the target bucket in the target page, and the target position in the target bucket according to the page location information, the bucket location information and the key value information, as well as the offset of the data fragment in the storage space, the size of the target data fragment, the number of pages included in the target data fragment, and the number of buckets included in each page, includes:

[0046] Performing a modulo calculation on the number of pages by the page position information in the target data fragment to obtain the target page corresponding to the offset in the target data fragment;

[0047] Performing a modulo calculation on the number of buckets by the bucket position information in the target page to obtain the target bucket corresponding to the offset in the target page;

[0048] The key-value data matching the key-value information is determined in the target bucket.

[0049] Optionally, the data fragment is located in a target data fragment group, the target data fragment group is any one of a first fragment group and a second fragment group in which the data fragments are divided according to attributes, each fragment group includes at least two data fragments, and the metadata information further includes an offset of the data fragment group in the target storage space and a size of the target data fragment group;

[0050] Before determining the target position of the key-value data at the second index level in the target data fragment according to the data fragment position information and the size of the target data fragment, the method further includes:

[0051] Determine the target shard group where the key-value data is located according to the attribute characteristics of the key of the key-value data and the offset of the data shard group in the target storage space;

[0052] The using the offset of the data fragment in the target storage space and the Bloom filter to determine the target data fragment among the at least two data fragments where the key-value data is located includes:

[0053] The target data shard where the key-value data is located is determined using the offset of the data shard in the target data shard group, the Bloom filter, and the size of the target data shard group.

[0054] In a third aspect, an embodiment of the present application provides a data storage device, the device comprising:

[0055] An acquisition module, used to acquire key-value data to be stored;

[0056] a calculation module, configured to perform a hash calculation on a key of the key-value data to obtain a hash value of the key, wherein the hash value includes location information of the key-value data in at least two index levels in a target storage space, wherein the at least two index levels include a first index level and a second index level, and the second index level is an index level below the first index level;

[0057] The first storage module is configured to store the key value data to a target location corresponding to the second index level in the first index level according to the hash value of the key.

[0058] In a fourth aspect, an embodiment of the present application provides a data query device, the device comprising:

[0059] A first acquisition module, used to acquire the key of the key-value data in the target storage space to be queried;

[0060] a calculation module, configured to perform a hash calculation on a key of the key-value data to obtain a hash value of the key, wherein the hash value includes location information of the key-value data in at least two index levels in a target storage space, wherein the at least two index levels include a first index level and a second index level, and the second index level is an index level below the first index level;

[0061] The first determination module is used to determine, according to the hash value, a target position of the key-value data corresponding to the second index level in the first index level.

[0062] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising: a processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, the steps of the data storage method described in the first aspect are implemented or the steps of the data query method described in the second aspect are implemented.

[0063] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the data storage method described in the first aspect or the steps of the data query method described in the second aspect are implemented.

[0064] In a seventh aspect, a computer program product is provided, comprising computer instructions, which, when executed by a processor, implement the steps of the data storage method as described in the first aspect or the steps of the data query method as described in the second aspect.

[0065] In the embodiment of the present application, the key-value data is stored by using the hash value of the key. Thus, when querying the key-value data, the position of the key-value data in each index level can be located according to the hash value of the key, thereby improving the query efficiency of the key value. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0067] Figure 1 This is a flowchart of a general key-value query method provided in an embodiment of the present application;

[0068] Figure 2 This is one of the flow charts of a key-value storage method provided in an embodiment of the present application;

[0069] Figure 3 This is a schematic diagram of a key-value storage structure provided by an embodiment of the present application;

[0070] Figure 4 This is one of the flow charts of a key value query method provided in an embodiment of the present application;

[0071] Figure 5 This is the second flowchart of a key-value storage method provided in an embodiment of the present application;

[0072] Figure 6 This is the second flowchart of a key value query method provided in an embodiment of the present application;

[0073] Figure 7 It is a structural diagram of a key-value storage device provided in an embodiment of the present application;

[0074] Figure 8 It is a structural diagram of a key-value query device provided in an embodiment of the present application;

[0075] Fig. 9 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0076] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0077] The core idea of ​​the key-value storage engine is to give full play to the performance advantage of disk continuous writing and exchange the short-term memory and IO overhead for the maximum writing performance. This key-value data organization method is widely used in databases, such as NoSQL-type LevelDB, RocksDB, HBase and Cassandra, and OLAP-type ClickHouse, StarRocks and Doris.

[0078] The above-mentioned key-value storage engines are designed for storing data. When facing index-type key-value data types, their query performance is not ideal. Index-type key-value data has the following characteristics: the value is fixed length, and the key is variable length. For example, in a primary key index, the key is the primary key value in the table data, and the value is the location information (fixed length) of the entire row of data identified by the primary key. The query of index-type key-value data has the characteristic of no range, and the general-purpose key-value storage engine is more concerned with the optimization of query performance in the range query scenario. Therefore, the query performance of using general-purpose key-value storage to store index-type key-value data is not ideal.

[0079] There are two main methods to accelerate the range query method of the key-value storage engine. One is to use distributed computing to accelerate the query process. The other is to filter out a lot of data that does not meet the conditions by designing complex indexes for each data block (storage unit) in the data file, reduce the amount of data read, and speed up the data block where the key is located. For example, ClickHouse uses prefix indexes to quickly locate data blocks that match the prefix, and Apache Doris uses bitmap indexes to play a similar role. StarRocks uses interval mapping indexes to quickly filter out data based on extreme values.

[0080] In addition, some indexing technologies are used to speed up searches, such as graph indexing, and hardware technologies are used to speed up queries, such as vectorized computing.

[0081] In the above methods, the key-value storage engine has the problem of complex distributed system or high time complexity of querying key-value data.

[0082] In order to speed up the analysis and query efficiency, dense indexes are gradually being used. In order to provide a global data of dense index data, this application proposes a key-value data storage structure for storing index-type data, which provides another effective solution for the method of storing index-type key-value data using a general-purpose embedded key-value storage engine in the industry. The embodiment of this application provides a data storage method, device, and electronic device to solve the problem of low query efficiency of key values ​​when querying key values.

[0083] See also Figure 2 , Figure 2 is a flow chart of a data storage method provided in an embodiment of the present application, such as Figure 2 As shown, the method comprises the following steps:

[0084] Step 201: Obtain key value data to be stored;

[0085] Step 202: Perform a hash calculation on the key of the key-value data to obtain a hash value of the key, wherein the hash value includes location information of the key-value data in at least two index levels in the target storage space, wherein the at least two index levels include a first index level and a second index level, and the second index level is an index level below the first index level;

[0086] Step 203: store the key value data to a target location corresponding to the second index level in the first index level according to the hash value of the key.

[0087] Among them, a LSM-Tree-like key-value storage structure is as follows Figure 3 As shown in the figure, three-layer storage is used. Layer0 (L0 for short) is the memory key-value data represented by the hash map (HashMap), and Layer1 (L1 for short) and Layer2 (L2 for short) are internal unordered and immutable index data files. Because there is no range query, the indexes Layer0, Layer1 and Layer2 do not need to be ordered.

[0088] When the flush threshold is reached at Layer 0, the data is persisted to Layer 1, and the old Layer 1 files are migrated to Layer 2. The background logic compacts multiple files in Layer 2 into one file.

[0089] When storing key-value data in the above storage structure, the hash value of the key of the key-value data is calculated first, and the hash value of the key determines the position of the key-value data in the immutable data file.

[0090] According to the characteristics of the key-value data, at least two index levels are set for the key-value data, wherein the at least two index levels include a first index level and a second index level, and the second index level is the next level of the first index level. In addition, a third index level, a fourth index level, etc. may also be included, which may be specifically determined according to the characteristics of the key-value data and the size of the storage space. When querying the key-value data, the query may be performed in order from the upper level to the lower level, so as to quickly locate the key-value data.

[0091] For example, in an immutable index data file, key value data is composed of multiple data fragments, each data fragment (shard) includes multiple pages, and each page (page) includes multiple buckets (buckets). The key value data can be set to two index levels of page and bucket, or the key value data can be set to three index levels of fragment, page and bucket.

[0092] After setting multiple index levels, hash calculations can be performed on the key-value data to be persisted based on the multiple index levels to obtain hash information of the key, that is, the hash value of the key, which includes the location information of the key-value data at each index level.

[0093] According to the location indicated by the hash value, the key-value data to be persisted is stored in the corresponding location in the target storage space.

[0094] For example, Figure 3 As shown, the hash value of the key includes 8 bytes, of which 4 bytes identify the shard identity (shard_id), 2 bytes identify the page identity (page_id), 1 byte identifies the bucket identity (bucket_id), and the remaining 1 byte identifies the identity of the key value in the bucket (tag).

[0095] According to the hash value of the key, the shard corresponding to the key-value data, the page in the shard, and the bucket in the page are determined, and the key-value data is stored in the bucket.

[0096] The embodiment of the present application uses the hash value of the key to organize, divide, write and store key-value data. This is different from the organization method of general key-value storage engines that sort by key. This key-value data organization method does not require key sorting, does not require the design of complex indexes and the construction of indexes, and is therefore highly efficient when key-value data is stored on disk. The data storage method of the present application has a lightweight structure and reduces the time complexity of querying key values.

[0097] Optionally, in some embodiments, the first index level is a data fragment, the second index level is a page, the at least two index levels further include a third index level, the third index level is a bucket, wherein each of the data fragments includes at least two pages, and each of the pages includes at least two buckets;

[0098] The step of performing a hash calculation on the key of the key-value data to obtain a hash value of the key includes:

[0099] Performing hash calculation on the key of the key-value data to obtain data fragment location information corresponding to the key-value data in the target data fragment, page location information of the target page in the target data fragment, bucket location information of the target bucket in the target page, and key value information in the target bucket;

[0100] The storing, according to the hash value of the key, the key value data to a target location corresponding to the second index level in the first index level includes:

[0101] According to the hash value of the key, the key value data is stored in a target location, where the target location corresponds to the page location information of the target page in the data fragment, the bucket location information of the target bucket in the target page, and the key value information in the target bucket.

[0102] For example, combined with Figure 3 As shown in the figure, the key-value data is divided into three index levels, which are data fragments, pages, and buckets from top to bottom. The size of each shard is about 1MB, and each data fragment can include multiple pages. In order to more efficiently utilize the read and write capabilities of the SSD, a page logically corresponds to a SSD read and write page. The page size is designed to be 4KB, and each page is divided into 16 buckets, which store the actual key-value data.

[0103] When calculating the hash value of the key-value data, the target data fragment (such as shard_id) corresponding to the key-value data, the page location information of the target page in the target data fragment (such as page_id), the bucket location information of the target bucket in the target page (such as bucket_id), and the tag information of the key-value data can be obtained, so that the key-value data can be stored in the target bucket.

[0104] This key-value data organization method determines the storage location based on the hash value. It does not require key sorting, designing complex indexes, or building indexes, and is highly efficient when key-value data is stored on disk.

[0105] Optionally, in some embodiments, before storing the key value data in a target location corresponding to the second index level in the first index level according to the hash value of the key, the method further includes:

[0106] Determine, according to the attribute characteristics of the key of the key-value data, a target fragment group to which the key-value data belongs, wherein the target fragment group is a fragment group in a first fragment group and a second fragment group into which data fragments are divided according to the attributes, and each fragment group includes at least two data fragments;

[0107] The step of performing a hash calculation on the key of the key-value data to obtain a hash value of the key includes:

[0108] In the target shard group, a hash calculation is performed on the key of the key-value data to obtain a hash value of the key.

[0109] According to the attribute characteristics of the key of the key-value data, the key-value data can be divided into the first shard group and the second shard group. For example, according to the fixed-length type and the variable-length type of key (Key), the shard is also divided into a fixed-length shard group and a variable-length shard group.

[0110] In addition, the key can also be divided according to other attribute characteristics.

[0111] After determining the target shard group described by the key-value data, a hash calculation is performed on the key of the key-value data according to the data shards in the target shard group.

[0112] Optionally, in some embodiments, the method further comprises:

[0113] Constructing metadata information of the at least two index levels, the metadata information including attribute information of the at least two index levels;

[0114] The metadata information is stored in the target storage space.

[0115] The attribute information of the at least two index levels includes information such as the size and quantity of each index level, and the offset of each lower level in the upper level.

[0116] Optionally, in some embodiments, the metadata information includes at least one of the following:

[0117] The offset of the shard group in the target storage space, the offset of the data shard in the shard group, the size of the shard group, the size of the data shard, the number of pages contained in each of the data shards, the number of buckets contained in each of the pages, and a Bloom filter.

[0118] For example, shard metadata information includes the offset and size of the shard group, the offset of the shard in the shard group, the shard size, the number of pages, and the number of buckets.

[0119] In addition, the metadata information may also include a Bloom filter, so that when querying key-value data, the type of shard group to which the key-value data belongs and the position in each index level may be determined based on the metadata information.

[0120] See also Figure 4 , Figure 4 This is a flow chart of a data query method provided by the present application, the data query method comprising:

[0121] Step 401, obtaining the key of the key-value data in the target storage space to be queried;

[0122] Step 402: Perform a hash calculation on the key of the key-value data to obtain a hash value of the key, wherein the hash value includes location information of the key-value data in at least two index levels in the target storage space, wherein the at least two index levels include a first index level and a second index level, and the second index level is an index level below the first index level;

[0123] Step 403: Determine, according to the Hash value, a target position of the key-value data corresponding to the second index level in the first index level.

[0124] When querying key-value data, the key-value data is hashed according to the key to obtain the hash value of the key, which is the same as the hash value calculated when the key-value data is stored.

[0125] According to the hash value, in the order of the index levels from top to bottom, the position of the key-value data in each index level can be obtained, so that the key-value data can be queried.

[0126] Optionally, in some embodiments, the method further comprises:

[0127] Acquire metadata information of the at least two index levels, wherein the metadata information includes attribute information of the at least two index levels;

[0128] The determining, according to the hash value, a target position of the key-value data corresponding to the second index level in the first index level includes:

[0129] Determine a target position of the key-value data at the second index level in the first index level according to the hash value and the metadata information.

[0130] The metadata information includes the attribute information of each index level, for example, the number of lower-level index levels contained in each index level, the offset of the lower-level index level in the upper-level index level, etc. According to the hash value and the metadata information of each index level, the position of the key-value data in each index level can be obtained, and the key-value data can be searched in the target storage space according to the metadata information.

[0131] Optionally, in some embodiments, the first index level is a data fragment, the target storage space includes at least two data fragments; the metadata information includes an offset of the data fragment in the target storage space and the size of the at least two data fragments, and the metadata information further includes a Bloom filter;

[0132] Before performing a hash calculation on the key of the key-value data to obtain the hash value of the key, the method further includes:

[0133] Determine a target data fragment among the at least two data fragments where the key-value data is located, using an offset of the data fragment in the target storage space and the Bloom filter;

[0134] The step of performing a hash calculation on the key of the key-value data to obtain the hash value of the key includes:

[0135] Performing hash calculation on the key of the key-value data to obtain data fragment location information corresponding to the key-value data in the target data fragment;

[0136] The determining, according to the hash value and the metadata information, a target position of the key-value data corresponding to the second index level in the first index level includes:

[0137] A target position of the key-value data at the second index level in the target data fragment is determined according to the data fragment position information and the size of the target data fragment.

[0138] The target storage space includes multiple data fragments, each data fragment includes offset information in the target storage space, and the Bloom filter can narrow the query range of the key-value data to the target data fragment.

[0139] The second index level can be a page or a bucket. In the target data fragment, the second index level where the key value data is located is further determined based on the hash value of the key value data. When the second index level also includes a lower-level index level, the third index level where the key value data is located is determined in the second index level, as well as the position of the key value data in the third index level.

[0140] The metadata information includes the offset of the data fragment in the target storage space, the size of at least two data fragments, the number of pages contained in each data fragment, the number of buckets contained in each page, and the Bloom filter.

[0141] The location information included in the hash value of the key of the key-value data may include the location at each index level, for example, the shard_id of the data shard, the page_id of the page in the data shard, and the bucket_id of the bucket in the page.

[0142] When querying key-value data, a Bloom filter is first used to determine the target data fragment where the key-value data is located. According to the location information of the key-value data in each index level indicated by the hash value of the key of the key-value data, the location of the key-value data is determined in the target storage space according to the metadata information.

[0143] For example, according to shard_id and page_id, as well as the size of the target data shard and the number of pages contained in the target data shard, the target page where the key value data is located is determined in the target data shard, and according to bucket_id and the number of buckets contained in the page, the target bucket where the key value data is located is determined.

[0144] The key-value data query method can quickly locate the specific offset of the key in the page or bucket according to the hash value of the key, thereby improving the query efficiency of the key.

[0145] Optionally, in some embodiments, determining the target page of the key-value data in the target data fragment, the target bucket in the target page, and the target position in the target bucket according to the page location information, the bucket location information and the key value information, as well as the offset of the data fragment in the storage space, the size of the target data fragment, the number of pages included in the target data fragment and the number of buckets included in each page, comprises:

[0146] Performing a modulo calculation on the number of pages by the page position information in the target data fragment to obtain the target page corresponding to the offset in the target data fragment;

[0147] Performing a modulo calculation on the number of buckets by the bucket position information in the target page to obtain the target bucket corresponding to the offset in the target page;

[0148] The key-value data matching the key-value information is determined in the target bucket.

[0149] For example, the number of pages in the target data fragment is modulo calculated based on page_id to obtain the page offset in the shard and locate the target page in the shard. According to the number of buckets in the target page, the number of buckets is modulo the bucket_id in the key hash value to obtain the bucket offset stored in the 64-byte page header, which can quickly locate the bucket where the key is located.

[0150] The tag information of each bucket stores the key value information of the last byte in the stored key hash value. By comparing the key value information in the key hash value with each tag information in the bucket in turn, the key value data can be quickly located.

[0151] The key-value data query method can quickly locate the specific offset of the key in the page or bucket according to the hash value of the key, thereby improving the query efficiency of the key.

[0152] Optionally, in some embodiments, the data fragment is located in a target data fragment group, the target data fragment group is any one of a first fragment group and a second fragment group in which the data fragments are divided according to attributes, each fragment group includes at least two data fragments, and the metadata information further includes an offset of the data fragment group in the target storage space and a size of the target data fragment group;

[0153] Before determining the target position of the key-value data at the second index level in the target data fragment according to the data fragment position information and the size of the target data fragment, the method further includes:

[0154] Determine the target shard group where the key-value data is located according to the attribute characteristics of the key of the key-value data and the offset of the data shard group in the target storage space;

[0155] The using the offset of the data fragment in the target storage space and the Bloom filter to determine the target data fragment among the at least two data fragments where the key-value data is located includes:

[0156] The target data shard where the key-value data is located is determined using the offset of the data shard in the target data shard group, the Bloom filter, and the size of the target data shard group.

[0157] According to the attribute characteristics of the key of the key-value data, the key-value data can be divided into a first shard group and a second shard group, each shard group includes at least two data shards. For example, according to the fixed-length type and the variable-length type of key (Key), the shard is also divided into a fixed-length shard (shard) group and a variable-length shard group.

[0158] When querying, the shard group where the key-value data is located can be determined based on the type and length of the key of the key-value data to be queried, as well as the offset and size of the shard group in the target storage space. A shard group includes multiple shards, and the Bloom filter and the offset of the shard in the shard group are used to further determine the target shard where the key-value data is located.

[0159] Through the above method, the position of the key value data at each index level is determined in turn.

[0160] In order to facilitate understanding of the above embodiments, the storage and query process of key-value data is described below through a specific embodiment.

[0161] 1. Key-value data storage

[0162] Combination Figure 3 and Figure 5 As shown, the key-value data writing process includes the following steps:

[0163] Step 501: Divide the key values ​​in the memory into corresponding shard groups according to the key length. String-type keys are divided into variable-length shard groups, and other fixed-length keys are divided into corresponding fixed-length shard groups.

[0164] Step 502: Perform a hash calculation on the key value data to be persisted according to the key to obtain the hash value of the key. The hash value of the key determines the position of the key value data in the immutable data file. The hash value of the key includes 8 bytes, of which 4 bytes identify the shard identity, 2 bytes identify the page identity, 1 byte identifies the bucket identity, and the remaining 1 byte identifies the identity of the key value in the bucket.

[0165] Step 503: Store the key value data in the bucket according to the hash information.

[0166] Step 504: construct shard metadata information and Bloom filter. The shard metadata information includes the offset and size of the shard group, the offset of the shard in the shard group, the shard size, the number of pages, and the number of buckets. The Bloom filter is used to read the metadata information when querying key-value data, and determine whether the key is in the shard based on the metadata information.

[0167] Step 505: Serialize the data and save it to disk.

[0168] The above information storage method divides the key hash value information into shard identity information, page identity information, bucket identity information and tag identity information. Correspondingly, the LSM-Tree-like data organization method is also divided into fixed-length shard groups and variable-length shard groups, shards, pages, and buckets. This key-value data organization method that organizes key-value data based on the key hash value avoids sorting operations and has a relatively high data storage efficiency.

[0169] 2. Key-value data query process

[0170] Based on the above key-value storage structure, query the key-value data, such as Figure 6 As shown, the key-value data query method includes the following steps:

[0171] Step 601: Narrow the query scope to the shard group according to the type and length of the key to be queried and the shard metadata (the offset and size of the shard group).

[0172] Step 602: According to the shard metadata information (the offset of the shard in the shard group), query the Bloom filter to further narrow the query scope to one shard.

[0173] Step 603: Calculate the hash value of the key.

[0174] Step 604: According to the number of pages in the shard meta information, the page identity information (page_id) in the hash value of the key is used to calculate the modulus of the number of pages to obtain the offset of the page in the shard and locate the page in the shard. At this point, a single disk input / output (IO) can read the data containing the key value being searched.

[0175] Step 605: Further, according to the number of buckets in the shard meta information, the bucket identity information (bucket_id) in the hash value of the key is used to calculate the modulus of the number of buckets to obtain the bucket offset stored in the 64 bytes of the page header, which can quickly locate the bucket position in the page where the key is located.

[0176] Step 606: The tag information of each bucket stores the key value identity of the last byte in the stored key hash value. The tag information occupies 16 bytes. The tag in the key hash value is compared with the tag information in sequence to quickly locate the key value data.

[0177] The page where the key is located in the disk file is calculated based on the key's hash value, and the block containing the key value data can be read in one IO. The key value data is found through the bucket identity information and tag identity information in the key's hash value.

[0178] Compared with the general key-value storage engine's method of locating the physical block where the key is located through complex indexes, this method of locating the physical block where the key is located according to the key's hash value is more concise and efficient. Compared with the general key-value storage engine's binary search data retrieval process in the physical block (the algorithm time complexity is O(log2N)), the key's hash value (bucket identity information and tag identity information) is used to quickly locate the key value position, with a time complexity of 0(1), which improves the query speed of the key value.

[0179] See also Figure 7 , Figure 7 is a schematic diagram of a data storage device provided in an embodiment of the present application. Figure 7 As shown, the data storage device 700 includes:

[0180] The acquisition module 701 is used to acquire the key value data to be stored;

[0181] A calculation module 702 is used to perform a hash calculation on the key of the key-value data to obtain a hash value of the key, wherein the hash value includes location information of the key-value data in at least two index levels in the target storage space, wherein the at least two index levels include a first index level and a second index level, and the second index level is an index level below the first index level;

[0182] The first storage module 703 is configured to store the key value data to a target location corresponding to the second index level in the first index level according to the hash value of the key.

[0183] Optionally, the first index level is a data fragment, the second index level is a page, the at least two index levels further include a third index level, the third index level is a bucket, wherein each of the data fragments includes at least two pages, and each of the pages includes at least two buckets;

[0184] The calculation module 702 is specifically used for:

[0185] Performing hash calculation on the key of the key-value data to obtain data fragment location information corresponding to the key-value data in the target data fragment, page location information of the target page in the target data fragment, bucket location information of the target bucket in the target page, and key value information in the target bucket;

[0186] The first storage module 703 is specifically used for:

[0187] According to the hash value of the key, the key value data is stored in a target location, where the target location corresponds to the page location information of the target page in the data fragment, the bucket location information of the target bucket in the target page, and the key value information in the target bucket.

[0188] Optionally, the device further comprises:

[0189] A determination module, configured to determine a target fragment group to which the key-value data belongs according to the attribute characteristics of the key of the key-value data, wherein the target fragment group is a fragment group in a first fragment group and a second fragment group into which the data fragments are divided according to the attributes, and each of the fragment groups includes at least two data fragments;

[0190] The calculation module 702 is specifically used for:

[0191] In the target shard group, a hash calculation is performed on the key of the key-value data to obtain a hash value of the key.

[0192] Optionally, the device further comprises:

[0193] A construction module, configured to construct metadata information of the at least two index levels, wherein the metadata information includes attribute information of the at least two index levels;

[0194] The second storage module is used to store the metadata information in the target storage space.

[0195] Optionally, the metadata information includes at least one of the following:

[0196] The offset of the shard group in the target storage space, the offset of the data shard in the shard group, the size of the shard group, the size of the data shard, the number of pages contained in each of the data shards, the number of buckets contained in each of the pages, and a Bloom filter.

[0197] The data storage device enables Figure 2 The various processes implemented in the method embodiment can achieve the same technical effect, and to avoid repetition, they will not be described here.

[0198] See also Figure 8 , Figure 8 is a structural diagram of a data query device provided in an embodiment of the present application, such as Figure 8 As shown, the data query device 800 includes:

[0199] The first acquisition module 801 is used to acquire the key of the key-value data in the target storage space to be queried;

[0200] A calculation module 802 is configured to perform a hash calculation on a key of the key-value data to obtain a hash value of the key, wherein the hash value includes location information of the key-value data in at least two index levels in a target storage space, wherein the at least two index levels include a first index level and a second index level, and the second index level is an index level below the first index level;

[0201] The first determination module 803 is used to determine, according to the Hash value, a target position of the key-value data corresponding to the second index level in the first index level.

[0202] Optionally, the device further comprises:

[0203] A second acquisition module, configured to acquire metadata information of the at least two index levels, wherein the metadata information includes attribute information of the at least two index levels;

[0204] The first determining module 803 is specifically used for:

[0205] Determine a target position of the key-value data at the second index level in the first index level according to the hash value and the metadata information.

[0206] Optionally, the first index level is a data fragment, the target storage space includes at least two data fragments; the metadata information includes an offset of the data fragment in the target storage space and a size of the at least two data fragments, and the metadata information further includes a Bloom filter;

[0207] The device also includes:

[0208] A second determination module is configured to determine a target data fragment among the at least two data fragments where the key-value data is located, using an offset of the data fragment in the target storage space and the Bloom filter;

[0209] The calculation module 802 is specifically used for:

[0210] Performing hash calculation on the key of the key-value data to obtain data fragment location information corresponding to the key-value data in the target data fragment;

[0211] The first determining module 804 is specifically used for:

[0212] A target position of the key-value data at the second index level in the target data fragment is determined according to the data fragment position information and the size of the target data fragment.

[0213] Optionally, the second index level is a page, the at least two index levels further include a third index level, the third index level is a bucket, and each page includes at least two buckets; the metadata information further includes the number of pages included in the target data fragment and the number of buckets included in each page;

[0214] The calculation module 802 is specifically used for:

[0215] Performing hash calculation on the key of the key-value data to obtain page location information of a target page in the target data fragment corresponding to the key-value data, bucket location information of a target bucket in the target page, and key value information of the key-value data in the target bucket;

[0216] The first determining module 804 is specifically used for:

[0217] The target page of the key-value data in the target data fragment, the target bucket in the target page, and the target position in the target bucket are determined based on the page location information, the bucket location information and the key value information, as well as the offset of the data fragment in the storage space, the size of the target data fragment, the number of pages contained in the target data fragment and the number of buckets contained in each page.

[0218] Optionally, the first determining module 804 includes:

[0219] A first calculation submodule is used to perform a modulo calculation on the page position information in the target data fragment and the number of pages to obtain the target page corresponding to the offset in the target data fragment;

[0220] A second calculation submodule is used to perform a modulo calculation on the bucket position information in the target page and the number of buckets to obtain the target bucket corresponding to the offset in the target page;

[0221] A determination submodule is used to determine the key-value data matching the key-value information in the target bucket.

[0222] Optionally, the data fragment is located in a target data fragment group, the target data fragment group is any one of a first fragment group and a second fragment group in which the data fragments are divided according to attributes, each fragment group includes at least two data fragments, and the metadata information further includes an offset of the data fragment group in the target storage space and a size of the target data fragment group;

[0223] The device also includes:

[0224] A third determination module is used to determine the target fragment group where the key-value data is located according to the attribute characteristics of the key of the key-value data and the offset of the data fragment group in the target storage space;

[0225] The first determining module is specifically used for:

[0226] The target data shard where the key-value data is located is determined using the offset of the data shard in the target data shard group, the Bloom filter, and the size of the target data shard group.

[0227] The data query device can realize Figure 4 The various processes implemented in the method embodiment can achieve the same technical effect, and to avoid repetition, they will not be described here.

[0228] The data storage device and data query device in the embodiments of the present application may be an electronic device or a component in the electronic device, such as an integrated circuit or a chip. The electronic device may be a terminal or other device other than a terminal. Exemplarily, the electronic device may be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, etc., and may also be a server, a network attached storage (NAS), a personal computer (PC), etc., which is not specifically limited in the embodiments of the present application.

[0229] like Fig. 9 As shown, an embodiment of the present application also provides an electronic device 900, including: a processor 901, a memory 902, and a program stored in the memory 902 and executable on the processor 901. When the program is executed by the processor 901, each process of the above-mentioned data storage method and data query method embodiments is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0230] The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, each process of the above-mentioned data storage method and data query method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. Among them, the computer-readable storage medium, such as read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0231] The present application also provides a computer program product, including computer instructions, which, when executed by a processor, implement the above Figure 2 and Figure 4 The various processes of the method embodiment shown can achieve the same technical effect, and will not be described again here to avoid repetition.

[0232] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0233] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0234] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A data storage method, characterized in that: include: Get the key value data to be stored; Performing a hash calculation on a key of the key-value data to obtain a hash value of the key, wherein the hash value includes location information of the key-value data in at least two index levels in the target storage space, wherein the at least two index levels include a first index level and a second index level, and the second index level is an index level below the first index level; According to the hash value of the key, the key value data is stored in a target location corresponding to the second index level in the first index level.

2. The method according to claim 1, characterized in that The first index level is a data fragment, the second index level is a page, the at least two index levels further include a third index level, the third index level is a bucket, wherein each of the data fragments includes at least two pages, and each of the pages includes at least two buckets; The step of performing a hash calculation on the key of the key-value data to obtain a hash value of the key includes: Performing hash calculation on the key of the key-value data to obtain data fragment location information corresponding to the key-value data in the target data fragment, page location information of the target page in the target data fragment, bucket location information of the target bucket in the target page, and key value information in the target bucket; The storing, according to the hash value of the key, the key value data to a target location corresponding to the second index level in the first index level includes: According to the hash value of the key, the key value data is stored in a target location, where the target location corresponds to the page location information of the target page in the data fragment, the bucket location information of the target bucket in the target page, and the key value information in the target bucket.

3. The method according to claim 2, characterized in that Before storing the key value data in a target location corresponding to the second index level in the first index level according to the hash value of the key, the method further includes: Determine, according to the attribute characteristics of the key of the key-value data, a target fragment group to which the key-value data belongs, wherein the target fragment group is a fragment group in a first fragment group and a second fragment group into which data fragments are divided according to the attributes, and each fragment group includes at least two data fragments; The step of performing a hash calculation on the key of the key-value data to obtain a hash value of the key includes: In the target shard group, a hash calculation is performed on the key of the key-value data to obtain a hash value of the key.

4. The method according to any one of claims 1 to 3, characterized in that The method further comprises: Constructing metadata information of the at least two index levels, the metadata information including attribute information of the at least two index levels; The metadata information is stored in the target storage space.

5. The method according to claim 4, characterized in that The metadata information includes at least one of the following: The offset of the shard group in the target storage space, the offset of the data shard in the shard group, the size of the shard group, the size of the data shard, the number of pages contained in each of the data shards, the number of buckets contained in each of the pages, and the Bloom filter.

6. A data query method, characterized in that: The method comprises: Obtain the key of the key-value data in the target storage space to be queried; Performing a hash calculation on a key of the key-value data to obtain a hash value of the key, wherein the hash value includes location information of the key-value data in at least two index levels in the target storage space, wherein the at least two index levels include a first index level and a second index level, and the second index level is an index level below the first index level; Determine, according to the hash value, a target position of the key-value data at the second index level in the first index level.

7. The method according to claim 6, characterized in that The method further comprises: Acquire metadata information of the at least two index levels, wherein the metadata information includes attribute information of the at least two index levels; The determining, according to the hash value, a target position of the key-value data corresponding to the second index level in the first index level includes: Determine a target position of the key-value data at the second index level in the first index level according to the hash value and the metadata information.

8. The method according to claim 7, characterized in that The first index level is a data fragment, the target storage space includes at least two data fragments; the metadata information includes an offset of the data fragment in the target storage space and the size of the at least two data fragments, and the metadata information also includes a Bloom filter; Before performing a hash calculation on the key of the key-value data to obtain the hash value of the key, the method further includes: Determine a target data fragment among the at least two data fragments where the key-value data is located, using an offset of the data fragment in the target storage space and the Bloom filter; The step of performing a hash calculation on the key of the key-value data to obtain the hash value of the key includes: Performing hash calculation on the key of the key-value data to obtain data fragment location information corresponding to the key-value data in the target data fragment; The determining, according to the hash value and the metadata information, a target position of the key-value data corresponding to the second index level in the first index level includes: A target position of the key-value data at the second index level in the target data fragment is determined according to the data fragment position information and the size of the target data fragment.

9. The method according to claim 8, characterized in that The second index level is a page, the at least two index levels further include a third index level, the third index level is a bucket, and each page includes at least two buckets; the metadata information further includes the number of pages included in the target data fragment and the number of buckets included in each page; The step of performing a hash calculation on the key of the key-value data to obtain data fragment location information corresponding to the key-value data in the target data fragment includes: Performing hash calculation on the key of the key-value data to obtain page location information of a target page in the target data fragment corresponding to the key-value data, bucket location information of a target bucket in the target page, and key value information of the key-value data in the target bucket; The determining, according to the data fragment location information and the offset of the data fragment in the storage space, a target location of the key-value data in the second index level in the target data fragment comprises: The target page of the key-value data in the target data fragment, the target bucket in the target page, and the target position in the target bucket are determined based on the page location information, the bucket location information and the key value information, as well as the offset of the data fragment in the storage space, the size of the target data fragment, the number of pages contained in the target data fragment and the number of buckets contained in each page.

10. The method according to claim 9, characterized in that The determining, according to the page location information, the bucket location information and the key value information, the offset of the data fragment in the storage space, the size of the target data fragment, the number of pages included in the target data fragment and the number of buckets included in each page, the target page of the key value data in the target data fragment, the target bucket in the target page, and the target position in the target bucket comprises: Performing a modulo calculation on the number of pages by the page position information in the target data fragment to obtain the target page corresponding to the offset in the target data fragment; Performing a modulo calculation on the number of buckets by the bucket position information in the target page to obtain the target bucket corresponding to the offset in the target page; The key-value data matching the key-value information is determined in the target bucket.

11. The method according to any one of claims 8 to 10, characterized in that The data fragments are located in a target data fragment group, the target data fragment group is any one of a first fragment group and a second fragment group in which the data fragments are divided according to attributes, each fragment group includes at least two data fragments, and the metadata information further includes an offset of the data fragment group in the target storage space and a size of the target data fragment group; Before determining the target position of the key-value data at the second index level in the target data fragment according to the data fragment position information and the size of the target data fragment, the method further includes: Determine the target shard group where the key-value data is located according to the attribute characteristics of the key of the key-value data and the offset of the data shard group in the target storage space; The using the offset of the data fragment in the target storage space and the Bloom filter to determine the target data fragment among the at least two data fragments where the key-value data is located includes: The target data shard where the key-value data is located is determined using the offset of the data shard in the target data shard group, the Bloom filter, and the size of the target data shard group.

12. A data storage device, characterized in that: include: An acquisition module, used to acquire key-value data to be stored; a calculation module, configured to perform a hash calculation on a key of the key-value data to obtain a hash value of the key, wherein the hash value includes location information of the key-value data in at least two index levels in a target storage space, wherein the at least two index levels include a first index level and a second index level, and the second index level is an index level below the first index level; The first storage module is configured to store the key value data to a target location corresponding to the second index level in the first index level according to the hash value of the key.

13. A data query device, characterized in that: include: A first acquisition module, used to acquire the key of the key-value data in the target storage space to be queried; a calculation module, configured to perform a hash calculation on a key of the key-value data to obtain a hash value of the key, wherein the hash value includes location information of the key-value data in at least two index levels in a target storage space, wherein the at least two index levels include a first index level and a second index level, and the second index level is an index level below the first index level; The first determination module is used to determine, according to the hash value, a target position of the key-value data corresponding to the second index level in the first index level.

14. An electronic device, characterized in that: include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the data storage method as described in any one of claims 1 to 5, or implements the steps of the data query method as described in any one of claims 6 to 11.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the data storage method according to any one of claims 1 to 5, or implements the steps of the data query method according to any one of claims 6 to 11.

16. A computer program product, characterized in that The method comprises computer instructions, which, when executed by a processor, implement the steps of the data storage method according to any one of claims 1 to 5, or implement the steps of the data query method according to any one of claims 6 to 11.

Images