Method for performing data search in disk, and disk data index generation method

Abstract

Provided in the present disclosure are a method for performing a data search in a disk, and a disk data index generation method. The method for performing a data search in a disk comprises: receiving a target query key for target data that needs to be searched for; on the basis of the target query key, querying a multi-level data index for the memory location of the target query key level by level, wherein the multi-level data index is generated during data storage and is obtained by means of compression, and comprises multiple levels of query key indexes, and each query key index is generated on the basis of a binary trie and is used for mapping a next level of query key index or the memory location; and acquiring stored key-value pairs on the basis of the memory location, and searching for a target query value corresponding to the target query key, wherein the target query value is used for indicating the target data.

Description

[0001] Methods for data retrieval on disks and methods for generating disk data indexes are both technical fields.

[0002]

[0001] This disclosure relates to the field of data storage technology, and more particularly to a method for data retrieval on a disk and a method for generating a disk data index. Background Art

[0003]

[0002] A disk stores data in fixed-size data blocks, which are the smallest unit of reading and writing on the disk. A data block can be viewed as an array, with each data block having a unique ID. The specific data is stored within each data block. Data is arranged in sequential order according to its key (query key), without repetition. A core issue in disk retrieval is how to determine the data block ID given a query key.

[0004]

[0003] In related technologies, a direct search method is usually used for searching. This method can result in a particularly long search time when the amount of data is large. To address this, related technologies utilize mathematical methods to improve search efficiency, such as binary search, which can improve search efficiency to some extent. However, in big data scenarios, this method still has significant limitations and often fails to find the required data in a timely and effective manner.

[0005]

[0004] Therefore, the disk lookup method in the related art suffers from low lookup efficiency. Summary of the Invention

[0006]

[0005] This disclosure provides a method for data retrieval on a disk and a method for generating a disk data index to solve one or more of the above-mentioned technical problems.

[0007]

[0006] In a first aspect, embodiments of this disclosure provide a method for searching data on a disk, comprising: receiving a target query key for target data to be searched; querying the memory location of the target query key level by level in a multi-level data index according to the target query key, wherein the multi-level data index is generated and compressed during data storage, including multiple levels of query key indexes, the query key index being generated based on a binary prefix tree and used to map the next level query key index or the memory location; obtaining stored key-value pairs based on the memory location, and searching for a target query value corresponding to the target query key, wherein the memory location stores multiple key-value pairs, each key-value pair including a query key and a query value corresponding to the query key, the query value being used to indicate corresponding data, and the target query value being used to indicate the target data.

[0008]

[0007] In a second aspect, embodiments of this disclosure provide a disk data index generation method, comprising: receiving target data to be stored; generating key-value pairs of the target data, wherein the key-value pairs include a target query key and a target query value of the target data, the target query key being used to find the corresponding target query value, and the target query value being used to indicate the target data; writing the key-value pairs into a current unit memory location; after the unit memory location is full, extracting the identifier of the unit memory location; generating a lowest-level query key index of a multi-level data index based on the identifier; as the stored data increases, generating and compressing the multi-level data index level by level based on the lowest-level query key index, wherein the multi-level data index includes multiple levels of query key indexes for finding the unit memory location to which the target query key belongs, the query key index being generated based on a binary prefix tree and used to map the next-level query key index or the memory location.

[0009]

[0008] In a third aspect, embodiments of the present disclosure provide an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor implements the method of any one of the embodiments of the present disclosure when executing the computer program.

[0010]

[0009] In a fourth aspect, embodiments of the present disclosure provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method described in any one of the embodiments of the present disclosure.

[0011]

[0010] In a fifth aspect, embodiments of this disclosure provide a computer program product, including a computer program, wherein the computer program, when executed by a processor, implements a method according to any one of the embodiments of this disclosure.

[0011] The multi-level data index according to embodiments of this disclosure can efficiently compress the query keys of stored data, reduce the resource consumption of the data index, and establish a mapping relationship between the memory location of stored data and the query key. Thus, the multi-level data index can quickly find the key-value pair corresponding to the query key of the target data to be searched, that is, the memory location of the target data on the disk. During querying, the multi-level data index can be compressed and directly searched without decompressing the entire index, which not only improves query efficiency but also reduces the resource consumption during querying. Moreover, the lowest-level query key index of the multi-level data index directly maps to the memory location. During searching, only one read / write operation is needed to obtain the memory page corresponding to the memory location of the target data to complete the search for the target data, greatly reducing the consumption of read / write resources. This solves the problem of low search efficiency in disk search methods in related technologies.

[0012]

[0012] The above description is merely an overview of the technical solution of this disclosure. In order to better understand the technical means of this disclosure, it can be implemented according to the contents of the specification. Furthermore, in order to make the above and other objects, features, and advantages of this disclosure more apparent and understandable, specific embodiments of this disclosure are described below. (See attached drawings.)

[0013]

[0013] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the various drawings denote the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings depict only some embodiments according to this disclosure and should not be considered as limiting the scope of this disclosure.

[0014]

[0014] Figure 1 shows a schematic diagram of the binary search method of the relevant technology;

[0015]

[0015] Figure 2 shows a flowchart of a method for searching data on a disk provided in an embodiment of the present disclosure;

[0016]

[0016] Figure 3 shows a flowchart of a disk data index generation method provided in an embodiment of the present disclosure;

[0017]

[0017] Figure 4 shows a schematic diagram of a binary index trie according to an embodiment of the present disclosure;

[0018]

[0018] Figure 5 shows a schematic diagram of the query key of an example embodiment of the present disclosure;

[0019]

[0019] Figure 6 shows a schematic diagram of the trie index compression and query principle of an example of an embodiment of the present disclosure;

[0020]

[0020] Figure 7 shows a schematic diagram of trie index jump according to an embodiment of the present disclosure;

[0021]

[0021] Figure 8 shows a schematic diagram of the impact of the direct lookup jump table size on query performance in an embodiment of this disclosure;

[0022]

[0022] Figure 9 shows a schematic diagram of the overall architecture of the multi-level data index level-trie according to an embodiment of the present disclosure;

[0023]

[0023] Figure 10 shows a schematic diagram of the memory usage analysis of a multi-level data index level-trie according to an embodiment of the present disclosure;

[0024]

[0024] FIG11 shows a structural block diagram of an apparatus for searching data on a disk according to an embodiment of the present disclosure;

[0025]

[0025] Figure 12 shows a structural block diagram of a disk data indexing generation apparatus provided in an embodiment of this disclosure; and

[0026]

[0026] FIG13 shows a block diagram of an electronic device used to implement embodiments of the present disclosure. Detailed Description

[0027]

[0027] In the following description, only certain exemplary embodiments are briefly described. As will be appreciated by those skilled in the art, the described embodiments can be modified in various ways without departing from the spirit or scope of this disclosure. Therefore, the drawings and description are to be considered exemplary in nature and not restrictive.

[0028]

[0028] To facilitate understanding of the technical solutions of the embodiments of this disclosure, the related technologies of the embodiments of this disclosure are described below. The following related technologies are optional solutions and can be combined with the technical solutions of the embodiments of this disclosure in any way, and all of them fall within the protection scope of the embodiments of this disclosure.

[0029]

[0029] In related technologies, two metrics are typically used to evaluate the performance of disk retrieval algorithms: 1. The speed at which the data block ID is calculated given a query key; 2. Memory resource consumption. The goal of a retrieval algorithm is to maximize retrieval speed while minimizing resource consumption during the process.

[0030] The simplest approach is to use binary search, as shown in Figure 1. Figure 1 illustrates a schematic diagram of the binary search method in this technique. Disk data blocks are identified as memory pages. Disk data is ordered; for a given query key, binary search can eventually determine the final data block by reading the corresponding data blocks multiple times. However, this method involves multiple disk input / output I / O operations during a single query key retrieval, thus consuming significant resources and resulting in poor search performance.

[0030]

[0031] Another approach is to extract a key representing the query from each data block. For example, within each memory page, the minimum or maximum value can be selected to build a multi-level data index, forming a multi-way prefix tree. Data retrieval is then performed layer by layer down the multi-way prefix tree. This method requires at most the number of I / O operations equal to the number of index levels per retrieval. Since index data blocks are usually smaller than the original data, they can be cached to further reduce the I / O required for retrieval.

[0031]

[0032] However, this approach still has two drawbacks: 1. The index data blocks are built up by extracting the minimum / maximum value of each data block layer by layer. When the query key is large, the index data blocks will occupy a lot of space. 2. When the data volume is large, the index data blocks may occupy a lot of storage space and are difficult to cache completely, so a single retrieval still requires multiple I / O operations.

[0032]

[0033] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0033]

[0034] The technical solutions of this disclosure and how they solve the aforementioned technical problems are described in detail below with specific embodiments. The listed specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of this disclosure will be described in detail below with reference to the accompanying drawings.

[0034]

[0035] Figure 2 shows a flowchart of a method for searching data on a disk according to an embodiment of the present disclosure. The present disclosure provides a data searching method, as shown in Figure 2, which is a flowchart of a method for searching data on a disk according to an embodiment of the present disclosure. The method may include the following steps.

[0035]

[0036] Step S201: Receive the target query key for the target data to be searched.

[0036]

[0037] Step S202: Based on the target query key, query the memory location of the target query key level by level in the multi-level data index. The multi-level data index is generated and compressed during data storage and includes multiple levels of query key indexes. The query key index is generated based on a binary prefix tree and is used to map the next level query key index or memory location.

[0037]

[0038] Step S203: Obtain the stored key-value pairs based on the memory location, and find the target query value corresponding to the target query key. The memory location stores multiple key-value pairs. Each key-value pair includes a query key and a query value corresponding to the query key. The query value is used to indicate the corresponding data, and the target query value is used to indicate the target data.

[0038]

[0039] According to the data lookup method described in this embodiment, after receiving the target query key of the target data to be searched, the memory location corresponding to the target query key is queried level by level in the multi-level data index using a compressed direct lookup method. The multi-level data index can efficiently compress the query keys of stored data, reducing the resource consumption of the data index, and establish a mapping relationship between the memory location of stored data and the query key. Therefore, the key-value pair corresponding to the query key of the target data to be searched can be quickly found through this multi-level data index, which is the memory location of the target data on the disk.

[0039]

[0040] During queries, multi-level data indexes can be compressed and directly accessed without decompressing the entire index. This not only improves query efficiency but also reduces resource consumption. Furthermore, the lowest-level query key index of the multi-level data index directly maps to memory locations. During a search, only a single read / write operation is needed to retrieve the memory page corresponding to the target data, thus significantly reducing the consumption of read / write resources.

[0040]

[0041] Then, the key-value pair of the memory location is obtained, and the target query value corresponding to the target query key is found. The target query value can indicate the target data. In this embodiment, the target query value can be the target data. This solves the problem of low search efficiency in disk search methods in related technologies.

[0041]

[0042] The target data mentioned above is data stored on the disk. Data on the disk is stored in units of data blocks (slots). Data blocks can be memory pages, or other storage structures. A data block can store multiple sets of data, and each set of data is stored in the form of key-value pairs.

[0042]

[0043] When searching for data, you only need to use the target query key to find the corresponding key-value pair, determine the target query value of the target query key, and then determine the target data based on the target query value.

[0043]

[0044] When searching for target data, the system first receives the target query key, which can be provided by the party issuing the search request. After receiving the target query key, the system uses a compressed direct lookup method to search for the memory location of the target query key, which is the storage location of the corresponding key-value pair, and also the storage location of the target query value, within the compressed multi-level data index.

[0044]

[0045] The aforementioned compressed direct lookup is achieved because multi-level data indexes use compression methods that support compressed direct lookups, such as integer compression. Therefore, when querying through the multi-level data index, the direct lookup jump tables and prefix trees of each level's query key index can be used for fast queries, significantly improving query efficiency when dealing with large amounts of data. Further details will be explained later.

[0045]

[0046] The aforementioned multi-level data index is generated and compressed during data storage. It includes multiple levels of query key indexes for finding the memory location of the target query key. The lowest level query key index is used to map the memory location. The memory location stores multiple key-value pairs. Each key-value pair includes the query key and the query value corresponding to the query key.

[0046]

[0047] Multi-level indexes establish a mapping between memory locations and data keys. Each level in a multi-level index includes at least one query key index, and each higher-level query key index includes multiple lower-level query key indexes. Therefore, the higher the level, the fewer query key indexes are needed. This allows for faster, level-by-level searching when comparing with the target query key, gradually narrowing down the range of the target query key and ultimately determining the memory location range corresponding to the target query key.

[0047]

[0048] Data is stored in memory in units of memory pages. Since a single memory page contains relatively little data, it can be retrieved in a single read / write operation. After retrieving the memory page, a search is performed based on the target query key to find the corresponding key-value pair and the target query value.

[0048]

[0049] That is, after determining the memory location in the multi-level data index, the key-value pair of the memory location is obtained, and the target query value corresponding to the target query key is found; the target data is determined based on the target query value, where the target query value is used to indicate the target data.

[0049]

[0050] It should be noted that the memory location of the target query key determined by a multi-level data index may be a specific memory page or a range of two adjacent memory pages. This is determined by the query method of the lowest-level query key index in the multi-level data index.

[0050]

[0051] The lowest-level query key index uses a prefix tree for lookup. The final result may be a single memory page or a range of several memory pages, typically a range of two memory pages. The specific principles will be explained in detail later.

[0051]

[0052] It's important to note that the key-value pairs described above cannot be stored across disk blocks. When providing a mapping from query keys to data blocks, if the disk block size is 4KB, the size of a single key-value pair cannot exceed 4KB. In other words, key-value pairs cannot be stored across multiple data blocks. This limitation can be mitigated by splitting large key-value pairs into data and index portions, with the data blocks storing only the index portion and the data portion being queried via a secondary index. The same applies to memory pages; key-value pairs cannot be stored across memory pages.

[0053] The lowest-level query key index in the aforementioned multi-level data index maps to memory pages, and theoretically, it can also map to data blocks or other memory structures. When providing a mapping from query keys to data blocks, directly establishing a mapping from query keys to key-value pairs can result in a very large index size if the number of key-value pairs is large. However, a data block stores multiple key-value pairs; if only a mapping from keys to data blocks is established, the index size will theoretically be much smaller. Since the data block can be a memory page, meaning only a mapping from query keys to memory pages is established, the index size will be significantly reduced. Because each disk data query involves at least one I / O operation, if a memory page contains multiple key-value pairs, a binary search can be performed within the read memory page to perform a secondary location, thus utilizing only one I / O operation to retrieve the data.

[0052]

[0054] In one implementation, the process of querying the memory location of the target query key in a compressed multi-level data index, level by level, includes: Based on the target query key, searching the highest-level query key index LN containing the target query key in the compressed multi-level data index, where N is the number of levels in the multi-level data index; Starting from the highest-level query key index LN, searching the next-level query key index Lk-1 containing the target query key in the previous-level query key index Lk, until the lowest-level query key index L1 is found, where each previous-level query key index Lk in the multi-level data index includes multiple next-level query key indices Lk-1, k ∈ [1, N]; and finding the corresponding memory location based on the lowest-level query key index L1.

[0053]

[0055] The number of top-level query key indexes (LNs) is generally small. The top-level query key index (LN) to which the target query key belongs can be determined directly by comparing the query key recorded in the LN with the target query key. While certain search methods can be used to improve search efficiency, theoretically, the improvement is not significant.

[0054]

[0056] For intermediate-level query key indexes, the relationship between the number of upper-level query key indexes (Lk) and lower-level query key indexes (Lk-1) may not be fixed. For example, a second-level query key index might contain 1024 first-level query key indexes, while a third-level query key index might contain 4096 second-level query key indexes. This would lead to using the same approach, impacting query speed. Different query methods can be employed for different levels to ensure query efficiency.

[0055]

[0057] When querying the lowest-level query key index L1, a matching search method is also used to locate the corresponding memory location, ensuring search efficiency. Thus, based on multi-level data indexes, the memory location of the target query key can be quickly located.

[0056]

[0058] In one implementation, based on the target query key, the highest-level query key index LN containing the target query key is searched in the compressed multi-level data index. This includes: if there is only one highest-level query key index LN, the next-level query key index LN-1 containing the target query key is directly searched within the highest-level query key index LN; if there are multiple highest-level query key indexes LN, which record the query keys of some data in chronological order, the next-level query key index LN-1 containing the target query key is searched within the highest-level query key indexes LN using a binary search or traversal method, wherein the number of multiple highest-level query key indexes LN does not exceed a first preset number.

[0057]

[0059] The highest-level query key index LN can be single. When there is only one highest-level query key index LN, the search term directly checks the next-level query key index LN-L containing the target query key within the highest-level query key index LN. Alternatively, there can be multiple highest-level query key indexes LN. In this case, multiple highest-level query key indexes LN record query keys for some data in chronological order. The search term then uses a binary search or traversal method to find the next-level query key index LN-L containing the target query key within the highest-level query key index LN.

[0058]

[0060] The number of multiple top-level query key indexes (LNs) does not exceed a first preset number. This is to facilitate fast searching and improve the query efficiency of the query key. Preferably, in this embodiment, the number of top-level query key indexes (LNs) is 1, so that no search is needed at the top level, and the query can directly proceed to the next level.

[0059]

[0061] In one implementation, searching for the next-level query key index Lk-1 containing the target query key in the parent query key index Lk, until the lowest-level query key index L1 is found, includes: if the next-level query key index Lk-1 in the parent query key index Lk records a query key, querying the next-level query key index Lk-1 containing the target query key according to the number of next-level query key indexes Lk-1; if the next-level query key index Lk-1 in the parent query key index Lk records a memory location, querying the next-level query key index Lk-1 containing the target query key according to the attribute information of the parent query key index Lk and the prefix tree, wherein the attribute information includes information related to the prefix tree used to generate the parent query key index Lk, the prefix tree recording the data of the next-level query key index Lk-1 recorded in the order of the previous query key index Lk, generated according to the left-to-right data order of the query keys of the multiple next-level query key indexes Lk-1 recorded by the parent query key index Lk; through a step-by-step query method... Find the lowest-level query key index L1 containing the target query key.

[0060]

[0062] For different levels, appropriate query methods can be used to ensure query efficiency. Specifically, if a record in the next-level query key index Lk-1 of the previous-level query key index Lk contains a query key, the query method is determined based on the number of records in the next-level query key index Lk-1 containing the target query key. If the number of records in the next-level query key index Lk-1 is small, the query key can be used directly to ensure query speed. If the number of records in the next-level query key index Lk-1 is large, other efficient query methods can be used.

[0061]

[0063] If a record in the next-level query key index Lk-1 of the parent query key index Lk has a memory location, the next-level query key index Lk-1 containing the target query key is queried based on the attribute information of the parent query key index Lk and the prefix tree. This is because when the memory location is summarized to the lowest-level query key index, a mapping is established using the corresponding maximum / minimum query keys. In this way, in multi-level data indexes, the memory location may also be indicated by the query key of the record. Since the amount of memory is often large, it is necessary to use attribute information and the prefix tree for querying.

[0062]

[0064] The attribute information includes the bitstream used to generate the parent query key index Lk, which is related to the prefix tree. The prefix tree records the data of the next-level query key index Lk-1 recorded in the order they were generated when the parent query key index Lk was created. The prefix tree is generated from left to right based on the data order of the multiple query keys of the next-level query key index Lk-1 recorded by the parent query key index Lk. The specific query method will be explained in detail later.

[0063]

[0065] Here, based on the attribute information of the upper-level query key index Lk and the prefix tree, the lower-level query key index Lk-1 where the target query key is located can be queried. The direct lookup jump table of the corresponding prefix tree in the attribute information can be used to quickly query the prefix tree, so as to realize the fast direct lookup of the compressed multi-level data index, which is the compressed direct lookup of the multi-level data index mentioned above.

[0064]

[0066] It should be noted that the prefix tree records the data of the next-level query key index Lk-1 recorded in the order they were generated when the previous-level query key index Lk was generated. The next-level query key index Lk-1 records the query keys using binary data. The first generated next-level query key index Lk-1 is stored to the left, and the query keys of the generated next-level query key index Lk-1 are recorded from left to right. In other words, the prefix tree is generated based on the left-to-right data order of the query keys of the multiple next-level query key indexes Lk-1 recorded by the previous-level query key index Lk.

[0065]

[0067] In other words, the query key indexes at each level are generated through ordered partitioning. This allows for a quick search using binary search when the number of query key indexes at the next lower level is small. Combined with the methods described above, which utilize direct lookup jump tables and prefix trees, the query key index for the target query key can be found quickly even with large datasets. This enables the efficient and fast retrieval of the target query value using multi-level data indexes.

[0066]

[0068] Depending on the specific circumstances of each level, the lowest-level query key index L1 containing the target query key is found through a step-by-step query approach.

[0067]

[0069] In one implementation, when a query key is recorded in the next-level query key index Lk-1 within the parent query key index Lk, the next-level query key index Lk-1 containing the target query key is queried based on the number of next-level query key indexes Lk-1. This includes: if the number of next-level query key indexes Lk-1 within the parent query key index Lk does not exceed a second preset number, searching for the next-level query key index Lk-1 containing the target query key using a binary search method. Here, the multiple next-level query key indexes of the highest-level query key index record the compressed query key (with common prefixes removed) in ascending order. If the number of next-level query key indexes Lk-1 within the parent query key index Lk exceeds a second preset number, searching for the next-level query key index Lk-L containing the target query key using the attribute information of the parent query key index Lk and the prefix tree.

[0068]

[0070] If the number of next-level query key indexes Lk-1 in the parent query key index Lk does not exceed a second preset number (e.g., the second preset number is 10), a binary search is used to find the next-level query key index Lk-L containing the target query key. It should be noted that the multiple next-level query key indexes Lk-1 of the highest-level query key index record the compressed query keys after removing common prefixes, in ascending order. The corresponding query key index can be quickly found using a simple binary search or a direct lookup method.

[0069]

[0071] If the number of sub-level query key indexes Lk-1 within the parent query key index Lk exceeds the second preset number, it indicates that a simple search method may slow down the search speed. In this case, by using the attribute information of the parent query key index Lk and the prefix tree, searching for the sub-level query key index Lk-1 containing the target query key can ensure a faster search speed.

[0070]

[0072] In one implementation, if the next-level query key index Lk-1 in the parent query key index Lk has a memory location, the next-level query key index Lk-1 containing the target query key is located using the attribute information of the parent query key index Lk and the prefix tree. This includes: obtaining the direct lookup jump table of the prefix tree from the attribute information of the parent query key index Lk, wherein the direct lookup jump table records the jump data of different subtrees in the historical queries of the prefix tree; during the search for the target query key, if a corresponding subtree jump record is found, a jump is performed directly using the subtree jump record in the direct lookup jump table; if no corresponding subtree jump record is found, the prefix tree is traversed until the leaf node corresponding to the target query key is found; and the next-level query key index to which the target query key belongs is determined based on the leaf node.

[0071]

[0073] The lookup method of a prefix tree requires node searching, which can affect the search speed when the number of nodes in the prefix tree is large. Therefore, this embodiment provides a direct lookup jump table for the prefix tree. This table records the jump data between different subtrees during historical queries of some nodes in the prefix tree.

[0072]

[0074] During the traversal of the prefix tree, when passing through nodes recorded in the direct lookup jump table, fast jumps can be achieved based on the jump information recorded in the table. In this embodiment, some nodes recorded in the direct lookup jump table can be high-frequency nodes in the prefix tree, that is, nodes with a high probability of being traversed. The direct lookup jump table records jump information of subtrees closer to the root node in the prefix tree as much as possible. This is because, generally, the closer to the root node, the more leaf nodes it covers, and the higher the jump efficiency.

[0073]

[0075] In the process of finding the target query key, the direct lookup jump table of the prefix tree is first obtained. The direct lookup jump table is then checked to see if a corresponding subtree jump record exists. If a corresponding subtree jump record is found, the jump is performed directly through the subtree jump record in the direct lookup jump table, which greatly improves query efficiency.

[0074]

[0076] If no corresponding subtree jump record is found, the prefix tree is traversed directly until the leaf node corresponding to the target query key is found. Then, the next-level query key index to which the target query key belongs is determined based on the leaf node.

[0075]

[0077] The above direct lookup redirect table can be quickly redirected using an acceleration plugin. The acceleration plugin can be hints, which are recorded in the attribute information, and the attribute information is recorded in the corresponding query key index.

[0076]

[0078] In one implementation, finding the corresponding memory location based on the lowest-level query key index L1 includes: based on the attribute information of the lowest-level query key index L1 and the prefix tree, querying the memory location to which the target query key belongs in the lowest-level query key index L1. The attribute information of the lowest-level query key index L1 records the maximum or minimum query key of the mapped memory page, serving as the basis for query key lookup. The query result of the target query key in the prefix tree serves as the basis for looking up the target memory page, or the range of multiple adjacent memory pages. Memory locations are stored in units of memory pages. If the query result clearly indicates that the target query key belongs to the target memory page or multiple adjacent memory pages, the target memory page or multiple adjacent memory pages are used as the corresponding memory location.

[0079] When generating a multi-level data index based on memory pages, the maximum / minimum lookup key is used to indicate the range of the corresponding lookup key. When searching for the memory page of the target lookup key in the multi-level data index, the attribute information of the lowest-level lookup key index L1 and the prefix tree are used to query the memory page to which the target lookup key belongs in the lowest-level lookup key index L1.

[0077]

[0080] The multi-level data index described above maps query keys to memory pages, and a memory page often contains multiple query keys. When using a prefix tree for querying, the final query result may be the search criteria for a specific target memory page, or the search criteria range for two adjacent memory pages.

[0078]

[0081] If the query results clearly identify the target query key and it belongs to the target memory page or two adjacent memory pages, then the target memory page or the two adjacent memory pages are used as the corresponding memory page. The corresponding memory page is also the range of key-value pairs to which the final target query key belongs, and thus the range of memory pages that store the key-value pairs.

[0079]

[0082] In one implementation, obtaining the key-value pairs of a memory page and searching for the target query value corresponding to the target query key includes: loading the corresponding memory page from the disk storing the data through a single read / write operation, obtaining the key-value pairs in the corresponding memory page, wherein the memory page is the data storage structure of the disk; and searching for the target query value corresponding to the target query key in the key-value pairs of the corresponding memory page using a binary search method.

[0080]

[0083] Multi-level data indexes are stored in memory, while key-value pairs are stored on disk. Memory pages are also a disk storage structure. The purpose of multi-level data indexes is to determine a sufficiently small range of memory pages based on a given target query key. Then, using a single disk I / O operation, the corresponding memory page is retrieved into memory. In memory, queries and comparisons are performed to determine the key-value pairs matching the target query key and the corresponding target query value. Compared to existing technologies, this method consumes minimal disk resources, minimizing the impact of query operations on disk operations.

[0081]

[0084] Figure 3 shows a flowchart of a disk data index generation method provided in an embodiment of this disclosure. This disclosure also provides a disk data index generation method, as shown in Figure 3, which is a flowchart of a disk data index generation method according to an embodiment of this disclosure. The method may include the following steps.

[0082]

[0085] Step S301: Receive the target data to be stored and generate key-value pairs for the target data. The key-value pairs include the target query key and the target query value of the target data. The target query key is used to find the corresponding target query value, and the target query value is used to indicate the target data.

[0083]

[0086] Step S302: Write the key-value pair into the current unit memory location. After the unit memory location is full, extract the identifier of the unit memory location and generate the lowest-level query key index of the multi-level data index based on the identifier.

[0084]

[0087] In step S3O3, as the stored data increases, a multi-level data index is generated and compressed level by level based on the lowest-level query key index. The multi-level data index includes multiple levels of query key indexes used to find the memory location of the target query key. The query key index is generated based on a binary prefix tree and is used to map the next level query key index or memory location.

[0085]

[0088] According to the disk data index generation method described in this embodiment, after receiving the target data to be stored and generating key-value pairs for the target data, the key-value pairs are written into the current unit memory location. After the unit memory location is full, the identifier of the unit memory location is extracted, and the lowest-level query key index of the multi-level data index is generated based on the page identifier. As the stored data increases, multi-level data indexes are generated level by level. The multi-level data index can efficiently compress the query keys of the stored data, reduce the resource consumption of the data index, and establish a mapping relationship between the memory location of the stored data and the query key. Thus, through this multi-level data index, the key-value pair corresponding to the query key of the target data to be searched can be quickly found, that is, the memory location of the target data on the disk.

[0086]

[0089] During queries, multi-level data indexes can be compressed and directly accessed without decompressing the entire index. This not only improves query efficiency but also reduces resource consumption. Furthermore, the lowest-level query key index of the multi-level data index directly maps to memory locations. During a search, only a single read / write operation is needed to retrieve the memory location corresponding to the target data, significantly reducing read / write resource consumption. This solves the problem of low search efficiency inherent in disk-based search methods in related technologies.

[0087]

[0090] The process involves receiving the target data to be stored, generating key-value pairs for the target data, including a target query key and a target query value. The target query key is used to locate the corresponding target data, and the target query value indicates the memory location of the target data. Then, the target data is stored.

[0088]

[0091] In other words, key-value pairs are written to the current unit memory location, where the capacity is fixed, such as a memory page. Once the unit memory location is full, its identifier is written to the lowest-level query key index of the multi-level data index. The amount of data in the query key index of the multi-level data index is also fixed, and multi-level data indexes are generated progressively as the stored data increases.

[0089]

[0092] After multi-level data indexes are generated, they can be compressed. The compression method supports compressed direct lookup, ensuring fast queries while consuming less storage and computing resources. The compression method will be explained in detail later.

[0090]

[0093] In one implementation, key-value pairs are written to the current unit memory location. After the unit memory location is full, the identifier of the unit memory location is written to the lowest-level query key index of the multi-level data index. This includes: writing key-value pairs to the current unit memory location; after the unit memory location is full, extracting the largest / smallest query key in the unit memory location; generating a prefix tree of the lowest-level query key index L1 based on the largest / smallest query key; when the number of unit memory locations corresponding to the prefix tree of the same lowest-level query key index L1 reaches a third preset number, generating the lowest-level query key index L1 based on the prefix tree and saving the corresponding attribute information, wherein the attribute information includes relevant information for generating the prefix tree of the next-level query key index Lk, and the prefix tree records the next-level query key index Lk-1 arranged in size order; creating a new lowest-level query key index L1 and generating the corresponding prefix tree; and continuing to process the newly stored data.

[0091]

[0094] The aforementioned prefix tree can exist in the form of a bit stream, and the data recorded in the bit stream is also an array of prefix trees.

[0092]

[0095] The third preset number mentioned above is also the unit memory location that a lowest-level query key index L1 can accommodate, which is the maximum number of memory pages, such as 1024, 2048, 4096, etc. It should be noted that since the length of the query key may vary, the memory pages that the lowest-level query key index can accommodate can be limited from the perspective of data volume, for example, 4KB.

[0093]

[0096] As shown in Figure 9, the aforementioned attribute information can be used for the level 0 bucket LO_bucket: off, len, and skip are used to locate the starting position of the bit_stream search; key_off and key_sz are used to record the position of the maximum lookup key in the lookup key array within the level 0 bucket L0_bucket. Hints are used to accelerate the bit_stream search, meaning that with hints, it is not necessary to decompress the entire bit_stream during the search... Its size can be adjusted according to actual business needs.

[0094]

[0097] In other words, the attribute information includes parameters used to locate the starting position of the lookup bitstream, and the position of the maximum lookup key in the lookup key array. Based on this information, it is possible to find out which next-level lookup key index Lk-L the target lookup key belongs to.

[0095]

[0098] In one implementation, as the stored data increases, the next-level query key index Lk-1 gradually accumulates. When the next-level query key index Lk-1 reaches a fourth preset number that a previous-level query key index Lk can tolerate, a corresponding previous-level query key index Lk is generated, and the corresponding attribute information is saved; this continues until the highest-level query key index LN is generated, wherein the number of the highest-level query key index LN does not exceed a fifth preset number.

[0096]

[0099] The aforementioned fourth preset quantity can be multiple, referring to the number of lower-level query key indexes Lk-1 that a higher-level query key index Lk can allow to be summarized. It should be noted that the number of lower-level query key indexes Lk-1 that each higher-level query key index Lk can allow to be summarized can be different or the same. For example, as shown in Figure 9, a level 2 bucket L2_bucket can hold 4096 level 1 buckets Ll_bucket, and a level 1 bucket Ll_bucket can hold 4096 level 0 buckets L0_bucket.

[0097]

[0100] In one implementation, the method further includes: determining the common prefix of the query keys corresponding to the query key indexes at each level, and pruning the common prefix to retain the identifier portion of the query key; creating a binary prefix tree of the identifier portion of the query keys of the stored data; compressing the binary prefix tree into a compact array using a compact recursive representation; and compressing the compact array to obtain a prefix tree, wherein the prefix tree is used for multi-level data index lookup to find the query key indexes at each level to which the target query key belongs.

[0098]

[0101] When generating multi-level data indexes, a prefix tree can be generated based on the query key to serve as the basis for subsequent queries. It's important to note that memory pages only provide the maximum / minimum query key. A single lowest-level query key index can accommodate many query keys, each representing the maximum / minimum query key of a memory page. When the number of query keys is large, conventional query methods will become ineffective, necessitating the use of a prefix tree.

[0099]

[0102] The construction of a prefix tree involves generating a prefix tree based on the query key recorded in a query key index after the index is generated. The generation method and principle of the prefix tree are similar for query key indexes at other levels in a multi-level data index. Each higher-level query key index is generated by summarizing multiple lower-level query key indexes, and each lower-level query key index can provide the maximum / minimum query key that represents its corresponding query key range.

[0100]

[0103] The aforementioned prefix tree exists physically in the form of the aforementioned bit stream.

[0101]

[0104] In one implementation, compression based on a compact array to obtain a prefix tree includes: constructing a Huffman tree based on the compact array and determining the value of each element; compressing the compact array using Huffman compression when the value of each element exceeds a sixth preset number; and compressing the compact array using a preset integer compression method when the value of each element does not exceed the sixth preset number, wherein the preset integer compression method is a non-Huffman compression method.

[0102]

[0105] Although a compact array is just an array, it still occupies a significant amount of memory. Each intermediate node must occupy a position in the array. Therefore, integer compression techniques are used for compression. Each element in a compact array is an unsigned integer. Its values ​​generally exhibit a binomial distribution, which allows for the direct construction of a Huffman tree without sampling the data in the compact array.

[0103]

[0106] However, if there is too much data, the Huffman decompression process can be very lengthy. Therefore, Huffman compression can be used for frequently occurring data, while direct integer compression, such as gamma coding, can be used for less frequently occurring data.

[0104]

[0107] The closer a middle node is to the root node, the less frequently it appears; similarly, the closer a middle node is to a leaf node, the more frequently it appears. A threshold can be set; for example, Huffman compression can be used when the value of an element in the array is <= 16, and gamma compression can be used when the value of an element is > 16.

[0105]

[0108] It should be noted that this embodiment also provides an optional implementation method, which proposes a novel highly compressed multi-level data index, Level-Trie. This method extracts the maximum value of each data block and compresses it into high-density binary retrieval data. This multi-level data index has the following three characteristics: 1. Extremely low memory usage: For any type of data, the average extraction query key for each data block occupies only 3 bits of memory, which can be loaded into memory at once. 2. Extremely fast generation speed: A single thread can theoretically support an average disk I / O of 34GB / s, far exceeding the current hardware limits. 3. Compressible direct lookup: No decompression is required for direct lookup; the data block ID corresponding to a given query key can be calculated in O(1) time.

[0106]

[0109] The Level-Trie proposed in this implementation is a high-level data structure for disk retrieval, improved upon related technologies. Among these technologies, SILT is a memory-oriented algorithm for calculating the minimum perfect hash, offering high compression efficiency but lacking direct lookup support and exhibiting very slow query performance with large datasets. SuRF is a range filter capable of direct lookup. SuRF is also a type of index trie, building prefix indexes on a byte-by-byte basis, thus occupying relatively large space, typically about 30%-50% of the original data. COIndex is an improvement on SuRF, providing a mapping from query keys to data block IDs, and is used in ToplingDB and TerakDB. However, the large space consumption issue of SuRF remains unresolved, making it difficult to load entirely into memory.

[0107]

[0110] SILT, being a binary prefix index, boasts high compression efficiency and fast generation speed. However, it cannot perform direct lookups with compression, and its query speed is slow when dealing with large datasets. SuRF, a memory range filter, is essentially a byte prefix index that allows for direct lookups with compression. However, SuRF typically consumes a lot of memory and cannot be directly used for disk retrieval. In other words, its purpose is to calculate the existence of a given range, not to map a given query key to a data block ID. This implementation's Level-Trie inherits the high compression ratio and fast generation speed of SILT's binary prefix index, supports direct lookups with compression, and proposes several optimizations for disk retrieval, providing adjustable query performance even with large datasets.

[0108]

[0111] Level-Trie addresses the following three shortcomings of SILT: 1. It does not support compressed direct lookups. 2. The larger the data volume, the slower the query speed. 3. It does not support disk retrieval. It proposes the following three targeted optimization strategies: 1. Introducing a direct lookup skip list to achieve direct jump queries without consuming significant CPU resources for decompression operations. 2. Ordered data partitioning ensures a fixed compressed block size, thus keeping the overall cost of index retrieval constant and independent of data volume. 3. Introducing a multi-level index and sampling key mechanism to establish a mapping relationship between the index trie and disk data blocks, transforming the calculation of data block IDs into the index trie query process.

[0109]

[0112] A compressed index trie is essentially a binary prefix tree. An index trie indexes data through prefixes, and its main advantages are threefold: (1) it only needs to record the smallest distinguishable prefix to uniquely index a given query key; (2) common prefixes can be skipped; and (3) the tree structure naturally preserves order. Assuming a binary index trie is built for seven data items, its final structure is shown in Figure 4 above. Figure 4 shows a schematic diagram of the binary index trie of this embodiment. If the data are all of type uintl6, the common 0 bits in the higher bits can be skipped directly without recording. In this example, the first 11 bits of both query keys are 0, so skip_bits = 11, meaning that the higher 11 bits are skipped during the query.

[0110]

[0113] The query key can be stored in big-endian order. For a given query key, common prefixes can be skipped first, and then prefix matching can be performed based on the binary sequence of the query key. The corresponding data can be retrieved when a leaf node is reached. The trie index does not need to record the entire binary sequence of the query key. As shown in Figure 5, which illustrates a schematic diagram of the query key in an example of this embodiment, only the smallest distinguishable prefix needs to be recorded. For example, for data 2, recording 01 is sufficient for a unique match.

[0111]

[0114] Directly storing the generated trie index would incur significant memory overhead. For example, each intermediate node would need to record pointers to its left and right child nodes, and each leaf node would need to record its corresponding value. Secondly, this structure is not conducive to data serialization and deserialization, which is crucial for rapid index generation and startup. Furthermore, trie indexes may lead to erroneous queries for keys with the same common prefix; for example, two binary sequences might both return 3 when queried in the trie index. Therefore, the index needs additional detection and correction logic for erroneous queries.

[0112]

[0115] Compact recursive representation refers to converting the general trie indexes mentioned above into another compact data structure, such as an array, which makes it easier to persist and load from disk. The compact form of a trie can be represented by the following formula: repr(T) := |L|repr(L)repr(R)

[0113]

[0116] This means that for a compressed trie T, L represents the left subtree of T; similarly, R represents the right subtree of T; |L| represents the number of leaf nodes in trie L. This method can transform a general trie into a trie with information about the number of leaf nodes.

[0114]

[0117] Figure 6 illustrates the trie index compression and query principle of an example embodiment of this disclosure. As shown in Figure 6, (a) represents the initial general trie index. By counting the number of leaf nodes in the trie index, the form with intermediate node information in Figure (b) can be obtained. Then, an inorder traversal of the trie index in Figure (b) yields the compact form in Figure (c). Figure (c) is logically equivalent to Figure (a). For example, querying 10010, Figure (d) shows its query path on the general trie index; if the highest bit is 1, its right subtree is queried; if the second bit is 0, its left subtree is queried; if the third highest bit is 0, the left subtree is queried, eventually reaching the leaf node, whose value is 3.

[0115]

[0118] For graph (e), the query logic is as follows: First, look at the root node (the first one in the array), which is 3, a non-leaf node; the highest bit of the query key is 1, so the right subtree should be queried, skipping the left node, and the right subtree is 3; this node is not a leaf node, the second highest bit of the query key is 0, so the left subtree needs to be queried, and it is still not a leaf node; the third highest bit is 0, so the left subtree needs to be queried, but its left subtree is a leaf node (1 indicates that its left subtree is a leaf node); thus, we can know that the position of the leaf node is 3 (root) + 1 (left) = 4, that is, its position is the fourth one in the disk data. Querying the data on the disk shows that the query value corresponding to the given query key is 3. The equivalent transformation of the graph requires an efficient algorithm. Given a sorted dataset, transform the order of (b) and (c). The generation of the structure in graph (c) can be directly achieved through a quicksort strategy. Given an ordered dataset T, first scan the bits sequentially, denote bits = 0 as the left subtree and generate the dataset accordingly, and bits = 1 as the right subtree. Recursively generate an array representation of each subtree, with the depth of the scanned bits increasing with each recursive level.

[0116]

[0120] As can be seen from the above algorithm, the compact array is generated in a left-to-right order. This order is important because it indicates that the compressed trie is generated in a streaming manner, allowing data blocks to be generated and indexes to be generated simultaneously during data writing. It eliminates the need to wait for all data to be scanned before generating the index.

[0117]

[0121] Data querying is relatively simple. Given a query key `key` and a compact representation array `trepr`, the left and right subtrees of `trepr` are recursively queried based on the corresponding bit of the query key. If the bit corresponding to the query key is 1, it indicates that the right subtree needs to be queried. However, querying the right subtree requires knowing how many intermediate nodes the left subtree has, so as to directly jump to the corresponding node in the right subtree. This requires first recursively visiting all intermediate nodes in the left subtree and determining their number.

[0118]

[0122] In a recursive query, if a leaf node is reached, the query ends. The values ​​of the intermediate nodes accessed along the query path are summed to obtain the position of the corresponding query key in the ordered dataset. Then, the corresponding value is read from the disk based on this position information to complete the entire query. Therefore, for each right subtree, the entire left subtree of the trie needs to be dynamically accessed during the query. When the data volume is large, the decompression process is very lengthy.

[0119]

[0123] Prior array compression: Compact arrays represent the same trie index as an array, but still occupy a large amount of memory. Each intermediate node in the trie index must occupy a position in the array. Typically, the number of intermediate nodes in the trie index is three times that of the leaf nodes, meaning the number of elements in the compact array is three times the number of key-value pairs. Assuming a large number of key-value pairs, such as hundreds of thousands to millions, each element in the array may require 4 bytes to represent. 100 million key-value pairs would require an average of 381MB of memory, which is clearly insufficient.

[0120]

[0124] The compact array stores unsigned integers, which can be compressed using existing mature integer compression techniques. However, compared to general integer compression algorithms, the compact array trie index has its own data distribution characteristics, thus allowing for targeted compression. Each element in the compact array trie index is an unsigned integer. For a trie T, the leaf nodes in its left subtree L have an average value of |L|=|R|=|T| / 2, therefore its values ​​generally exhibit a binomial distribution characteristic. Based on this data distribution characteristic, a Huffman tree can be directly pre-constructed without sampling the data in the compact array.

[0121]

[0125] However, if there is too much data, the Huffman tree decompression process may be very lengthy. Therefore, Huffman compression can be used for data that occurs frequently, while direct integer compression, such as gamma encoding, can be used for data that occurs less frequently. Obviously, the closer the intermediate node is to the root node, the lower its frequency of occurrence; similarly, the closer the intermediate node is to the leaf node, the higher its frequency of occurrence. A threshold can be set, for example, Huffman compression can be used when the number of array elements is <= 16, and gamma compression can be used when the number of elements is > 16.

[0122]

[0126] In addition, the pre-construction of the Huffman tree can be further optimized. One approach is to pre-construct a large Huffman tree for the data distribution; however, this may result in a large Huffman tree, reducing the efficiency of compression and decompression. Another strategy is to pre-construct a Huffman tree for each data type, for a total of 14 trees. During a query, the corresponding Huffman tree can be queried based on the number of leaf nodes in the subtree for compression and decompression, which can further improve the efficiency of compression and decompression.

[0127] Gamma compression is a context-free compression for a single integer. Its core idea is to reduce the number of invalid digits in the integer. For example, for the uintl6 type, it first calculates the number of leading zeros of the integer, 11, then the number of significant digits is 16-11=5. Then, 5-1=4 zeros are added before the number of significant digits, i.e., 000010010, compressing from 16 bits to 9 bits. For a trie T, the expected number of leaf nodes in its left and right subtrees is given by a certain value. Therefore, we can store the difference between the actual number of leaf nodes in the subtrees and the expected number (the idea of ​​delta-coding) to minimize the number of significant digits. This difference may be negative, so it needs to be transformed for easy compression. The commonly used Golomb compression coding scheme can be used. With this scheme, each element in the compact array requires an average of only 3 bits of space.

[0123]

[0128] The Level-Trie index of this embodiment will be described in detail below. SILT essentially maps query keys to data block slot IDs, that is, it maps different types of ordered query keys to continuously increasing integer IDs. Its compact array is compressed using a variety of methods, achieving very low memory usage. However, it must be decompressed before querying. When the data volume is large, its query efficiency is very low. Level-Trie is essentially an extension of SILT, but it introduces new mapping semantics for disk data. o Level-Trie enables fast retrieval of disk data blocks by mapping index keys to data block IDs. The design of Level-Trie follows three principles:

[0124]

[0129] First, key-value pairs are not allowed to be stored across data blocks. Level-Trie is used to provide a mapping from query key to data block. That is, if the disk data block size is 4KB, the size of a single key-value pair cannot exceed 4KB. If key-value pairs are stored across multiple data blocks, a unique data block ID cannot be calculated for a given query key. This limitation can be mitigated by splitting large key-value pairs into data and index parts. The data block only stores the index part, and the data part is queried through a secondary index. The original SILT requires establishing a mapping from query key to data block slot, which poses a significant problem when there are many key-value pairs. If only a mapping from query key to memory page is established, the amount of index will be greatly reduced. Since each disk data query involves at least one I / O, and since one I / O is always required to read data from the disk, if a memory page contains multiple key-value pairs, a binary search can be performed within the read memory page for secondary location.

[0125]

[0130] The key issue here is how to establish a mapping from the query key "key" to the ID of the memory page "page". SILT establishes a mapping from the query key "key" to the data block "slot", which is a one-to-one mapping relationship. However, the mapping from the query key "key" to the memory page "page" ID is a many-to-one mapping relationship. It is necessary to calculate a representative value for multiple key-value pairs within a memory page "page" and establish a many-to-one mapping from the query key "key" to this representative value. The approach of Level-Trie is to select the maximum query key "key" within a memory page "page", and then combine the maximum query keys "key" within multiple memory pages "page" to establish SILT. For example, 256 memory pages "page" are grouped together, 256 query keys "key" are selected, and then SIKT is established. In this way, only 256 query keys "key" need to be established for SILT indexing of 1MB of data. Assuming that the average number of bits required for each query key "key" is 3, the index for 1MB of data only requires an average of 96 bytes. <OO00262><OO00263>

[0131] However, there is a problem with this. The original SILT establishes a one-to-one mapping from the query key "key" to the data block "slot", and the queried query key "key" must exist. However, the above scheme selects a query key "key" from each memory page "page", so in fact most query keys "key" do not exist during querying. For non-existent query keys "key", there is a natural defect in the trie index data structure. The trie index performs prefix matching, and the smallest prefix match can uniquely determine the corresponding query key "key". However, when the query key "key" does not exist, only the data in the trie index with the most matching prefix to this query key "key" can be matched, so incorrect data may be returned. <OO00264><OO00265>

[0132] Extracting the maximum value from each memory page "page" to establish a trie index returns the maximum value of the corresponding memory page "page". The queried query key "key" may actually be less than, equal to, or greater than the query key "key" corresponding to the maximum value. Assuming that the maximum query key "key" of a certain memory page "page" is a, and the queried query key "key" is x, then if x < a, it indicates that x must belong to the memory page "page" where a is located; if x > a, it indicates that x is within the next memory page "page" of a. <OO00266><OO00267> It should be noted that there may be some inaccuracies in the original text's format and content. This translation is done based on the best understanding of the text. If possible, it is recommended to check and correct the original text for more accurate translation.

[0133] Since it is impossible to determine exactly which memory page x is located in, it is necessary to read two memory pages separately for querying. Two consecutive memory pages can be read in one I / O operation. However, if the left subtree of a certain trie index node is 0 during a query, it indicates that the given query key is definitely less than the query key being matched. If the query depth equals the length of the query key, it indicates that the query key being queried is equal to the query key being matched. Otherwise, other query results cannot determine the size relationship between the given query key and the query key being matched, requiring two memory pages to be loaded at once for further judgment.

[0129]

[0134] II. Random Jump Query under Variable-Length Compression. SILT is highly compressed, and queries usually require decompression. For a trie T repr(T):=|L|repr(L)repr(R), to query the right subtree R, it is necessary to skip the parts of |L| and repr(L). Since each child node in the trie compression is variable-length, when jumping to R, it is necessary to decompress the left subtree L first in order to specifically locate the starting decoding position of R in the compressed compact array. This will cause the query time to become slower and slower as the number of leaf nodes of T increases.

[0130]

[0135] The Level-Trie solution constructs a direct lookup jump table when generating the SILT. When querying a jump, the system first checks if a jump record exists in the direct lookup jump table; if it exists, there's no need to decompress the left subtree, and the query can directly jump to the right subtree. If the direct lookup jump table does not exist, the left subtree still needs to be decompressed. The size of the direct lookup jump table is limited, mainly to balance query performance and memory consumption. Theoretically, if each intermediate node stores jump information, decompression is not needed during queries, but this would result in the largest memory usage.

[0131]

[0136] The direct lookup jump table is constructed based on the principle of recording jump information of subtrees as close to the root node as possible. This is because, generally, the closer to the root node, the more leaf nodes it covers, and the higher the jump efficiency. Figure 7 shows a schematic diagram of the trie index jump according to an embodiment of this disclosure. Figure 7 shows an example of a jump. The direct lookup jump table only stores the jump information of each left subtree, because the location information of the left subtree is only needed when jumping to the right subtree. Assuming T is a complete binary tree, the direct lookup jump table only needs to record nodes with node number 2n*l. The construction of the direct lookup jump table is very simple; when generating the compact trie T, only the constructed position needs to be added to T. Definition: repr(T) := |L|repr(L)[N L ]repr(R)

[0132]

[0137] Where t represents the jump point of the left subtree L, and [] indicates that the jump information recording operation is performed. Since the construction of trie T is strictly from left to right, after L is constructed, the length of the currently constructed bit array is directly obtained as the decoding starting point of the right subtree R. This jump point is stored separately in the lookup jump table. Since the capacity of the lookup jump table is limited, only the jump information close to the root node will be recorded. Jump information exceeding the capacity of the lookup jump table will be directly discarded.

[0133]

[0138] However, since trie T is not necessarily a full binary tree, the recording strategy of the direct lookup jump table can be improved in the future to record as many subtrees as possible. In actual testing, a random query key with a length of 100 bytes was constructed, and then 100,000 random queries were performed. Figure 8 shows a schematic diagram of the impact of the size of the direct lookup jump table on query performance in the embodiments of this disclosure. As shown in Figure 8, the improvement result of the size of the direct lookup jump table on query performance is demonstrated.

[0134]

[0139] Without a direct lookup table, the average single query time for trie is as high as 1367us, degrading performance to the point of being unusable. When the size of the direct lookup table is set to 256 (1KB), the average query time is 5us, a performance improvement of 271 times. At this time, the memory usage of the direct lookup table is only 1KB, and the compressed compact array is 23KB, with a memory usage ratio of 4%. When the size of the direct lookup table is 10KB, the performance improvement is 1366 times, but the memory usage ratio is 40%, which is already very high. Therefore, in practice, a trade-off between query performance and memory usage needs to be made based on business requirements.

[0135]

[0140] III. Multi-level Composite Index. As can be seen from Principle 1 above, each memory page extracts a maximum query key, and then a fixed number of memory pages are grouped together. In this way, a 1MB data segment is sampled to build a SILT index. The problem here is how to index between data segments. One possible approach is to record the maximum query key within each data segment. When searching, a binary search is performed between data segments to determine the data segment where the query key is located. Then, within the corresponding segment, the trie index is searched to determine the memory page where the query key is located.

[0136]

[0141] On the one hand, this approach requires recording the maximum query key in each data segment. Assuming there is 500GB of data, 500*1024 data segments are needed. Further assuming, in an extreme case, the query key length is 3KB, then recording the maximum query key requires 1.5GB of memory. Although this is an extreme case, it still requires a significant amount of memory. Secondly, binary search also consumes considerable time.

[0137]

[0142] Another approach is to extract the maximum query key again on top of the data segment and build a secondary index. Assume a secondary data segment contains 1024 primary data segments, meaning the secondary data segment size is 1GB. Then, the average size of the compact array required to build the index on the secondary data segment is 1024*3=3072. Each element in the compact data is recorded using uint16, requiring 6KB of memory. Memory consumption is very low, therefore, compression of the secondary data segment is unnecessary. Theoretically, 1TB of data requires 6MB of memory for the secondary data segment index and 96MB for the primary data segment compression index. Therefore, theoretically, a total of 102MB of memory is needed.

[0138]

[0143] Figure 9 shows a schematic diagram of the overall architecture of the multi-level data index level-trie according to the embodiments of this disclosure. The overall structure of the level-trie is shown in Figure 9. From low to high, they are memory pages, level 0 buckets (LO_buckets), level 1 buckets (L1_bucket), and level 2 buckets (L2_bucket). o A single memory page stores one or more key-value pairs. This can lead to some space waste; for example, a single 4KB memory page can only store a 3KB key-value pair, resulting in a 25% space waste. The size of the memory page can be configured according to actual business needs, or when the query key is too large, the key-value pair data can be split and stored elsewhere using a secondary index.

[0139]

[0144] 4096 memory pages form a level 0 bucket LO_bucket, which is 16MB in size. The value of 4096 is not fixed and can be dynamically configured according to business needs. A larger level 0 bucket LO_bucket will slow down the query speed, while a smaller level 0 bucket LO_bucket will require more storage space and make metadata management more complicated. Each level 0 bucket LO_bucket will store an additional query key, which is the largest query key covered by that level 0 bucket LO_bucket. The reason for storing this query key is that when querying the trie index, the result may return an uncertain size order, so it is necessary to further compare its size with the query key to determine whether to query the current level 0 bucket LO_bucket or the next level 0 bucket LO_bucket. o

[0140]

[0145] This will further lead to more I / O. oA Level 1 bucket, Ll_bucket, is composed of 4096 Level 0 buckets (LO_bucket). This value can be dynamically configured according to business needs. Thus, a Level 1 bucket (Ll_bucket) can manage 64GB of data. Since compact arrays are not compressed at the L1 level, the maximum memory required for a single Level 1 bucket (Ll_bucket) compact array is 4096 * 2 * 3 = 24KB. Level 1 buckets (Ll_bucket) can use two indexing strategies to search for data. When there are few Level 1 buckets (e.g., less than 10), binary search can be used directly. When the data volume is large (e.g., hundreds), a trie index query can be used. Trie index queries may return uncertain results, requiring the maximum query key (max key) under that bucket to further confirm whether to query the current Level 0 bucket (LO_bucket) or the next Level 0 bucket (LO_bucket). Level-Trie only sets up one Level 2 bucket (L2_bucket), whose indexed data volume is related to N and can be set as a configurable parameter. In practice, a single file of 1TB is relatively rare, meaning it contains 16 level 1 buckets (Ll_bucket). o Since the number of level 1 buckets (Ll_bucket) is generally small, each level 1 bucket (Ll_bucket) only stores the maximum value lookup key, and performs a direct binary search across rows to reduce the complexity of the project implementation.

[0141]

[0146] All level 0 buckets (LO_bucket) share the same data stream, i.e., bit_stream in Figure 9. oThis design is primarily for ease of data persistence and loading. If each level 0 bucket (LO_bucket) had its own compact trie storage space, firstly, the compact trie is variable-length, making space management cumbersome; secondly, when data is persisted to disk and loaded, the corresponding level 0 bucket (LO_bucket)'s SILT needs to be loaded separately, potentially generating significant random I / O and thus failing to effectively utilize storage bandwidth. Each level 0 bucket (LO_bucket) has six attributes: off, len, and skip are used to locate the starting position of the bit_stream search; key_off and key_sz are used to record the position of the maximum lookup key in the lookup key array within that level 0 bucket (LO_bucket). Hints are used to accelerate bit_stream searches; that is, through hints, the entire bit_stream does not need to be decompressed during the search. o Its size can be adjusted according to actual business needs.

[0142]

[0147] Given n query keys, each query key has a size of s. Assuming a single memory page has a size of 4KB, a single memory page can store (4KB-8) / (s+8) key-value pairs. The number of memory pages required for n query keys is n / ((4KB-8) / (s+8)). o On average, each memory page requires an index size of 3 bits. Therefore, the memory required for bit_stream is (n / ((4KB-8) / (s+8)))*3 / 8. 4096 memory pages require one level 0 bucket L0_bucket. Each level 0 bucket LO_bucket requires (80+s) bytes of memory. Therefore, the total memory required for level 0 bucket LO_bucket is (n / ((4KB-8) / (s+8))) / 4096*(80+s). 4096 level 0 buckets LO_bucket form one level 1 bucket Ll_bucket. Each level 1 bucket Ll_bucket requires (6+s) bytes of memory. Therefore, the total memory required for level 1 bucket Ll_bucket is n / ((4KB-8) / (s+8)) / 4096 / 4096*(6+s) bytes. Total memory usage: 80 + s 12 + s\ 4096 + 4096 2 J Figure 10 illustrates the memory usage analysis of a level-trie. It shows the theoretical memory usage analysis results for a level-trie. The left side shows the relationship between memory usage and the number and size of query keys, while the right side shows the relationship between query key size and memory usage under a fixed data volume. In the left side of Figure 10, generally, the more query keys there are and the longer their length, the more memory is required. With 1 billion query keys and a query key size of 3KB, a minimum of 800MB of memory is needed; this is the theoretical worst-case memory usage. In this case, the total space occupied by key-value pairs (KV) is 2.79TB. Assuming the total amount of key-value pair KV data is 1TB, and the length of the query keys increases from 4 bytes to 3KB, then theoretically, the memory usage of a level-trie is below 105.7MB.

[0143]

[0149] Index generation and retrieval: The process of generating bit_stream in the level-trie is the same as that of SILT, but a new query key grouping strategy is designed. During generation, the generation thread (single thread) continuously appends new data to the Level-Trie. When a memory page is generated during the writing process, the largest query key in the memory page is placed into the level 0 bucket query key array (LO key array). When the level 0 bucket query key array reaches a threshold, such as 256, the generation thread scans all query key keys in the level 0 bucket query key array and generates the data stream bit_stream of the level 0 bucket, while recording the starting offset, size, and common prefix length of the level 0 bucket in the bit_stream. After generation is complete, the level 0 bucket query key array is cleared, and the append writing operation continues.

[0144]

[0150] When the number of level 0 buckets (LO_bucket) reaches the threshold of level 1 buckets (Ll_bucket), such as 32768, the maximum query key of the level 0 bucket (LO_bucket) is placed into the query key array (LI key array) of the level 1 bucket. Similarly, when the query key array of the level 1 bucket reaches the threshold of the level 1 bucket (Ll_bucket), such as 32768, the generation thread scans all query keys in the query key array of the level 1 bucket and generates an uncompressed compact trie of the level 1 bucket. At the same time, it records the starting offset, size, common prefix length, and maximum query key of the level 1 bucket in the inti 6 stream.

[0145]

[0151] When generation is complete, the data may not exactly fill a memory page; therefore, the number of key-value pairs in that memory page is recorded and placed into the level 0 bucket lookup key array. When generation is complete, the level 0 bucket lookup key array may not have reached the threshold; in this case, the generation thread calls `finalize` to force the data in the level 0 bucket lookup key array to be generated as a bitstream. The data in the level 1 bucket lookup key array is processed according to the same logic.

[0146]

[0152] When querying a given query key x, first obtain the query entry from the level 2 bucket L2_bucket and check the number of level 1 buckets L1_bucket. If the level 1 bucket L1_bucket only records the query key information, a binary search is needed to determine the level 1 bucket L1_bucket to which it belongs. After obtaining the ID of the level 1 bucket L1_bucket, mark it as L1_id. Query intl6_stream based on the off, len, and skip parameters recorded in the bucket to obtain the level 0 bucket L0_bucket where the query key is located, and mark it as L0_id. Finally, query bit_stream based on the information recorded in L0_bucket to obtain the page identifier page id of the memory page where the query key is located, and mark it as page_id. Then the actual memory page page where the query key x is located is: page

[0147]

[0153] After obtaining the memory page, first check if the memory page exists in the cache. If it does not exist, the corresponding memory page needs to be loaded from the common memory storage. Then, obtain the number of records in the memory page and then perform a binary search. The entire process of locating the memory page is completed in memory, requiring at most one I / O operation. The specific retrieval process is as follows:

[0154] 1. Perform a binary search on the maximum key array of the level 2 bucket L2_bucket to determine the level 1 bucket L1_bucket...

[0148]

[0155] 2. Check the number of level 0 buckets LO_bucket contained in the level 1 bucket Ll_bucket. If the number is less than a certain number, such as 128, use binary search to locate the corresponding level 0 bucket LO_bucket; if the number is greater than 128, query the trie index and use the max key array to speed up the confirmation of the level 0 bucket LO_bucket it belongs to.

[0149]

[0156] 3. Query the bit_stream position corresponding to the level 0 bucket LO_bucket to find the memory page page corresponding to the query key. The Trie index may return an uncertain result, that is, it is impossible to determine whether the query key is in memory page page=a or memory page page=a+l. In this case, two consecutive memory pages need to be read for the search, which will consume at most _ I / O operations.

[0150]

[0157] 4. After loading the corresponding memory page, locate the position of the given key-value pair within the page using a binary search.

[0151]

[0158] 5. Read the query value at this position, and the query ends.

[0152]

[0159] This embodiment uses a binary prefix tree, which has a higher compression rate than the byte prefix tree of SuRF, requiring only 3 bits per query key on average. It supports compressed direct lookup by using ordered data segmentation and a direct lookup skip list, solving the problem of slow querying in SILT with large data volumes. It supports disk retrieval using a multi-level index, unlike SuRF and SILT which do not directly support disk retrieval. The aforementioned binary prefix tree achieves efficient index compression. Ordered data partitioning and auxiliary direct lookup skip lists enable fast compressed direct lookup in large data volumes. The multi-level index table provides efficient support for disk retrieval scenarios.

[0153]

[0160] FIG11 shows a structural block diagram of an apparatus for searching data on a disk provided in an embodiment of the present disclosure. Corresponding to the application scenario and method of the method provided in the embodiment of the present disclosure, the embodiment of the present disclosure also provides an apparatus for searching data on a disk. Figure 11 shows a structural block diagram of a device for data retrieval on a disk according to an embodiment of the present disclosure. The device may include: a receiving module 1101, used to receive the target query key of the target data to be retrieved; an indexing module 1102, connected to the receiving module 1101, used to query the memory location of the target query key level by level in a multi-level data index, wherein the multi-level data index is generated and compressed during data storage and includes multiple levels of query key indexes, the query key index being generated based on a binary prefix tree and used to map the next level query key index or memory location; and a searching module 1103, connected to the indexing module 1102, used to obtain stored key-value pairs based on memory locations and search for the target query value corresponding to the target query key, wherein the memory location stores multiple key-value pairs, each key-value pair including a query key and a query value corresponding to the query key, the query value indicating the corresponding data, and the target query value indicating the target data.

[0154]

[0161] According to the data lookup device described in this embodiment, after receiving the target query key of the target data to be searched, the device queries the memory location corresponding to the target query key level by level in the multi-level data index using a compressed direct lookup method. The multi-level data index can efficiently compress the query keys of the stored data, reducing the resource consumption of the data index, and establishing a mapping relationship between the memory location of the stored data and the query key. Thus, the key-value pair corresponding to the query key of the target data to be searched can be quickly found through the multi-level data index, which is the memory location of the target data on the disk.

[0155]

[0162] During querying, multi-level data indexes can be compressed and directly searched without decompressing the entire index, which not only improves query efficiency but also reduces the resources consumed during querying. Moreover, the lowest-level query key index of the multi-level data index directly maps to the memory location. During the search, only one read / write operation is needed to obtain the memory page corresponding to the target data, which can complete the search for the target data, greatly reducing the consumption of read / write resources.

[0156]

[0163] Figure 12 shows a structural block diagram of a disk data index generation device provided in an embodiment of the present disclosure. Corresponding to the application scenario and method provided in the embodiment of the present disclosure, the present disclosure also provides a disk data index generation device. As shown in Figure 12, a structural block diagram of a disk data index generation device according to an embodiment of the present disclosure is provided. The device may include the following modules.

[0164] Generation module 1201 is used to receive the target data to be stored and generate key-value pairs of the target data. The key-value pairs include the target query key and the target query value of the target data. The target query key is used to find the corresponding target query value, and the target query value is used to indicate the target data. Writing module 1202 is connected to the generation module 1201 and is used to write the key-value pairs into the current unit memory location. After the unit memory location is full, the identifier of the unit memory location is extracted, and the lowest-level query key index of the multi-level data index is generated according to the identifier. Index module 1203 is connected to the writing module 1202 and is used to generate and compress a multi-level data index step by step based on the lowest-level query key index as the stored data increases. The multi-level data index includes multiple levels of query key indexes for finding the memory location to which the target query key belongs. The query key index is generated based on a binary prefix tree and is used to map the next level query key index or memory location.

[0157]

[0165] According to the disk data index generation apparatus of this embodiment, after receiving the target data to be stored and generating key-value pairs of the target data, the key-value pairs are written into the current unit memory location. After the unit memory location is full, the identifier of the unit memory location is extracted, and the lowest-level query key index of the multi-level data index is generated according to the identifier. As the stored data increases, the multi-level data index is generated level by level. The multi-level data index can efficiently compress the query key of the stored data, reduce the resource occupation of the data index, and establish a mapping relationship between the memory location of the stored data and the query key. Thus, the key-value pair corresponding to the query key of the target data to be searched can be quickly found through the multi-level data index, that is, the memory page of the target data in the disk.

[0158]

[0166] During querying, multi-level data indexes can be compressed and directly searched without decompressing the entire index, which not only improves query efficiency but also reduces resource consumption during queries. Furthermore, the lowest-level query key index of the multi-level data index directly maps to memory locations. During a search, only a single read / write operation is needed to obtain the memory location corresponding to the target data, thus greatly reducing the consumption of read / write resources. This solves the problem of low search efficiency associated with disk-based search methods in related technologies.

[0159]

[0167] The functions of each module in each device of the embodiments of this disclosure can be found in the corresponding description in the above method, and have corresponding beneficial effects, which will not be repeated here.

[0160]

[0168] FIG13 is a block diagram of an electronic device used to implement embodiments of the present disclosure. As shown in FIG13, the electronic device includes: a memory 1301 and a processor 1302. The memory 1301 stores a computer program that can run on the processor 1302. When the processor 1302 executes the computer program, it implements the methods in the above embodiments. The number of the memory 1301 and the processor 1302 can be one or more.

[0161]

[0169] The electronic device also includes: a communication interface 1303, used to communicate with external devices and perform data exchange and transmission.

[0162]

[0170] If the memory 1301, processor 1302, and communication interface 1303 are implemented independently, they can be interconnected via a bus to communicate with each other. This bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, only one thick line is used in Figure 13, but this does not mean that there is only one bus or one type of bus.

[0163]

[0171] Optionally, in a specific implementation, if the memory 1301, the processor 1302 and the communication interface 1303 are integrated on a single chip, the memory 1301, the processor 1302 and the communication interface 1303 can communicate with each other through the internal interface.

[0164]

[0172] This disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method provided in this disclosure.

[0165]

[0173] This disclosure also provides a chip, which includes a processor for calling and executing instructions stored in a memory, causing a communication device with the chip installed to perform the method provided in this disclosure.

[0174] This disclosure also provides a chip, including: an input interface, an output interface, a processor, and a memory. The input interface, output interface, processor, and memory are connected through an internal connection path. The processor is used to execute code in the memory. When the code is executed, the processor is used to perform the method provided in the application embodiment.

[0166]

[0175] It should be understood that the aforementioned processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor. It is worth noting that the processor may be a processor supporting Advanced Reduced Instruction Set Machines (ARM) architecture.

[0167]

[0176] Further, optionally, the above-mentioned memory may include read-only memory and random access memory. The memory may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. Non-volatile memory may include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available. Examples include Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced SDRAM (ESDRAM), Sync Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0168]

[0177] In the above embodiments, the implementation can be carried out entirely or partially by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the flow or function according to this disclosure is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0169]

[0178] In the description of this specification, references to the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0170]

[0179] Furthermore, the terms “first” and “second” are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as “first” or “second” may explicitly or implicitly include at least one of that feature. In the description of this disclosure, “a plurality of” means two or more, unless otherwise explicitly specified.

[0171]

[0180] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process. Furthermore, the scope of the preferred embodiments of this disclosure includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functionality involved.

[0172]

[0181] The logic and / or steps described in the flowchart or otherwise herein, for example, can be considered as a ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from an instruction execution system, apparatus or device).

[0173]

[0182] It should be understood that various parts of this disclosure can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. All or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware, the program being stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiments.

[0174]

[0183] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. This storage medium can be a read-only memory, a disk, or an optical disk, etc.

[0175]

[0184] The above description is merely an exemplary embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope described in this disclosure, and these should all be covered within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

Claims

Claims 1. A method for retrieving data on a disk, comprising: Receive the target query key for the target data to be searched; Based on the target query key, the memory location of the target query key is queried level by level in the multi-level data index. The multi-level data index is generated and compressed during data storage and includes multiple levels of query key indexes. These query key indexes are generated based on a binary prefix tree and are used to map to the next level query key index or the memory location. Based on the memory location, the stored key-value pairs are retrieved, and the target query value corresponding to the target query key is found. The memory location stores multiple key-value pairs, each including a query key and a corresponding query value. The query value indicates the corresponding data, and the target query value indicates the target data.

2. The method according to claim 1, wherein, Based on the target query key, the memory location of the target query key is queried level by level in the compressed multi-level data index, including: based on the target query key, searching for the highest-level query key index LN containing the target query key in the compressed multi-level data index, where N is the level of the multi-level data index; starting from the highest-level query key index LN, searching for the next-level query key index Lk-1 where the target query key is located in the previous-level query key index Lk, until the lowest-level query key index L1 is found, where each previous-level query key index Lk in the multi-level data index includes multiple next-level query key indices Lk-1, k C [1,N]; and finding the corresponding memory location based on the lowest-level query key index L1.

3. The method according to claim 2, wherein, Based on the target query key, in the compressed multi-level data index, the highest-level query key index LN containing the target query key is searched, including: if there is only one highest-level query key index LN, the next-level query key index LN-1 containing the target query key is searched within the highest-level query key index LN; if there are multiple highest-level query key indexes LN, where the multiple highest-level query key indexes LN record query keys for some data in chronological order, the next-level query key index LN-1 containing the target query key is searched within the highest-level query key indexes LN using a binary search or traversal method, wherein the number of the multiple highest-level query key indexes LN does not exceed a first preset number.

4. The method according to claim 2, wherein, The process involves searching for the next-level query key index Lk-1 containing the target query key in the parent query key index Lk, continuing until the lowest-level query key index L1 is found. This includes: if the next-level query key index Lk-1 in the parent query key index Lk records a query key, then searching for the next-level query key index Lk-1 containing the target query key based on the number of next-level query key indexes Lk-1; if the next-level query key index Lk-1 in the parent query key index Lk records a memory location, then searching for the next-level query key index Lk-1 containing the target query key based on the attribute information of the parent query key index Lk and the prefix tree. The attribute information includes information related to the generation of the prefix tree for the parent query key index Lk. The prefix tree records the data of the next-level query key index Lk-1 recorded in the order of their occurrence when the parent query key index Lk was generated. The prefix tree is generated from left to right based on the data order of the query keys of the multiple next-level query key indexes Lk-1 recorded by the parent query key index Lk. By querying level by level, the lowest-level query key index L1 containing the target query key is found.

5. The method according to claim 4, wherein, If a query key is recorded in the next-level query key index Lk-1 within the parent query key index Lk, the next-level query key index Lk-1 containing the target query key is queried based on the number of next-level query key indexes Lk-1. This includes: if the number of next-level query key indexes Lk-1 within the parent query key index Lk does not exceed a second preset number, searching for the next-level query key index Lk-1 containing the target query key using a binary search method. Here, the multiple next-level query key indices of the highest-level query key index record the compressed query key (with common prefixes removed) in ascending order. If the number of next-level query key indexes Lk-1 within the parent query key index Lk exceeds a second preset number, the target query key is searched using the attribute information of the parent query key index Lk and the prefix tree. The next level query key index Lk-L contains the key.

6. The method according to claim 5, wherein, If the next-level query key index Lk-1 in the parent query key index Lk has a memory location, the next-level query key index Lk-1 of the target query key is found using the attribute information of the parent query key index Lk and the prefix tree. This includes: obtaining the direct lookup jump table of the prefix tree from the attribute information of the parent query key index Lk, wherein the direct lookup jump table records the jump data of different subtrees in the historical queries of the prefix tree; during the search for the target query key, if a corresponding subtree jump record is found, a jump is performed directly through the subtree jump record in the direct lookup jump table; if no corresponding subtree jump record is found, the prefix tree is traversed until the leaf node corresponding to the target query key is found; and the next-level query key index Lk-1 to which the target query key belongs is determined based on the leaf node.

7. The method according to claim 2, wherein, Locating the corresponding memory location based on the lowest-level query key index L1 includes: Based on the attribute information of the lowest-level query key index L1 and the prefix tree, querying the memory location to which the target query key belongs within the lowest-level query key index L1. The attribute information of the lowest-level query key index L1 records the maximum or minimum query key of the mapped memory page, serving as the basis for querying the query key. The query result of the target query key in the prefix tree is either the basis for searching the target memory page or a range of search results for multiple adjacent memory pages. The memory location is stored in units of memory pages. If the query result indicates that the target query key belongs to the target memory page or multiple adjacent memory pages, the target memory page or multiple adjacent memory pages are used as the corresponding memory location.

8. The method according to claim 7, wherein, Obtaining the key-value pairs of the memory page and searching for the target query value corresponding to the target query key includes: loading the corresponding memory page from the disk storing data through a single read / write operation, obtaining the key-value pairs in the corresponding memory page, wherein the memory page is the data storage structure of the disk; and searching for the target query value corresponding to the target query key in the key-value pairs of the corresponding memory page using a binary search method.

9. A method for generating a disk data index, comprising: The system receives target data to be stored, generates key-value pairs for the target data, wherein each key-value pair includes a target query key and a target query value, the target query key is used to find the corresponding target query value, and the target query value is used to indicate the target data; the key-value pairs are written into the current unit memory location; after the unit memory location is full, the identifier of the unit memory location is extracted, and the lowest-level query key index of the multi-level data index is generated based on the identifier; as the stored data increases, the multi-level data index is generated and compressed level by level based on the lowest-level query key index, wherein the multi-level data index includes multiple levels of query key indexes for finding the memory location to which the target query key belongs, and the query key index is generated based on a binary prefix tree and is used to map the next level query key index or the memory location.

10. The method of claim 9, wherein, The process involves writing the key-value pairs into the current unit memory location. After the unit memory location is full, the identifier of the unit memory location is written into the lowest-level query key index L1 of the multi-level data index. This includes: writing the key-value pairs into the current unit memory location; after the unit memory location is full, extracting the largest or smallest query key from the unit memory location and generating a prefix tree for the lowest-level query key index L1 based on the largest or smallest query key; when the number of unit memory locations corresponding to the prefix tree of the same lowest-level query key index L1 reaches a third preset number, generating the lowest-level query key index L1 based on the prefix tree and saving the corresponding attribute information, wherein the attribute information includes relevant information for generating the prefix tree of the next-level query key index Lk, and the prefix tree records the next-level query key index Lk-1 arranged in size order; creating a new lowest-level query key index L1 and generating the corresponding prefix tree, and continuing to process the newly stored data. deal with.

11. The method of claim 10, wherein, As the stored data increases, the multi-level data index is generated progressively, including: as the stored data increases, the next-level query key index Lk-1 gradually accumulates; when the next-level query key index Lk-1 reaches a fourth preset number allowed by the previous-level query key index Lk, the corresponding previous-level query key index Lk is generated, and the corresponding attribute information is saved; until the highest-level query key index LN is generated, wherein the number of the highest-level query key index LN does not exceed a fifth preset number.

12. The method according to claim 9, wherein, The method further includes: determining the common prefix of the query keys corresponding to the query key indexes at each level, and pruning the common prefix to retain the identifier part of the query key; creating a binary prefix tree of the identifier part of the query keys of the stored data; compressing the binary prefix tree into a compact array using a compact recursive representation; and compressing the compact array to obtain a prefix tree, wherein the prefix tree is used by the multi-level data index to find the query key indexes at each level to which the target query key belongs.

13. An electronic device comprising a memory, a processor, and a computer program stored in the memory, wherein the processor, when executing the computer program, implements the method of any one of claims 1 to 12.

14. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the method of any one of claims 1 to 12.

15. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 12.

Images