Method for searching data in disk and disk data index generation method

Abstract

The invention provides a method for searching for data in a disk and a disk data index generation method. The method comprises the steps that a target query key of target data needing to be searched for is received; according to the target query key, the memory position of the target query key is queried step by step in a multi-level data index, the multi-level data index is generated and compressed during data storage and comprises query key indexes of multiple levels, and the query key indexes are generated based on a binary prefix tree and used for mapping the next-level query key index or the memory position; the stored key value pair is obtained based on the memory position, a target query value corresponding to the target query key is searched, and the target query value is used for indicating the target data. According to the embodiment of the invention, the speed of searching the disk data can be improved, and when the data volume is relatively large, the query can be effectively and quickly carried out, so that better query performance is provided.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data storage, and particularly relates to a method for searching data in a disk and a disk data index generation method. BACKGROUND

[0002] A disk stores data in fixed-size data blocks, and the data block is the minimum read-write unit of the disk. The data block can be regarded as an array, each data block has a unique id, and specific data is stored in each data block. The data is arranged in order according to its key (query key key) without repetition. A core problem of disk retrieval is how to determine the data block id where the query key key is located.

[0003] In the related art, a direct search method is usually used for searching, and this method has a particularly long search time when the data volume is large. Therefore, the related art uses mathematical methods to improve the search efficiency, for example, binary search, which can improve the search efficiency to a certain extent, but in the big data scenario, the limitations of this method are still great, and the required data cannot be searched in time and effectively.

[0004] Therefore, the disk search method in the related art has the problem of low search efficiency. SUMMARY

[0005] Embodiments of the present application provide a method for searching data in a disk and a disk data index generation method to solve one or more technical problems described above.

[0006] In a first aspect, the embodiments of the present application provide a method for searching data in a disk, comprising: receiving a target query key of target data to be searched; querying the memory location of the target query key level by level in a multi-level data index according to the target query key, wherein the multi-level data index is generated and compressed when data is stored, and comprises a plurality of levels of query key indexes, the query key index is generated based on a binary prefix tree, and is used to map a next-level query key index or the memory location; obtaining a stored key-value pair based on the memory location to search a target query value corresponding to the target query key, wherein the memory location stores a plurality of key-value pairs, each key-value pair comprises a query key and a query value corresponding to the query key, the query value is used to indicate corresponding data, and the target query value is used to indicate the target data.

[0007] In a second aspect, the embodiments of the present application provide a disk data index generation method, comprising: receiving target data to be stored, and generating a key-value pair of the target data, wherein the key-value pair comprises a target query key and a target query value of the target data, the target query key is used to find the corresponding target query value, and the target query value is used to indicate the target data; writing the key-value pair into a current unit memory location, extracting an identifier of the unit memory location after the unit memory location is full, and generating a lowest-level query key index of a multi-level data index according to the identifier; with the increase of the stored data, generating and compressing the multi-level data index level by level based on the lowest-level query key index, wherein the multi-level data index comprises a plurality of levels of query key indexes used to find the unit memory location to which the target query key belongs, the query key index is generated based on a binary prefix tree, and is used to map a next-level query key index or the memory location.

[0008] In a third aspect, the embodiments of the present application provide an electronic device, comprising a memory, a processor, and a computer program stored in the memory, and the processor implements the method of any of the embodiments of the present application when executing the computer program.

[0009] In a fourth aspect, the embodiments of the present application provide a computer readable storage medium, and the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method of any of the embodiments of the present application.

[0010] In a fifth aspect, the embodiments of the present application provide a computer program product, comprising a computer program, and the computer program is executed by a processor to implement the method of any of the embodiments of the present application.

[0011] The multi-level data index according to the embodiments of the present application can efficiently compress the query key of the stored data, reduce the occupied resources of the data index, and establish the mapping relationship between the memory location of the stored data and the query key, and then the multi-level data index can quickly find the key-value pair corresponding to the query key of the target data to be found, that is, the memory location of the target data in the disk. In the query, the multi-level data index can be compressed and directly searched, and it is not necessary to decompress all, which not only improves the query efficiency, but also reduces the occupied resources in the query. Moreover, the lowest-level query key index of the multi-level data index directly maps the memory location, and in the search, only one read-write operation is needed to obtain the memory page of the memory location corresponding to the target data, and the search of the target data can be completed, which greatly reduces the occupation of the read-write resources. Thus, the problem of low search efficiency in the related art disk search method is solved.

[0012] The above description is only a summary of the technical solutions of the present application. In order to make the technical means of the present application more clearly understood, the present application can be implemented according to the content of the description, and in order to make the above and other purposes, characteristics and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application are described below. BRIEF DESCRIPTION OF DRAWINGS

[0013] In the drawings, like reference numerals refer to same or similar functionalities throughout the several views. The drawings are not necessarily to scale. It is to be understood that the drawings only depict several embodiments in accordance with the present application and should not be considered as limiting the scope of the present application.

[0014] Figure 1 A schematic diagram of a binary search method of the related art is shown;

[0015] Figure 2 A flow chart of a method for searching data in a disk is shown;

[0016] Figure 3 A flow chart of a method for generating a disk data index is shown;

[0017] Figure 4 A schematic diagram of a binary index trie of an embodiment of the present application is shown;

[0018] Figure 5 A schematic diagram of a query key key of an embodiment of the present application is shown;

[0019] Figure 6 A schematic diagram of trie index compression and query principle of an embodiment of the present application is shown;

[0020] Figure 7 A schematic diagram of trie index jump of an embodiment of the present application is shown;

[0021] Figure 8 A schematic diagram of the influence of direct search jump table size on query performance of an embodiment of the present application is shown;

[0022] Figure 9 A schematic diagram of the overall architecture of a multi-level data index level-trie of an embodiment of the present application is shown;

[0023] Figure 10 A schematic diagram of memory occupancy analysis of a multi-level data index level-trie of an embodiment of the present application is shown;

[0024] Figure 11 A structural block diagram of a device for searching data in a disk is shown;

[0025] Figure 12 A structural block diagram of a disk data index generation apparatus provided in an embodiment of the present application is shown; and

[0026] Figure 13 A block diagram of an electronic device for implementing an embodiment of the present application is shown. DETAILED DESCRIPTION

[0027] In the following, only some exemplary embodiments are simply described. As can be appreciated by those skilled in the art, the described embodiments can be modified in various different ways without departing from the concept or scope of the present application. Therefore, the drawings and the description are considered to be exemplary in nature rather than limiting.

[0028] To facilitate understanding of the technical solutions of the embodiments of the present application, the related technologies of the embodiments of the present application are described below. The following related technologies can be combined with the technical solutions of the embodiments of the present application in any manner as optional solutions, and all of them belong to the protection scope of the embodiments of the present application.

[0029] In the related technologies, there are usually two indexes for evaluating the pros and cons of disk retrieval algorithms: 1. the speed of calculating the data block id given the query key key; and 2. the memory resource consumption. The goal of the retrieval algorithm is to improve the retrieval speed as much as possible while reducing the resource consumption of the process.

[0030] One of the simplest ways is to use binary search, as shown in Figure 1 Figure 1 A schematic diagram of the binary search method of the related technologies is shown. The disk data block is identified as a memory page page. The disk data is ordered, and for a given query key key, the corresponding data block is read multiple times through binary search, and finally the final data block can always be determined. However, this way involves multiple disk input / output I / O operations in the retrieval process of a key query key. Thus, it occupies more resources, resulting in poor search performance.

[0031] There is also a way to extract a representative query key key from each data block, such as selecting the minimum value or maximum value in each memory page page to establish a multi-level data index, forming a multi-way prefix tree, and the data retrieval is performed through the multi-way prefix tree layer by layer. This way requires at most the I / O number of the index layer number for one retrieval. Since the index data block is usually smaller than the original data, the index data block can be cached to further reduce the required I / O for retrieval.

[0032] ​But this way still has two shortcomings: 1. The index data block is established by extracting the minimum value / maximum value of each data block layer by layer. When the query key key is large, it will cause the index data block to occupy a large space. 2. When the data volume is large, the index data block may occupy a large storage space, and it is difficult to be cached in its entirety, so a single retrieval still needs to consume multiple I / Os.

[0033] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation portal for user to choose authorization or refusal.

[0034] The technical solutions of the present application and how the technical solutions of the present application solve the foregoing technical problems will be described in detail below with specific embodiments. Several specific embodiments listed can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0035] Figure 2 A flowchart of a method for data searching in a disk provided in an embodiment of the present application is shown, and the embodiment of the present application provides a data searching method, as shown in Figure 2 The flowchart of the method for data searching in the disk according to an embodiment of the present application is shown, and the method can include the following steps:

[0036] Step S201, receiving a target query key of target data to be searched;

[0037] Step S202, querying the memory location of the target query key in the multi-level data index according to the target query key, wherein the multi-level data index is generated and compressed when data is stored, including a plurality of levels of query key indexes, and the query key index is generated based on a binary prefix tree, and is used to map the next level query key index or memory location;

[0038] Step S203, obtaining the stored key-value pair based on the memory location to search the target query value corresponding to the target query key, wherein the memory location stores a plurality of key-value pairs, each key-value pair includes a query key and a query value corresponding to the query key, and the query value is used to indicate the corresponding data, and the target query value is used to indicate the target data.

[0039] According to the data searching method in the embodiment, after receiving the target query key of the target data to be searched, the target query key is searched in the multi-level data index by the compressed direct searching method to obtain the memory location corresponding to the target query key. The multi-level data index can compress the query keys of the stored data efficiently, reduce the resource occupation of the data index, and establish the mapping relationship between the memory location of the stored data and the query key. Thus, the key-value pair corresponding to the query key of the target data to be searched, i.e., the memory location of the target data in the disk, can be quickly searched through the multi-level data index.

[0040] The multi-level data index can be directly searched by the compressed direct searching method when searching, without decompression of all the data, which not only improves the searching efficiency, but also reduces the resource occupation during searching. Moreover, the lowest level query key index of the multi-level data index directly maps the memory location, and only one read-write operation is needed to obtain the memory page of the memory location corresponding to the target data, so that the searching of the target data can be completed, and the occupation of the read-write resource is greatly reduced.

[0041] Then, the key-value pair of the memory location is obtained, and the target query value corresponding to the target query key is searched, and the target query value can indicate the target data. Thus, the problem of low searching efficiency of the disk searching method in the related art is solved.

[0042] The target data is the data stored in the disk, and the data in the disk is stored in the unit of data block slot. The data block can be a memory page or other storage structure, and one data block can store multiple groups of data, and each group of data is stored in the form of key-value pair.

[0043] When searching for data, the corresponding key-value pair can be searched according to the target query key of the target data, the target query value of the target query key is determined, and the target data can be determined according to the target query value.

[0044] When searching for the target data, the target query key of the target data to be searched is received, and the target query key can be given by the party that issues the searching request. After receiving the target query key, the memory location to which the target query key belongs, i.e., the storage location of the corresponding key-value pair, i.e., the storage location of the target query value, is searched in the compressed multi-level data index by the compressed direct searching method.

[0045] The compression direct query is due to that the multi-level data index is compressed by a compression method that can support compression direct query, for example, integer compression. When the multi-level data index is queried level by level, the direct query jump table and the prefix tree of each level query key index can be relied on for fast query, and the query efficiency when the data amount is large can be improved to a high degree. Details are described later.

[0046] The multi-level data index is generated and compressed when data is stored, and includes multiple levels of query key indexes for finding the memory location where the target query key belongs. The lowest level query key index is used to map the memory location, and the memory location stores a plurality of key-value pairs, each of which includes a query key and a query value corresponding to the query key.

[0047] The multi-level data index establishes a mapping relationship between the memory location and the data key. Each level of the multi-level data index includes at least one query key index, and each upper level query key index includes a plurality of lower level query key indexes. Therefore, the higher the level of the query key index, the fewer the number of query key indexes, so that when compared with the target query key, the target query key can be queried level by level to gradually narrow the range of the target query key, and finally determine the memory location range corresponding to the target query key.

[0048] The memory location stores data in units of memory pages, and the data amount of a single memory page is small, which can be obtained by a single IO read-write operation. After the memory page is obtained, the corresponding key-value pair and the target query value are found according to the target query key.

[0049] That is, after the memory location where the target query key is located is determined in the multi-level data index, the key-value pair of the memory location is obtained, and the target query value corresponding to the target query key is found; the target data is determined according to the target query value, wherein the target query value is used to indicate the target data.

[0050] It should be noted that the memory location where the target query key determined by the multi-level data index may be a specific memory page, or the range of two adjacent memory pages, which is determined by the query method of the lowest level query key index in the multi-level data index.

[0051] The query method of the lowest level query key index is to query by prefix tree. The final query result of this method may be a specific memory page, or the range of several memory pages, and generally the range of two memory pages. The specific principle is described in detail later.

[0052] It should be noted that the key-value pair of the above data cannot be stored across disk data blocks. When providing a mapping of the query key to the data block, if the size of the disk data block is 4KB, the size of a single key-value pair cannot exceed 4KB. That is, the key-value pair cannot be stored across multiple data blocks. This limitation can be addressed by splitting a large key-value pair into a data part and an index part, the data block only stores the index part, and the data part is queried through a secondary index. The same is true for the memory page, and the key-value pair cannot be stored across the memory page.

[0053] The lowest level query key index of the above multi-level data index is the memory page mapping the memory location. In theory, it can also map the data block or other memory structure. When providing a mapping of the query key to the data block, a mapping of the query key to the key-value pair is directly established, and if the number of key-value pairs is relatively large, the index amount will be very large. A data block can store multiple key-value pairs, and if only a mapping of the key to the data block is established, the index amount will theoretically be much smaller. The above data block can be a memory page, that is, only a mapping of the query key to the memory page is established, and the index amount will be greatly reduced. Since the disk data has at least one I / O each time the data is queried. Since it is necessary to consume one I / O to read data from the disk, if a memory page contains multiple key-value pairs, a binary search can be performed in the read memory page to perform secondary positioning, so as to realize data query with one I / O.

[0054] In an embodiment, according to the target query key, the memory location of the target query key is queried in the compressed multi-level data index, comprising: according to the target query key, the highest level query key index L N containing the target query key is found in the compressed multi-level data index, and then starting from the highest level query key index L N , the next level query key index L k where the target query key is located is found in the upper level query key index L k-1 , until the lowest level query key index L1 is found, wherein each upper level query key index L k in the multi-level data index includes a plurality of next level query key indexes L k-1 , k∈[1,N]; and the corresponding memory location is found according to the lowest level query key index L1.

[0055] The number of the highest level query key index L N is generally small, and the highest level query key index L N to which the target query key belongs can be determined by comparing the query key recorded in the highest level query key index L N with the target query key. A certain search method can also be used to improve the search efficiency, but the effect of improving the efficiency is not obvious in theory.

[0056] For the query key index of the intermediate level, the upper level query key index Lk and the next level query key index L k-1 The quantitative relationship between them may not be fixed. For example, a second-level query key index may contain 1024 first-level query key indexes, while a third-level query key index may contain 4096 second-level query key indexes. This would lead to a situation where using the same approach would impact query speed. For different levels, appropriate query methods can be used to ensure query efficiency.

[0057] When querying the lowest-level query key index L1, a matching search method is also used to find the corresponding memory location, ensuring search efficiency. Thus, based on multi-level data indexes, the memory location of the target query key can be quickly located.

[0058] In one implementation, based on the target query key, the highest-level query key index L containing the target query key is located in the compressed multi-level data index. N This includes: in the highest-level query key index L N In the case of one, directly index L on the highest-level query key. N Find the next-level query key index L containing the target query key. N-1 ; In the highest-level query key index L N In the case of multiple, multiple top-level query key indexes L N The query keys for some data were recorded in chronological order. Using a binary search or traversal approach, the query key index L was used to find the query keys at the highest level. N Find the next-level query key index L containing the target query key. N-1 Among them, multiple top-level query key indexes L N The quantity shall not exceed the first preset quantity.

[0059] The highest-level query key index L N It can be one, in the highest-level query key index L N In the case of one, directly index L on the highest-level query key. N Find the next-level query key index L containing the target query key. N-1 The highest-level query key index L N It can also be viewed for multiple times, in the highest-level query key index L. N In the case of multiple, multiple top-level query key indexes L N The query keys for some data were recorded in chronological order. Using a binary search or traversal approach, the query key index L was used to find the query keys at the highest level. N Find the next-level query key index L containing the target query key. N-1 .

[0060] Multiple top-level query key indexes L NThe number of the second level query key index L2 is not more than the first preset number, so as to facilitate fast searching and improve the query efficiency of the query key. Preferably, the highest level query key index L0 of the embodiment has the number of the second level query key index L2 being 1, so that searching is not needed at the highest level and the next level is directly entered. N The number of the second level query key index L2 is 1, so that searching is not needed at the highest level and the next level is directly entered.

[0061] In an embodiment, searching for the next level query key index L2 where the target query key is located in the previous level query key index L1 is performed until the lowest level query key index L1 is found, including: searching for the next level query key index L2 where the target query key is located in the previous level query key index L1 according to the number of the next level query key index L2 in the case that the next level query key index L2 is recorded with the query key. k k-1 The number of the second level query key index L2 is not more than the first preset number, so as to facilitate fast searching and improve the query efficiency of the query key. Preferably, the highest level query key index L0 of the embodiment has the number of the second level query key index L2 being 1, so that searching is not needed at the highest level and the next level is directly entered. k k-1 The number of the second level query key index L2 is not more than the first preset number, so as to facilitate fast searching and improve the query efficiency of the query key. Preferably, the highest level query key index L0 of the embodiment has the number of the second level query key index L2 being 1, so that searching is not needed at the highest level and the next level is directly entered. k-1 The number of the second level query key index L2 is not more than the first preset number, so as to facilitate fast searching and improve the query efficiency of the query key. Preferably, the highest level query key index L0 of the embodiment has the number of the second level query key index L2 being 1, so that searching is not needed at the highest level and the next level is directly entered. k-1 The number of the second level query key index L2 is not more than the first preset number, so as to facilitate fast searching and improve the query efficiency of the query key. Preferably, the highest level query key index L0 of the embodiment has the number of the second level query key index L2 being 1, so that searching is not needed at the highest level and the next level is directly entered. k The number of the second level query key index L2 is not more than the first preset number, so as to facilitate fast searching and improve the query efficiency of the query key. Preferably, the highest level query key index L0 of the embodiment has the number of the second level query key index L2 being 1, so that searching is not needed at the highest level and the next level is directly entered. k-1 The number of the second level query key index L2 is not more than the first preset number, so as to facilitate fast searching and improve the query efficiency of the query key. Preferably, the highest level query key index L0 of the embodiment has the number of the second level query key index L2 being 1, so that searching is not needed at the highest level and the next level is directly entered. k The number of the second level query key index L2 is not more than the first preset number, so as to facilitate fast searching and improve the query efficiency of the query key. Preferably, the highest level query key index L0 of the embodiment has the number of the second level query key index L2 being 1, so that searching is not needed at the highest level and the next level is directly entered. k-1 The number of the second level query key index L2 is not more than the first preset number, so as to facilitate fast searching and improve the query efficiency of the query key. Preferably, the highest level query key index L0 of the embodiment has the number of the second level query key index L2 being 1, so that searching is not needed at the highest level and the next level is directly entered. k The number of the second level query key index L2 is not more than the first preset number, so as to facilitate fast searching and improve the query efficiency of the query key. Preferably, the highest level query key index L0 of the embodiment has the number of the second level query key index L2 being 1, so that searching is not needed at the highest level and the next level is directly entered. k The number of the second level query key index L2 is not more than the first preset number, so as to facilitate fast searching and improve the query efficiency of the query key. Preferably, the highest level query key index L0 of the embodiment has the number of the second level query key index L2 being 1, so that searching is not needed at the highest level and the next level is directly entered. k-1 The number of the second level query key index L2 is not more than the first preset number, so as to facilitate fast searching and improve the query efficiency of the query key. Preferably, the highest level query key index L0 of the embodiment has the number of the second level query key index L2 being 1, so that searching is not needed at the highest level and the next level is directly entered. k The number of the second level query key index L2 is not more than the first preset number, so as to facilitate fast searching and improve the query efficiency of the query key. Preferably, the highest level query key index L0 of the embodiment has the number of the second level query key index L2 being 1, so that searching is not needed at the highest level and the next level is directly entered. k-1 The number of the second level query key index L2 is not more than the first preset number, so as to facilitate fast searching and improve the query efficiency of the query key. Preferably, the highest level query key index L0 of the embodiment has the number of the second level query key index L2 being 1, so that searching is not needed at the highest level and the next level is directly entered.

[0062] The number of the second level query key index L2 is not more than the first preset number, so as to facilitate fast searching and improve the query efficiency of the query key. Preferably, the highest level query key index L0 of the embodiment has the number of the second level query key index L2 being 1, so that searching is not needed at the highest level and the next level is directly entered. k The number of the second level query key index L2 is not more than the first preset number, so as to facilitate fast searching and improve the query efficiency of the query key. Preferably, the highest level query key index L0 of the embodiment has the number of the second level query key index L2 being 1, so that searching is not needed at the highest level and the next level is directly entered. k-1 The number of the second level query key index L2 is not more than the first preset number, so as to facilitate fast searching and improve the query efficiency of the query key. Preferably, the highest level query key index L0 of the embodiment has the number of the second level query key index L2 being 1, so that searching is not needed at the highest level and the next level is directly entered. k-1 The number of the second level query key index L2 is not more than the first preset number, so as to facilitate fast searching and improve the query efficiency of the query key. Preferably, the highest level query key index L0 of the embodiment has the number of the second level query key index L2 being 1, so that searching is not needed at the highest level and the next level is directly entered. k-1 The number of the second level query key index L2 is not more than the first preset number, so as to facilitate fast searching and improve the query efficiency of the query key. Preferably, the highest level query key index L0 of the embodiment has the number of the second level query key index L2 being 1, so that searching is not needed at the highest level and the next level is directly entered.

[0063] The number of the second level query key index L2 is not more than the first preset number, so as to facilitate fast searching and improve the query efficiency of the query key. Preferably, the highest level query key index L0 of the embodiment has the number of the second level query key index L2 being 1, so that searching is not needed at the highest level and the next level is directly entered. k The number of the second level query key index L2 is not more than the first preset number, so as to facilitate fast searching and improve the query efficiency of the query key. Preferably, the highest level query key index L0 of the embodiment has the number of the second level query key index L2 being 1, so that searching is not needed at the highest level and the next level is directly entered. k-1 The number of the second level query​​k the attribute information and the prefix tree of the upper level query key index L k-1 , which is considering that the memory location is summarized to the lowest level query key index, is established by mapping the corresponding maximum / minimum query key, so in the multi-level data index, the memory location can also be indicated by the query key of the record, and the number of memory is often large, so it is necessary to query by using the attribute information and the prefix tree.

[0064] The attribute information includes the bit stream bit_stream of the upper level query key index L k , that is, the related information of the prefix tree, which records the data of the lower level query key index L k in the order of the first and the second when generating the upper level query key index L k-1 . The prefix tree is generated from left to right according to the data order of the query keys of the multiple lower level query key indexes L k recorded by the upper level query key index L k-1 .

[0065] Here, the attribute information and the prefix tree of the upper level query key index L k are used to query the lower level query key index L k-1 where the target query key is located, which can quickly query the prefix tree by using the direct jump table corresponding to the prefix tree in the attribute information, so as to realize the quick direct query of the compressed multi-level data index, that is, the above-mentioned compressed direct query of the multi-level data index.

[0066] It should be noted that the prefix tree records the data of the lower level query key index L k in the order of the first and the second when generating the upper level query key index L k-1 , and the order here is the generation order of the lower level query key index L k-1 . The lower level query key index L k-1 records the query key by binary data, and the lower level query key index L k-1 generated first is stored on the left, and the query keys of the lower level query key indexes L k-1 generated in sequence are recorded from left to right, that is, the prefix tree is generated from left to right according to the data order of the query keys of the multiple lower level query key indexes L k recorded by the upper level query key index L k-1 .

[0067] That is, the query key index of each level is generated in an ordered partition, so that when the number of query key indexes of the next level is small, the binary search method can be used to quickly find the query key index where the target query key is located. In combination with the above-mentioned direct jump table and prefix tree searching method, the target query key index where the target query key is located can be quickly found when the data volume is large. Further, the target query value corresponding to the target query key can be quickly and efficiently found by using the multi-level data index.

[0068] According to the specific situation of the specific level, the lowest level query key index L1 where the target query key is located is found by means of step-by-step query.

[0069] In an embodiment, in the next level query key index L k in the upper level query key index L k-1 , the next level query key index L k-1 where the target query key is located is recorded when the query key is recorded, according to the number of next level query key indexes L k-1 , including: in the case where the number of next level query key indexes L k in the upper level query key index L k-1 does not exceed the second preset number, the next level query key index L k-1 where the target query key is located is found by binary search method, wherein the plurality of next level query key indexes of the highest level query key index record the compressed query keys without common prefix in size order; in the case where the number of next level query key indexes L k in the upper level query key index L k-1 exceeds the second preset number, the next level query key index L k where the target query key is located is found by the attribute information of the upper level query key index L k- 1.

[0070] In the case where the number of next level query key indexes L k in the upper level query index L k-1 does not exceed the second preset number, for example, the second preset number is 10, the next level query key index L k-1 where the target query key is located is found by binary search method. It should be noted that the plurality of next level query key indexes L k-1 of the highest level query key index record the compressed query keys without common prefix in size order. The corresponding query key index can be quickly found by using simple binary search method or direct search method.

[0071] In the case where the number of next level query key indexes L k in the upper level query index L k-1If the number exceeds the second preset number, it indicates that a simple search method may slow down the search speed. In this case, the search should be performed using the index L from the previous query key. k Using attribute information and the prefix tree, find the next-level query key index L containing the target query key. k-1 This ensures faster search speeds.

[0072] In one implementation, the query key index L is used in the higher-level query key index. k The next-level query key index L in k-1 If the record has a memory location, use the lookup key L from the previous level. k Using attribute information and the prefix tree, find the next-level query key index L containing the target query key. k-1 This includes: the query key index L at the previous level. k The direct lookup table of the prefix tree is obtained from the attribute information. The direct lookup table records the jump data of different subtrees in the historical queries of the prefix tree. During the search for the target query key, if a corresponding subtree jump record is found, the jump is performed directly through the subtree jump record in the direct lookup table. If no corresponding subtree jump record is found, the prefix tree is traversed until the leaf node corresponding to the target query key is found. The next-level query key index to which the target query key belongs is determined based on the leaf node.

[0073] The lookup method of a prefix tree requires node searching, which can affect the search speed when the number of nodes in the prefix tree is large. Therefore, this embodiment provides a direct lookup jump table for the prefix tree, which records the jump data between different subtrees during historical queries of some nodes in the prefix tree.

[0074] During the traversal of the prefix tree, when passing through nodes recorded in the direct lookup jump table, fast jumps can be achieved based on the jump information recorded in the table. In this embodiment, some nodes recorded in the direct lookup jump table can be high-frequency nodes in the prefix tree, that is, nodes with a high probability of being traversed. The direct lookup jump table records jump information of subtrees closer to the root node in the prefix tree as much as possible. This is because, generally, the closer to the root node, the more leaf nodes it covers, and the higher the jump efficiency.

[0075] In the process of finding the target query key, the direct lookup jump table of the prefix tree is first obtained. The direct lookup jump table is then searched to see if a corresponding subtree jump record exists. If a corresponding subtree jump record is found, the jump is performed directly through the subtree jump record in the direct lookup jump table, which greatly improves the query efficiency.

[0076] If the corresponding sub-tree jump record cannot be found, the prefix tree is directly traversed until the leaf node corresponding to the target query key is found. Then, the next level query key index to which the target query key belongs is determined according to the leaf node.

[0077] The direct jump table can be implemented by an acceleration plug-in for fast jump. The acceleration plug-in can be a hint, which is recorded in attribute information and in the corresponding query key index.

[0078] In an embodiment, according to the lowest level query key index L1, the corresponding memory location is found, including: according to the attribute information of the lowest level query key index L1 and the prefix tree, in the lowest level query key index L1, the memory location to which the target query key belongs is queried, wherein the maximum query key or the minimum query key of the mapped memory page is recorded in the attribute information of the lowest level query key index L1 as the basis for query key search, the query result of the target query key in the prefix tree is the basis for search of the target memory page, or the basis for search of the range of multiple adjacent memory pages, and the memory location is stored in units of memory pages; in the case where the query result clearly indicates that the target query key belongs to the target memory page or multiple adjacent memory pages, the target memory page or multiple adjacent memory pages are taken as the corresponding memory location.

[0079] Since the maximum / minimum query key is used to indicate the corresponding query key range when generating the multi-level data index according to the memory page, when searching for the memory page of the target query key in the multi-level data index, according to the attribute information of the lowest level query key index L1 and the prefix tree, in the lowest level query key index L1, the memory page to which the target query key belongs is queried.

[0080] The above multi-level data index is a mapping of query keys and memory pages, and one memory page often includes multiple query keys. When querying by using the prefix tree, the final query result can be the basis for search of a specific target memory page, or the basis for search of the range of two adjacent memory pages.

[0081] In the case where the query result clearly indicates that the target query key belongs to the target memory page or two adjacent memory pages, the target memory page or two adjacent memory pages are taken as the corresponding memory page. The corresponding memory page is the finally determined range of key-value pairs to which the target query key belongs, that is, the range of memory pages that save the key-value pairs.

[0082] In an embodiment, the key-value pairs of the memory page are obtained, and the target query value corresponding to the target query key is found, including: the corresponding memory page is loaded from the disk storing data through a single read-write operation, and the key-value pairs in the corresponding memory page are obtained, wherein the memory page is a data storage structure of the disk; and the target query value corresponding to the target query key is found in the key-value pairs of the corresponding memory page through binary search.

[0083] The multi-level data index is stored in the memory, the data in the form of key-value pairs is stored in the disk, and the memory page is also a storage structure of the disk. The significance of the multi-level data index is to determine a small enough memory page range according to a given target query key, and then use one I / O operation to obtain the corresponding memory page to the memory, and perform query and comparison in the memory to determine the key-value pair matched with the target query key and the corresponding target query value. Compared with the prior art, the least disk IO resources are occupied, and the influence of the query operation on the disk work is minimized.

[0084] Figure 3 A flowchart of a disk data index generation method provided in an embodiment of the application is shown, and the application also provides a disk data index generation method. As shown in Figure 3 The flowchart of the disk data index generation method of one embodiment of the application is shown, and the method can include the following steps:

[0085] In step S301, the target data to be stored is received, and the key-value pair of the target data is generated, wherein the key-value pair includes the target query key and the target query value of the target data, the target query key is used to find the corresponding target query value, and the target query value is used to indicate the target data;

[0086] In step S302, the key-value pair is written into the current unit memory location, and after the unit memory location is full, the identifier of the unit memory location is extracted, and the lowest level query key index of the multi-level data index is generated according to the identifier;

[0087] In step S303, as the stored data increases, the lowest level query key index is used to generate and compress the multi-level data index level by level, wherein the multi-level data index includes a plurality of levels of query key indexes used to find the memory location to which the target query key belongs, and the query key index is generated based on a binary prefix tree and is used to map the next level query key index or the memory location.

[0088] According to the above disk data index generation method of the embodiment, after receiving the target data to be stored and generating the key-value pair of the target data, the key-value pair is written into the current unit memory location, and after the unit memory location is full, the identifier of the unit memory location is extracted, and the lowest level query key index of the multi-level data index is generated according to the page identifier; and as the stored data increases, the multi-level data index is generated level by level. The multi-level data index can efficiently compress the query key of the stored data, reduce the occupied resources of the data index, and establish the mapping relationship between the memory location of the stored data and the query key, so that the corresponding key-value pair of the query key of the target data to be searched can be quickly found through the multi-level data index, that is, the memory location of the target data in the disk.

[0089] During queries, multi-level data indexes can be compressed and directly accessed without decompressing the entire index, improving query efficiency and reducing resource consumption. Furthermore, the lowest-level query key index of the multi-level data index directly maps to memory locations. During a search, only a single read / write operation is needed to retrieve the memory location of the target data, significantly reducing read / write resource consumption. This solves the problem of low search efficiency inherent in disk-based search methods in related technologies.

[0090] The process involves receiving the target data to be stored, generating key-value pairs for the target data, including a target query key and a target query value. The target query key is used to locate the corresponding target data, and the target query value indicates the memory location of the target data. Then, the target data is stored.

[0091] In other words, key-value pairs are written to the current unit memory location, where the capacity is fixed, such as a memory page. After the unit memory location is full, the identifier of the unit memory location is written to the lowest-level query key index of the multi-level data index. The amount of data in the query key index of the multi-level data index is also fixed, and the multi-level data index is generated level by level as the stored data increases.

[0092] After multi-level data indexes are generated, they can be compressed. The compression method supports compressed direct lookup, ensuring fast querying while consuming less storage and computing resources. The compression method will be explained in detail later.

[0093] In one implementation, key-value pairs are written to the current unit memory location. After the unit memory location is full, the identifier of the unit memory location is written to the lowest-level query key index of the multi-level data index. This includes: writing key-value pairs to the current unit memory location; after the unit memory location is full, extracting the largest / smallest query key in the unit memory location; generating a prefix tree of the lowest-level query key index L1 based on the largest / smallest query key; when the number of unit memory locations corresponding to the prefix tree of the same lowest-level query key index L1 reaches a third preset number, generating the lowest-level query key index L1 based on the prefix tree and saving the corresponding attribute information, wherein the attribute information includes the generation of the previous-level query key index L. k The prefix tree contains information about the next-level query key index L, which is arranged in ascending order. k-1 Create a new lowest-level query key index L1 and generate the corresponding prefix tree, then continue processing the newly stored data.

[0094] The aforementioned prefix tree can exist in the form of a bit stream, and the data recorded in the bit stream is also an array of prefix trees.

[0095] The third preset number is also a minimum unit of memory location allowed to be summarized by the lowest level query key index L1, that is, the maximum number of memory pages, such as 1024, 2048, 4096, and the like. It should be noted that since the length of the query key may be different, the memory page allowed to be summarized by the lowest level query key index can be limited from the perspective of data volume, for example, 4KB.

[0096] As shown in Figure 9 , the attribute information can be off, len and skip of the 0-level bucket L0_bucket for locating the starting position of the bit_stream; key_off and key_sz for recording the position of the maximum query key in the query key array. hints for accelerating the search of the bit_stream, that is, through hints, the entire bit_stream does not need to be decompressed during the search. The size thereof can be adjusted according to actual business requirements.

[0097] That is, the attribute information includes parameters for locating the starting position of the bit_stream, and recording the position of the maximum query key in the query key array. According to the above information, it can be found out which next level query key index L k-1 belongs to the target query key.

[0098] In an embodiment, as the stored data increases, the next level query key index L k-1 is gradually accumulated, and when the next level query key index L k-1 reaches a fourth preset number allowed by the upper level query key index L k , the corresponding upper level query key index L k is generated and the corresponding attribute information is saved; until the highest level query key index L N is generated, wherein the number of the highest level query key index L N does not exceed a fifth preset number.

[0099] The fourth preset number can be multiple, which refers to the number of the next level query key index L k allowed to be summarized by each upper level query key index L k-1 . It should be noted that the number of the next level query key index L k allowed to be summarized by each upper level query key index L k-1 may be different or the same. For example Figure 9 , as shown in the figure, a 2-level bucket L2_bucket can accommodate 4096 1-level buckets L1_bucket, and a 1-level bucket L1_bucket can accommodate 4096 0-level buckets L0_bucket.

[0100] In an embodiment, the method further comprises: determining a common prefix of the query key corresponding to each level of the query key index, and performing common prefix pruning to retain an identification part of the query key; creating a binary prefix tree of the identification part of the query key of the stored data; compressing the binary prefix tree into a compact array by means of compact recursive representation; and compressing based on the compact array to obtain a prefix tree, wherein the prefix tree is used for multi-level data index to find a query key belonging to each level of the query key index.

[0101] When generating the multi-level data index, the prefix tree can be generated according to the query key, which is used as a basis for subsequent queries. It should be noted that the memory page only provides the maximum / minimum query key, and a lowest level query key index contains a large number of query keys, each of which represents the maximum / minimum query key of a memory page. When the number is large, the general query method will fail, and the prefix tree query needs to be used.

[0102] The establishment of the prefix tree is to generate the prefix tree based on the query key recorded in a certain query key index. The generation method and principle of the prefix tree for other levels of query key indexes in the multi-level data index are similar. The upper level query key index is generated by induction from multiple lower level query key indexes, and each lower level query key index can provide the maximum / minimum query key representing the range of the corresponding query key.

[0103] The above prefix tree physically exists in the form of the bit stream bit_stream.

[0104] In an embodiment, the prefix tree is obtained by compressing the compact array, comprising: constructing a Huffman tree according to the compact array to determine the value of each element; in a case where the value of the element of the compact array exceeds a sixth preset number, compressing the compact array by means of Huffman compression; and in a case where the value of the element of the compact array does not exceed the sixth preset number, compressing the compact array by means of preset integer compression, wherein the preset integer compression is a non-Huffman compression.

[0105] Although the compact array is only an array, the memory occupied by the compact array is still large. Each intermediate node must occupy a position in the array. Therefore, integer compression technology is used for compression. Each element in the compact array is an unsigned integer. The value of the element as a whole presents a binomial distribution feature, and a Huffman tree can be directly constructed according to the data distribution feature without sampling the data in the compact array.

[0106] However, the Huffman decompression process can be lengthy if there is too much data. Thus, data with a higher frequency of occurrence can be compressed using Huffman compression, while data with a lower frequency of occurrence can be directly compressed using integer compression, such as gamma coding.

[0107] The closer the intermediate node is to the root node, the lower the frequency of occurrence of the intermediate node; similarly, the closer the intermediate node is to the leaf node, the higher the frequency of occurrence of the intermediate node. A threshold value can be set, such as a specific value of an element in the array <= 16 using Huffman compression; a specific value of an element > 16 using gamma compression.

[0108] It should be noted that the embodiment also provides an optional implementation, which proposes a new high-compression multi-level data index Level-Trie. By extracting the maximum value of each data block and compressing it into high-density binary search data, the multi-level data index has the following three characteristics: 1. Extremely low memory usage. For any type of data, the average extraction query key key of each data block only occupies 3 bits of memory, which can be loaded into memory at one time. 2. Extremely fast generation speed. A single thread can theoretically support 34GB / s of disk I / O on average, which is much higher than the existing hardware limit. 3. Compressible direct search. Without decompression, it can be directly queried. In O(1) time, the data block id corresponding to the given query key key can be calculated.

[0109] The Level-Trie proposed in the embodiment is a high-level data structure for disk retrieval, which is improved based on related technologies. In related technologies, SILT is a memory-oriented generation algorithm for computing minimum perfect hash, which has high compression efficiency, but does not support compressed direct search, and the query efficiency will be very slow when the data volume is large. SuRF is a compressible direct search range filter. SuRF is also a kind of index trie, and is constructed in byte units to build a prefix index, so the space occupied is relatively large, usually about 30%-50% of the original data. COIndex is an improvement of SuRF, which is used to provide a mapping from the query key key to the data block id, and is used in ToplingDB and TerakDB. However, the problem of large space occupation of SuRF has not been solved, and it is difficult to load it into memory.

[0110] SILT is a binary prefix index, which has high compression efficiency and fast generation speed. However, SILT cannot support direct lookup and has slow query speed when the data volume is large. SuRF is an in-memory range filter, which is essentially a byte prefix index and can support direct lookup. However, SuRF usually occupies a large amount of memory and cannot be directly used for disk retrieval. That is, SuRF is originally used to determine whether a given range exists, and cannot map a given query key to a data block id. The Level-Trie of the embodiment inherits the high compression rate and fast generation speed of the binary prefix index of SILT, supports direct lookup, and proposes multiple optimizations for disk retrieval, so that the Level-Trie can still provide adjustable query performance even when the data volume is large.

[0111] The Level-Trie proposes the following three targeted optimization strategies to address the following three shortcomings of SILT: 1. does not support direct lookup; 2. the query speed is slower when the data volume is larger; and 3. does not support disk retrieval. 1. A direct lookup skip list is introduced to realize direct jump query without consuming a large amount of CPU resources for decompression operation. 2. The data is partitioned in order, so that the compression block size is fixed, and thus the cost of index retrieval is generally constant and independent of the data volume. 3. A multi-level index and sampling key mechanism is introduced to establish a mapping relationship between the index trie and the disk data block, and to convert the calculation of the data block id into the query process of the index trie.

[0112] The compressed index trie is essentially a binary prefix tree. The index trie is indexed by prefix, and has three main advantages: (1) only the minimum distinguishable prefix needs to be recorded to uniquely index the given query key; (2) the common prefix can be skipped; and (3) the tree structure is naturally ordered. Assuming that a binary index trie is established for seven data, the final structure is as shown in Figure 4 . Figure 4 The figure shows the schematic diagram of the binary index trie of the embodiment. If the data is of the uint16 type, the common 0 in the high bit can be directly skipped and does not need to be recorded. In this example, the first 11 bits of the two query keys key are 0, so skip_bits = 11, that is, the first 11 bits in the high bit are skipped during query.

[0113] The query key key can be stored in big-endian order, so for a given query key key, the common prefix can be skipped first, and then the binary sequence of the query key key is matched according to the prefix. When the leaf node is reached, the corresponding data can be queried. The trie index does not need to record the binary sequence of the entire query key key, as shown in Figure 5 . Figure 5The schematic diagram of the query key key of the example of the embodiment of the application is shown, and only the minimum distinguishable prefix needs to be recorded, such as data 2, and only 01 needs to be recorded to uniquely match.

[0114] If the generated trie index is directly stored, there will be a large memory overhead. For example, each intermediate node needs to record the pointers of the left and right child nodes; the leaf node needs to record the corresponding value. Secondly, this structure is not conducive to data structure serialization and deserialization, which is very important for the fast generation and fast start of the index. In addition, the trie index may have false queries for query keys key with the same public prefix, for example, two binary sequences are queried in the above trie index and both return 3, so the index needs to provide detection and correction for the false query logic.

[0115] The compact recursive representation refers to the equivalent conversion of the general trie index in the above to another compact data structure, such as an array, so that it can be more convenient to persist and load from the disk. The compact form of the trie can be represented using the following formula:

[0116] repr(T) := |L|repr(L)repr(R)

[0117] The meaning is that, for a compressed trie T, L represents the left subtree of T; similarly, R represents the right subtree of T; |L| represents the number of leaf nodes contained in the trie L. In this way, the general trie can be converted into a trie with leaf node number information.

[0118] Figure 6 The schematic diagram of the trie index compression and query principle of the example of the embodiment of the application is shown, as shown in Figure 6 The (a) figure in the figure represents the initial general trie index, and the leaf node number information of the trie index is obtained by statistics, and the form with intermediate node information in the (b) figure is obtained, and then the in-order traversal of the trie index in the (b) figure is performed to obtain the compact form in the (c) figure. The (c) figure is logically equivalent to the (a) figure. For example, the query 10010 is shown in the (d) figure to be the query path on the general trie index; the highest bit is 1, so the right subtree is queried; the second bit is 0, so the left subtree is queried; the third highest bit is 0, so the left subtree is continuously queried, and then the leaf node is reached, and the value is 3.

[0119] For (e) figure, its query logic is: first look at the root node (the first in the array), which is 3, a non-leaf node; query the highest bit of the key key, which is 1, so the right subtree should be queried, then skip the left node, at this time the right subtree is 3; the node is a non-leaf node, and the second highest bit of the query key key is 0, so the left subtree needs to be queried, at this time it is still a non-leaf node; read the third highest bit as 0, and need to query the left subtree, but its left subtree is a leaf node (1 indicates that its left subtree is a leaf node); thus it can be known that the position of the leaf node is 3 (root) + 1 (left) = 4, that is, the position of the disk data is the fourth. Query the data on the disk to know that the query value value corresponding to the given query key key is 3.

[0120] An efficient algorithm is needed for the equivalent conversion from (a) figure to (c) figure. Given a sorted data set, it does not need to be converted in the order of the above (a) (b) (c) figures. The generation of the structure in (c) figure can be directly generated by scanning the ordered data set. In essence, it is a quicksort strategy. Given an ordered data set T, first scan the first bit of the query key key in T, and mark the bit = 0 as the left subtree band generated data set, and the bit = 1 as the right subtree band generated data set. Then recursively generate the array representation of each subtree, and each recursion layer increases the scanned bit depth by one.

[0121] From the above algorithm, it can be seen that the compact array is generated in the order from left to right. This order is very important, which indicates that the compressed trie is generated in a streaming manner, and the data block can be generated while writing data, and the index can be generated at the same time. There is no need to wait for the data to be scanned completely before generating the index.

[0122] The data query is relatively simple. Given the query key key and the compact representation array trepr, recursively query the left subtree and the right subtree of repr according to the bit of the query key key. If the bit corresponding to the query key key is 1, it indicates that the right subtree needs to be queried. But querying the right subtree needs to know how many intermediate nodes the left subtree has, so as to directly jump to the node corresponding to the right subtree. This needs to recursively access all the intermediate nodes of the left subtree and determine the number of them.

[0123] In the recursive query, if a leaf node is reached, the query ends; the values of the intermediate nodes visited on the query path are added up, and the position of the corresponding query key key in the ordered data set is obtained. Then the corresponding value is read from the disk according to the position information, and the entire query is completed. As can be seen, for each right subtree, the entire left subtree of the trie needs to be dynamically accessed when querying, and when the data volume is large, the decompression process is very long.

[0124] Prior array compression, compact array representation Although the same trie index equivalence is represented as an array, the memory it occupies is still very large. Each intermediate node in the trie index must occupy a position in the array. In general, the intermediate nodes in the trie index are 3 times the leaf nodes, that is, the number of elements of the compact array is three times the number of key-value pairs. Assuming that the number of key-value pairs is large, such as hundreds of thousands to millions, each element in the array needs to be represented by 4 bytes. 1 million key-value pairs require an average of 381 MB of memory, which is obviously not enough.

[0125] The compact array stores all unsigned integers, which can be compressed using existing mature integer compression technology. However, compared with general integer compression algorithms, the compact array trie index has its own data distribution characteristics, so it can be compressed accordingly. Each element in the compact array trie index is an unsigned integer. For a trie T, the left subtree L contains leaf nodes with an average of |L| = |R| = |T| / 2, so the value as a whole presents a binomial distribution characteristic, that is, huffamantree can be directly pre-constructed according to this data distribution characteristic, without sampling the data in the compact array.

[0126] However, the data is too much, and the Huffman tree decompression process can be very long. Therefore, data with a higher frequency of occurrence can be compressed using Huffman compression, and data with a lower frequency of occurrence can be compressed using direct integer compression, such as gamma encoding. Obviously, the closer to the root node, the lower the frequency of the intermediate node; similarly, the closer to the leaf node, the higher the frequency of the intermediate node. A threshold can be set, such as using Huffman compression when the array element <= 16; and using gamma compression when the element > 16.

[0127] In addition, the pre-construction of the Huffman tree can be further optimized. One solution is to pre-construct a large Huffman tree for the data distribution; this may result in a larger Huffman tree and lower compression and decompression efficiency. Another strategy is to pre-construct a Huffman tree for each of the 14 trees. When querying, the corresponding Huffman tree can be queried for compression and decompression according to the number of leaf nodes of the subtree, which can further improve the efficiency of compression and decompression.

[0128] Gamma compression is a context-free compression for single integer, its core idea is to reduce the number of invalid digits in the integer. For example, for uint16, first calculate the number of pre-0, 11, then the number of valid digits is 16-11=5. After that, fill 5-1=4 0s in front of the valid digits, that is, 000010010, from 16 bits to 9 bits. For trie T, the expected value of the number of leaf nodes of the left and right sub-trees is 2^h, so the difference between the actual number of leaf nodes of the sub-tree and the expectation can be stored (the idea of delta-coding), so as to reduce the number of valid digits as much as possible. The difference can be negative, so it needs to be transformed before compression, and the commonly used golomb compression coding scheme can be used. Through this scheme, each element in the compact array only needs an average of 3 bits of space.

[0129] The Level-Trie index of the present embodiment is described in detail below. SILT essentially maps the query key key to the id of the data block slot, that is, maps different types of ordered query keys key to consecutive increasing integers id, and the compact array is compressed in multiple ways, which realizes very low memory occupation, but needs to be decompressed before querying. When the data volume is very large, the query efficiency is very low. Level-Trie is essentially an extension of SILT, but introduces new mapping semantics for disk data. Level-Trie realizes fast retrieval of disk data blocks by mapping index keys to data block ids. The design of Level-Trie follows the following three principles:

[0130] I. Key-value pairs are not allowed to be stored across data blocks. Level-Trie is used to provide mapping of query keys key to data blocks, that is, if the size of the disk data block is 4KB, the size of a single key-value pair cannot exceed 4KB. If the key-value pair is stored across multiple data blocks, the unique data block id cannot be calculated for a given query key key. This limitation can be solved by splitting large key-value pairs into data parts and index parts, and the data block only stores the index part, and the data part is queried through the secondary index. The original SILT needs to establish a mapping of query keys key to data blocks slot, which has a big problem when there are many key-value pairs. If only a mapping of query keys key to memory pages page is established, the amount of index will be greatly reduced. Since the disk data has at least one I / O every time it is queried, if a memory page page contains multiple key-value pairs, a binary search can be performed in the read memory page page to locate twice.

[0131] The key problem here is how to build the mapping from the query key key to the id of the memory page page. SILT builds the mapping from the query key key to the data block slot, which is a one-to-one mapping. But the mapping from the query key key to the id of the memory page page is a many-to-one mapping, which needs to calculate a representative value from multiple key-value pairs in a memory page page and build a many-to-one mapping from the query key key to the representative value. The Level-Trie method is to select the maximum query key key in a memory page page, and then combine the maximum query keys keys in multiple memory pages pages to build a SILT. For example, 256 memory pages pages are a group, 256 query keys keys are selected, and then a SILT is built. In this way, 1MB of data only needs to build a SILT index of 256 query keys keys. Assuming that the average bits required by each query key key is 3, the index of 1MB data only needs 96 bytes on average.

[0132] But there is a problem. The original SILT builds a one-to-one mapping from the query key key to the data block slot, and the queried query key key must exist. But the above scheme selects a query key key from each memory page page, so in the query, most of the query keys keys actually do not exist. Trie index data structure has a natural defect for the non-existent query key key. Trie index is matched by prefix, and the minimum prefix match can uniquely determine the corresponding query key key. However, when the query key key does not exist, only the data that the prefix of the query key key in the trie index is the most consistent can be matched, so the wrong data may be returned.

[0133] Extracting the maximum value of each memory page page to build a trie index, the returned is the maximum value corresponding to the memory page page. The queried query key key may actually be smaller than the maximum value corresponding to the query key key, may be equal, or may be greater. Assuming that the maximum value query key key of a certain memory page page is a, and the queried query key key is x, if x < a, it indicates that x must belong to the memory page page where a is located; if x > a, it indicates that x is in the next memory page page of a.

[0134] Since it is not possible to determine exactly which memory page x is in, it is necessary to read two memory pages page to query respectively, which can use one I / O to read two continuous memory pages page. But if the left subtree of a certain trie index node is 0 in a certain query, it indicates that the given query key key is certainly smaller than the query key key being matched; if the depth of the query is equal to the length of the query key key, it indicates that the query key key being queried is equal to the query key key being matched. In addition, other query results cannot determine the size relationship between the given query key key and the query key key being matched, and it is necessary to load two memory pages page to further determine.

[0135] II. Random jump query under variable-length compression. SILT is highly compressed, and in general, queries need to be decompressed. For a trie T repr(T): = |L|repr(L)repr(R), if the right subtree R is to be queried, it is necessary to skip |L| and the part of repr(L). Since the trie compression representation is variable-length for each child node, when jumping to R, it is necessary to first decompress the left subtree L to specifically locate the starting decoding position of R in the compressed compact array. This will cause the query time to become slower and slower as the number of leaf nodes of T increases.

[0136] The solution of Level-Trie is to construct a direct jump table when generating SILT. When querying the jump, it can be queried first whether there is a jump record in the direct jump table; if there is, the left subtree does not need to be decompressed, and it can be directly jumped to the right subtree to start querying. If there is no direct jump table, the left subtree still needs to be decompressed. The size of the direct jump table is limited, mainly to balance the query performance and memory consumption. In theory, each intermediate node stores a jump information, and then the query does not need to be decompressed at all, but the memory occupation is the largest.

[0137] The construction of the direct jump table is based on the principle of recording the jump information of the subtree as close to the root node as possible. Because generally, the closer to the root node, the more leaf nodes it covers, and the higher the jump efficiency will be. Figure 7 A schematic diagram of trie index jump of the embodiment of the application is shown, as Figure 7 shown is an example of a jump. The direct jump table only stores the jump information of each left subtree, because only when jumping to the right subtree is the positioning information of the left subtree needed. Assuming that T is a complete binary tree, then actually the direct jump table only needs to record the node number of 2n*1. The construction of the direct jump table is very simple, when generating the compact trie T, it only needs to add the constructed bit point in T. Define:

[0138] repr(T) := |L| repr(L) [N L ] repr(R)

[0139] where N L represents the jump point of the left sub-tree L, and [] represents the execution record jump information operation. Since the construction of trie T is strictly from left to right, when L is constructed, the length of the bit array currently constructed is directly obtained as the decoding starting point of the right sub-tree R. The jump point is separately stored in the direct lookup jump table. Since the capacity of the direct lookup jump table is limited, only the jump information close to the root node is recorded. Jump information exceeding the capacity of the direct lookup jump table is directly discarded.

[0140] However, since trie T is not necessarily a full binary tree, the recording strategy of the direct lookup jump table can be improved in the future to record more jump sub-trees as much as possible. In actual testing, 64K random query keys with a length of 100 bytes are constructed, and then 100,000 random queries are performed, Figure 8 a schematic diagram showing the influence of the size of the direct lookup jump table of the embodiment of the application on the query performance is shown, as Figure 8 shown, the improvement result of the size of the direct lookup jump table on the query performance is shown.

[0141] Without the direct lookup jump table, the average single query time of trie is as high as 1367us, and the performance is degraded to an unusable state. When the size of the direct lookup jump table is set to 256, that is, 1KB, the average query time is 5us, and the performance is improved by 271 times. At this time, the memory occupation of the direct lookup jump table is only 1KB, and the compact array after compression is 23KB, and the memory occupation ratio is 4%. When the size of the direct lookup jump table is 10KB, the performance is improved by 1366 times, but the memory occupation ratio is 40%, which is already very large. Therefore, in practice, a compromise needs to be made between query performance and memory occupation according to business requirements.

[0142] Three, multi-level joint index, according to the above principle one, a maximum query key key is extracted from each memory page page, and then a fixed number of memory pages page is a group, so that a 1MB data segment is sampled to establish a SILT index. The problem is how to index between data segments. One optional method is to record the maximum query key key in each data segment, and then perform binary search between data segments when searching, determine the data segment where the query key key is located, and then search the trie index in the corresponding segment to determine the memory page page where the query key key is located.

[0143] In one aspect, this approach requires recording the maximum query key key in each data segment. Assuming there are 500GB of data, 500*1024 data segments are required, and assuming an extreme case where the length of the query key key is 3KB, recording the maximum query key key requires 1.5GB of memory. Although the case is extreme, it still requires a large amount of memory. In addition, the binary search consumes a lot of time.

[0144] Another solution is to extract the maximum query key key again on top of the data segment to establish a two-level index. Assuming that a two-level data segment contains 1024 one-level data segments, i.e., the size of the two-level data segment is 1GB. The average size of the compact array required to establish the index of the two-level data segment is 1024*3=3072. Each element in the compact data is recorded using uint16, which requires 6KB of memory. The memory consumption is very small, so there is no need to compress the two-level data segment. Theoretically, 1TB of data requires 6MB of index memory overhead of the two-level data segment, and 96MB of compressed index overhead of the one-level data segment. Therefore, theoretically, a total of 102MB of memory is required.

[0145] Figure 9 A schematic diagram of the overall architecture of the multi-level data index level-trie of the embodiments of the present application is shown, and the overall structure of the level-trie is shown in Figure 9 From low to high, there are memory pages page, 0-level buckets L0_bucket, 1-level buckets L1_bucket, and 2-level buckets L2_bucket. One memory page page stores one or more key-value pairs. This may result in some space waste, such as a single 4KB memory page page can only store a 3KB key-value pair, resulting in 25% space waste. The size of the memory page page can be configured according to the actual business needs, or when the query key key is too large, the key-value pair data can be split out and stored elsewhere through the two-level index.

[0146] 4096 memory pages page form a 0-level bucket L0_bucket, i.e., 16MB size. The value of 4096 is not fixed and can be dynamically configured according to business needs. A larger 0-level bucket L0_bucket will result in slower query speed, and a smaller 0-level bucket L0_bucket will result in more storage space allocation and more difficult metadata management. Each 0-level bucket L0_bucket will additionally store a query key key, which is the maximum query key key covered by the 0-level bucket L0_bucket. The reason for storing the query key key is that the trie index query may return results of uncertain size and order, so further comparison of the size of the query key key is required to determine whether to query the current 0-level bucket L0_bucket or the next 0-level bucket L0_bucket.

[0147] Thus further cause more I / O. 4096 level 0 buckets L0_bucket make up a level 1 bucket L1_bucket, the value can also be dynamically configured according to business needs. In this way, a level 1 bucket L1_bucket can manage 64GB of data. Since the compact array is not compressed at this level L1, the maximum memory required by a single level 1 bucket L1_bucket compact array is 4096*2*3=24KB. Level 1 bucket L1_bucket can use two index strategies to find data. When the level 1 bucket L1_bucket is small, such as less than 10, a binary search can be directly used, and when the data volume is large, such as hundreds, a trie index query can be used. When trie index query may return uncertain results, the maximum query key max key under the bucket bucket needs to be found to further confirm whether to query the current level 0 bucket L0_bucket or the next level 0 bucket L0_bucket. Level-Trie only sets a level 2 bucket L2_bucket, and the amount of data indexed is related to N, which can be set as a configurable parameter. In actual situations, a single file of 1TB is relatively rare, that is, it contains 16 level 1 buckets L1_bucket. Given that the number of level 1 buckets L1_bucket is generally very small, each level 1 bucket L1_bucket only stores the maximum value query key key, and a binary search is directly performed to reduce the complexity of engineering implementation.

[0148] All level 0 buckets L0_bucket share the same data stream, that is, bit_stream in Figure 9 The reason for such design is mainly to facilitate data landing and loading. If each level 0 bucket L0_bucket has its own compact trie storage space, on the one hand, compact trie is variable length, and space management is more troublesome; on the other hand, data landing and loading need to load the corresponding SILT of level 0 bucket L0_bucket respectively, which may generate more random I / O, and thus cannot effectively utilize the bandwidth of storage. Each level 0 bucket L0_bucket has six attribute information, off, len and skip are used to locate the starting position of bit_stream; key_off and key_sz are used to record the position of the maximum query key key in the query key key array in the level 0 bucket L0_bucket. hints is used to speed up the search of bit_stream, that is, through hints, the entire bit_stream does not need to be decompressed when searching. Its size can be adjusted according to actual business needs.

[0149] Given n query keys key, the size of query key key is s. Assuming the size of a single memory page page is 4KB, a single memory page page can store (4KB-8) / (s+8) key-value pairs. The number of memory pages page required by n query keys key is n / ((4KB-8) / (s+8)). The average index size required by each memory page page is 3 bits, and the memory required by bit_stream is (n / ((4KB-8) / (s+8)))*3 / 8. 4096 memory pages page require a 0-level bucket L0_bucket, and each 0-level bucket L0_bucket requires memory of (80+s) bytes, so the total memory required by 0-level bucket L0_bucket is (n / ((4KB-8) / (s+8))) / 4096*(80+s). 4096 0-level buckets L0_bucket form a 1-level bucket L1_bucket, and each 1-level bucket L1_bucket requires memory of (6+s) bytes, so the total memory required by 1-level bucket L1_bucket is n / ((4KB-8) / (s+8)) / 4096 / 4096*(6+s) bytes. The total memory occupancy is:

[0150]

[0151] Figure 10 A schematic diagram of memory occupancy analysis of the multi-level data index level-trie of the embodiments of the present application is shown, as Figure 10 shown is the theoretical memory occupancy analysis result of level-trie, the left is the relationship between memory occupancy and the number and size of query keys key, and the right is the relationship between query key size and memory under a fixed data amount. Figure 10 In the left, generally the more the number of query keys key and the longer the length, the more memory occupied. Under 1 billion query keys key, and the size of the query key key is 3KB, then the minimum requires 800MB of memory, which is the memory occupancy in the worst case; at this time, the total key-value pair KV occupies 2.79TB of space. Assuming the total data amount of key-value pair KV is 1TB, the length of query key key increases from 4 bytes to 3KB, then theoretically the memory occupancy of level-trie is below 105.7MB.

[0152] Index generation and retrieval, the bit stream generation process in level-trie is the same as SILT, but a new query key grouping strategy is designed. When generating, the generation thread (single thread) continuously appends new data to the Level-Trie. When a memory page is generated in the writing process, the maximum query key in the memory page is put into the 0-level bucket query key array L0 key array. When the 0-level bucket query key array reaches the threshold, such as 256, the generation thread scans all the query keys in the 0-level bucket query key array, and generates the data stream bit_stream of the 0-level bucket, while recording the starting offset, size and common prefix length of the 0-level bucket in the bit_stream. After generation, the 0-level bucket query key array is cleared, and the append write operation continues.

[0153] When the number of 0-level buckets L0_bucket reaches the threshold of 1-level buckets L1_bucket, such as 32768, the maximum query key of the 0-level bucket L0_bucket is put into the 1-level bucket query key array L1 key array. Similarly, when the 1-level bucket query key array reaches the threshold of 1-level buckets L1_bucket, such as 32768, the generation thread scans all the query keys in the 1-level bucket query key array, and generates the non-compressed compact trie of the 1-level bucket, while recording the starting offset, size, common prefix length and maximum query key of the 1-level bucket in the int16 stream.

[0154] At the end of generation, the data may not exactly fill a memory page page; then record the number of key-value pairs in the memory page, and put it into the 0-level bucket query key array. At the end of generation, the 0-level bucket query key array may not have reached the threshold, then the generation thread calls finalize to force the data in the 0-level bucket query key array to generate bit_stream. The data in the 1-level bucket query key array is also processed according to the same logic.

[0155] When a given query key key x is queried, a query entry is obtained from the level 2 bucket L2_bucket, and the number of level 1 buckets L1_bucket is detected; if the level 1 bucket L1_bucket only records the query key key information, therefore, a binary search is required to determine the level 1 bucket L1_bucket to which it belongs. After the id of the level 1 bucket L1_bucket is obtained, it is marked as L1_id, and the int16_stream is queried according to the off, len and skip parameters recorded in the bucket to obtain the level 0 bucket L0_bucket in which the query key key is located, which is marked as L0_id. Finally, the bit_stream is queried according to the information recorded in the L0_bucket to obtain the page id of the memory page in which the query key key is located, which is marked as page_id. Then the memory page page in which the query key key x is actually located is:

[0156]

[0157] After the memory page page in which the query key key is located is obtained, it is first queried whether the memory page page exists in the cache, if not, the corresponding memory page page needs to be loaded from the shared storage. Then the number of records in the memory page page is obtained, and then a binary search is performed. The whole process of memory page page positioning is completed in the memory, and at most only one I / O operation is required. The specific retrieval process is as follows:

[0158] 1. Binary search the maximum query key array max key array of the level 2 bucket L2_bucket to determine the level 1 bucket L1_bucket in which it is located.

[0159] 2. Check the level 0 bucket L0_bucket contained in the level 1 bucket L1_bucket, if less than a certain number, such as 128, directly binary search to locate the corresponding level 0 bucket L0_bucket; if greater than 128, query the trie index to accelerate the confirmation of the level 0 bucket L0_bucket in which it is located.

[0160] 3. Query the bit_stream position corresponding to the level 0 bucket L0_bucket to find the memory page page corresponding to the query key key. The trie index may return an undetermined result, i.e. it cannot be determined whether the query key key is in the memory page page=a or the memory page page=a+1, in which case two consecutive memory pages page need to be read for searching, at most only consuming one I / O.

[0161] 4. After the corresponding memory page page is loaded, the given key-value pair is located in the page by binary search.

[0162] 5. Read the value value in the location, and the query is finished.

[0163] The embodiment adopts a binary prefix tree, which has a higher compression rate than the byte prefix tree of SuRF, and requires only 3 bits per query key key on average. The ordered partitioning of data and the direct lookup skip list are used to support compressed direct lookup, solving the problem of slow query in SILT in the case of large data volume. The multi-level index table is used to support disk retrieval, while SuRF and SILT do not directly support disk retrieval. The binary prefix tree described above realizes efficient index compression. The ordered partitioning of data and the auxiliary direct lookup skip list realize fast compressed direct lookup in the case of large data volume. The multi-level index table realizes efficient support for the disk retrieval scenario.

[0164] Figure 11 A structural block diagram of an apparatus for data lookup in a disk is shown, and the application embodiment also provides an apparatus for data lookup in a disk, which corresponds to the application scenario and method of the method provided by the application embodiment. As shown in Figure 11 The structural block diagram of the apparatus for data lookup in a disk is shown, and the apparatus can include: a receiving module 1101, configured to receive a target query key of target data to be looked up; an index module 1102, connected to the receiving module 1101, configured to query the memory location of the target query key in a multi-level data index according to the target query key, wherein the multi-level data index is generated and compressed during data storage, and includes a plurality of levels of query key indexes, the query key index is generated based on a binary prefix tree, and is used to map the next level query key index or the memory location; a lookup module 1103, connected to the index module 1102, configured to obtain the stored key-value pair based on the memory location, and find the target query value corresponding to the target query key, wherein the memory location stores a plurality of key-value pairs, each key-value pair includes a query key and a query value corresponding to the query key, the query value is used to indicate the corresponding data, and the target query value is used to indicate the target data.

[0165] According to the above data lookup apparatus of the embodiment, after receiving the target query key of the target data to be looked up, the memory location corresponding to the target query key is queried in the multi-level data index by the compressed direct lookup. The multi-level data index can efficiently compress the query keys of the stored data, reduce the occupied resources of the data index, and establish the mapping relationship between the memory location of the stored data and the query key, and then the key-value pair corresponding to the query key of the target data to be looked up, that is, the memory location of the target data in the disk, can be quickly found through the multi-level data index.

[0166] The multi-level data index can be compressed for direct query without decompression of all, thereby improving query efficiency and reducing resource occupation during query. Moreover, the lowest level query key index of the multi-level data index directly maps a memory location, and only one read-write operation is needed to obtain the memory page of the memory location corresponding to the target data, thereby greatly reducing the occupation of read-write resources.

[0167] Figure 12 A structural block diagram of a disk data index generation apparatus provided in an embodiment of the present application is shown. Corresponding to the application scenario and method of the method provided in the embodiment of the present application, the embodiment of the present application further provides a disk data index generation apparatus. As shown in Figure 12 A structural block diagram of a disk data index generation apparatus provided in an embodiment of the present application is shown. Corresponding to the application scenario and method of the method provided in the embodiment of the present application, the embodiment of the present application further provides a disk data index generation apparatus. As shown in

[0168] The generation module 1201 is configured to receive target data to be stored, and generate a key-value pair of the target data, wherein the key-value pair includes a target query key and a target query value of the target data, the target query key is used to find the corresponding target query value, and the target query value is used to indicate the target data. The writing module 1202, connected to the generation module 1201, is configured to write the key-value pair into a current unit memory location, extract an identifier of the unit memory location after the unit memory location is full, and generate a lowest level query key index of a multi-level data index according to the identifier. The index module 1203, connected to the writing module 1202, is configured to generate and compress the multi-level data index level by level based on the lowest level query key index as the stored data increases, wherein the multi-level data index includes a plurality of levels of query key indexes used to find a memory location to which the target query key belongs, the query key index is generated based on a binary prefix tree, and is used to map a next level query key index or a memory location.

[0169] According to the above disk data index generation apparatus of the embodiment, after receiving target data to be stored and generating a key-value pair of the target data, the key-value pair is written into a current unit memory location, an identifier of the unit memory location is extracted after the unit memory location is full, and a lowest level query key index of a multi-level data index is generated according to the identifier. The multi-level data index is generated level by level as the stored data increases. The multi-level data index can efficiently compress the query key of the stored data, reduce the occupation of data index, and establish a mapping relationship between the memory location of the stored data and the query key, so that the key-value pair corresponding to the query key of the target data to be found can be quickly found through the multi-level data index, that is, the memory page of the target data in the disk.

[0170] The multi-level data index can be compressed and directly inquired when inquiring, without decompressing all, thereby improving the inquiry efficiency and reducing the occupied resources when inquiring. Moreover, the lowest level inquiry key index of the multi-level data index directly maps the memory position, and only one read-write operation is needed to obtain the memory position corresponding to the target data when inquiring, so that the inquiry of the target data can be completed, and the occupation of the read-write resources is greatly reduced. Therefore, the problem of low inquiry efficiency in the related art is solved.

[0171] The functions of each module in each device of the embodiments of the present application can be referred to the corresponding description in the above method, and has the corresponding beneficial effects, which will not be repeated here.

[0172] Figure 13 A block diagram of an electronic device for implementing the embodiments of the present application is shown. As shown in the figure, the electronic device includes a memory 1301 and a processor 1302, and the memory 1301 stores a computer program that can run on the processor 1302. The processor 1302 implements the method in the above embodiments when executing the computer program. The number of the memory 1301 and the processor 1302 can be one or more. Figure 13

[0173] The electronic device further includes:

[0174] A communication interface 1303 for communicating with external devices and transmitting data.

[0175] If the memory 1301, the processor 1302 and the communication interface 1303 are independently implemented, the memory 1301, the processor 1302 and the communication interface 1303 can be connected to each other through a bus and complete the communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, Figure 13 only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0176] Optionally, in specific implementation, if the memory 1301, the processor 1302 and the communication interface 1303 are integrated on a chip, the memory 1301, the processor 1302 and the communication interface 1303 can complete the communication between them through an internal interface.

[0177] ​The embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the method provided in the embodiment of the present application.

[0178] The embodiment of the present application further provides a chip, which comprises a processor, and is used for calling and running instructions stored in a memory, so that a communication device installed with the chip executes the method provided in the embodiment of the present application.

[0179] The embodiment of the present application further provides a chip, which comprises an input interface, an output interface, a processor and a memory, and the input interface, the output interface, the processor and the memory are connected through internal connection paths, and the processor is used for executing code in the memory, and when the code is executed, the processor is used for executing the method provided in the embodiment of the present application.

[0180] It should be understood that the processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc. It should be noted that the processor can be a processor supporting an advanced RISC machine (ARM) architecture.

[0181] Further, the memory can optionally include a read-only memory and a random access memory. The memory can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memory. The non-volatile memory can include a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically EPROM (EEPROM), or a flash memory, among others. The volatile memory can include a random access memory (RAM), which is used as an external cache. By way of example, and not limitation, many forms of RAM are available. The RAM can be a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate SDRAM (DDR SDRAM), an enhanced SDRAM (ESDRAM), a Sync link DRAM (SLDRAM), and a direct Rambus RAM (DR RAM), among others.

[0182] In the above-described embodiments, all or part of the embodiments can be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to the present disclosure are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable apparatus. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another computer-readable storage medium.

[0183] In the description of the application, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the application. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in one or more embodiments or examples. In addition, different embodiments or examples described in the specification and characteristics of different embodiments or examples can be combined and combined by those skilled in the art without contradiction.

[0184] In addition, the terms "first", "second" are only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the application, the meaning of "multiple" is two or more, unless otherwise specifically limited.

[0185] Any process or method described in the flowchart or otherwise described herein can be understood as representing a module, a segment or a portion of code including one or more executable instructions for implementing specific logical functions or processes. And the scope of the preferred embodiments of the application includes additional implementations, in which the functions can be performed in the order shown or discussed, including in a substantially simultaneous manner or in reverse order according to the functions involved.

[0186] The logic and / or steps described in the flowchart or otherwise described herein, for example, can be considered as a list of executable instructions for implementing logical functions, which can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus, such as a computer-based system, a system including a processor or other system that can fetch and execute instructions from the instruction execution system, device or apparatus, or in conjunction with these instructions execution system, device or apparatus.

[0187] It should be understood that parts of the application can be implemented in hardware, software, firmware or a combination thereof. In the above embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. All or part of the steps of the above-described embodiment method can be completed by a program instructing the relevant hardware, which can be stored in a computer-readable storage medium, and the program includes one or a combination of the steps of the method embodiment when executed.

[0188] In addition, each of the function units in each embodiment of the present application can be integrated in one processing module, or each unit can be physically present separately, or two or more units can be integrated in one module. The integrated module can be realized in the form of hardware or in the form of a software function module. When the integrated module is realized in the form of a software function module and sold or used as an independent product, it can also be stored in a computer readable storage medium. The storage medium can be a read-only memory, a magnetic disk or an optical disk, etc.

[0189] The above is only exemplary embodiments of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of various changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for data lookup in a disk, comprising: receiving a target query key of target data to be looked up; level by level querying a memory location of the target query key in a multi-level data index according to the target query key, wherein the multi-level data index is generated and compressed when data is stored, and comprises a plurality of levels of query key indexes, the query key indexes are generated based on a binary prefix tree, and are used to map a next level query key index or the memory location; based on the memory location, obtaining a stored key-value pair to look up a target query value corresponding to the target query key, wherein the memory location stores a plurality of key-value pairs, each key-value pair comprises a query key and a query value corresponding to the query key, and the query value is used to indicate corresponding data, and the target query value is used to indicate the target data.

2. The method of claim 1, wherein, level by level querying a memory location of the target query key in a compressed multi-level data index according to the target query key, comprising: According to the target query key, in the compressed multi-level data index, a highest level query key index L containing the target query key is found N wherein N is the number of levels of the multi-level data index; from the highest level query key index L N Start, in the upper level query key index L k Find the target query key in the next level query key index L k-1 , until the lowest level query key index L1 is found, wherein each upper level query key index L k in the multi-level data index includes a plurality of next level query key indexes L k-1 , k∈[1, N] finding a corresponding memory location according to a lowest level query key index L1.

3. The method of claim 2, wherein, According to the target query key, in the compressed multi-level data index, a highest level query key index L containing the target query key is found N comprising: In the case where the highest-level query key index L N is one, the next-level query key index L N in which the target query key is located is searched in the highest-level query key index L N-1 ; In the highest level query key index L N In the case of multiple, multiple said highest level query key index L N The query key records part of the data in the order, through the dichotomy or traversal method, in the highest level query key index L N Find the next level query key index L N-1 Wherein, the number of multiple said highest level query key index L N The number of first preset number.

4. The method of claim 2, wherein, The previous query key index L k Find the next-level query key index L containing the target query key. k-1 Until the lowest-level query key index L1 is found, including: In the next level query key index L k In the next level query key index L k-1 In the case where the query key is recorded, the next level query key index L k-1 in which the target query key is located is searched according to the number of next level query key indexes L k-1 ; In the upper level query key index L k The lower level query key index L k-1 In the case of recording the memory location, according to the attribute information of the upper level query key index L k and the prefix tree, the lower level query key index L k-1 wherein the attribute information includes the relevant information of the prefix tree generating the upper level query key index L k The prefix tree records the data of the lower level query key index L k according to the order of recording the lower level query key index L k-1 The prefix tree is generated from left to right according to the data sequence of the query key of the plurality of lower level query key indexes L k recorded in the upper level query key index L k-1 ​ finding the lowest level query key index L1 where the target query key is located through level by level querying.

5. The method of claim 4, wherein, In the next level query key index L k In the next level query key index L k-1 In the case where the query key is recorded, the next level query key index L k-1 in which the target query key is located is searched according to the number of the next level query key index L k-1 , comprising: The previous level query key index L k The next-level query key index L in k-1 If the number does not exceed the second preset number, the next-level query key index L of the target query key is found by binary search. k-1 Among them, the multiple next-level query key indexes of the highest-level query key index record the compressed query keys after removing common prefixes in order of size; In case that the number of the lower level query key index L k exceeds the second preset number, the lower level query key index L k-1 where the target query key is located is found through the attribute information of the upper level query key index L k and the prefix tree. k-1 ​ 6. The method of claim 5, wherein, In the upper level query key index L k The lower level query key index L k-1 In the case that the memory location is recorded, the lower level query key index L k in which the target query key is located is searched through the attribute information of the upper level query key index L k-1 , including: In the upper level query key index L k The prefix tree direct jump table is obtained from the attribute information, wherein the direct jump table records jump data of different sub-trees in the history query of the prefix tree. in the process of looking up the target query key, if a corresponding subtree jump record is queried, directly jumping through the subtree jump record of the direct jump table, and if the corresponding subtree jump record cannot be queried, traversing in the prefix tree until a leaf node corresponding to the target query key is found; determining a next level query key index L to which the target query key belongs according to the leaf node k-1 .

7. The method of claim 2, wherein, finding a corresponding memory location according to a lowest level query key index L1, comprising: inquiring a memory location to which the target query key belongs in the lowest level query key index L1 according to attribute information of the lowest level query key index L1 and a prefix tree, wherein the maximum query key or the minimum query key of a mapped memory page is recorded in the attribute information of the lowest level query key index L1 as a lookup basis of the query key, a query result of the target query key in the prefix tree is a lookup basis of a target memory page or a lookup basis range of a plurality of adjacent memory pages, and the memory location is stored in units of memory pages; in a case where the query result is that the target query key belongs to a target memory page or a plurality of adjacent memory pages, taking the target memory page or the plurality of adjacent memory pages as the corresponding memory location.

8. The method of claim 7, wherein, obtaining a key-value pair of the memory page to look up a target query value corresponding to the target query key, comprising: loading the corresponding memory page from a disk storing data through a single read-write operation to obtain a key-value pair in the corresponding memory page, wherein the memory page is a data storage structure of the disk; finding a target query value corresponding to the target query key in the key-value pair of the corresponding memory page through binary search.

9. A disk data index generation method, comprising: receiving target data to be stored, and generating a key-value pair of the target data, wherein the key-value pair comprises a target query key and a target query value of the target data, the target query key is used to look up the corresponding target query value, and the target query value is used to indicate the target data; write the key-value pair into a current unit memory location, after the unit memory location is full, extract the identification of the unit memory location, generate a lowest level query key index of a multi-level data index according to the identification; with the increase of stored data, generate the multi-level data index level by level and compress it, wherein the multi-level data index comprises multiple levels of query key indexes for finding a memory location to which the target query key belongs, the query key index is generated based on a binary prefix tree and is used for mapping a next level query key index or the memory location.

10. The method of claim 9, wherein, write the key-value pair into a current unit memory location, after the unit memory location is full, write the identification of the unit memory location into a lowest level query key index L1 of a multi-level data index, comprising: write the key-value pair into a current unit memory location, after the unit memory location is full, extract the maximum or minimum query key in the unit memory location, and generate a prefix tree of the lowest level query key index L1 according to the maximum or minimum query key; In a case where the number of unit memory locations corresponding to the prefix tree of the same lowest-level query key index L1 reaches a third preset number, the lowest-level query key index L1 is generated according to the prefix tree, and corresponding attribute information is saved, wherein the attribute information includes relevant information of a prefix tree of an upper-level query key index L k generated by the prefix tree, and the prefix tree records the upper-level query key index L k-1 in a size order. create a lowest level query key index L1 and generate a corresponding prefix tree, and continue to process the newly stored data.

11. The method of claim 10, wherein, generate the multi-level data index level by level with the increase of stored data, comprising: With the increase of the stored data, the next level query key index L k-1 is gradually accumulated, the next level query key index L k-1 reaches a previous level query key index L k corresponding to the fourth preset number of the allowed cases, the corresponding previous level query key index L k is generated, and the corresponding attribute information is saved; until a highest-level query key index L is generated N wherein the highest-level query key index L N has a number not more than a fifth preset number.

12. The method of claim 9, wherein, the method further comprises: determine the common prefix of the query key corresponding to the query key index of each level, and perform common prefix pruning to retain the identification part of the query key; create a binary prefix tree of the identification part of the query key of the stored data; compress the binary prefix tree into a compact array through compact recursive representation; based on the compact array, obtain a prefix tree through compression, wherein the prefix tree is used for finding the query key index of each level to which the target query key belongs. 13.An electronic device, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor implements the method of any one of claims 1 to 12 when executing the computer program. 14.A computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program implements the method of any one of claims 1 to 12 when executed by a processor. 15.A computer program product, comprising a computer program, wherein the computer program implements the method of any one of claims 1 to 12 when executed by a processor.

Images