Method and apparatus for fast indexing and retrieval based on kv store engine

Abstract

The application provides a method and device for fast indexing and retrieval based on a kv storage engine. The method for fast indexing and retrieval based on the kv storage engine comprises the following steps: recombining a key structure of a main table according to a retrieval condition, and the key structure of the main table is composed of key fields of the retrieval condition; storing the composed key structure of the main table into an index table, and the index table conforms to a LSM-tree structure design; when a field is used as a query condition, if the field has the index table, the key structure meeting the condition is obtained by searching the index table, the key structure meeting the condition is recombined in a fixed form, the corresponding key structure in the main table is obtained, and then the value required is searched through the key structure of the main table; when a range is used as the query condition, the key structure meeting the condition is obtained by searching the index table, the key structure meeting the condition is recombined in a fixed form, the corresponding key structure in the main table is obtained, batch searching is performed in the main table, and all value values meeting the condition can be found.

Description

Technical Field

[0001] This invention relates to the field of traffic statistics, and more specifically, to a method and apparatus for fast indexing and retrieval based on a key-value storage engine. Background Technology

[0002] As we all know, among the current data storage engines, the NoSQL KV data storage engine has the highest storage efficiency. Among them, the storage engine designed and developed based on the LSM-tree concept has the highest efficiency and the most stable performance. It is widely used in distributed cloud and IoT scenarios. Its ability to efficiently store massive amounts of data has been sought after by major manufacturers, who have launched database storage engines or products based on LSM-tree.

[0003] Storing statistical information from network traffic using a key-value data storage engine can greatly improve storage efficiency and is adaptable to network communication information statistics environments with varying traffic volumes.

[0004] However, these NoSQL storage engines also have significant limitations. One is that the ordering of keys is achieved through comparison functions, which is actually detrimental to range searches using conditions. Another limitation is write amplification. Although it can achieve fast data storage and ordering, write amplification is unavoidable due to architectural design issues, especially when the value is large (exceeding a certain threshold), where the write amplification problem becomes more pronounced.

[0005] Current solutions and existing problems:

[0006] 1. To implement indexing, a sub-table is used to generate a new key-value structure based on the index conditions, where the index conditions are the keys and the values ​​are the keys from the main table. The sub-table ensures the ordered nature of the index conditions, thus achieving range indexing. This method is prone to write amplification, requiring multiple I / O operations during reads.

[0007] 2. To reduce I / O and write amplification, the index condition and key are combined to generate a sub-table. The key in the sub-table is a combination of the index structure and the key in the main table, eliminating the need for a value. The keys in the sub-table are ordered according to the index condition. Because this method combines the keys, the key portion becomes very large, resulting in significant read amplification. Furthermore, the increased key length leads to a greater computational load when the data storage engine calculates the index on the sub-table's keys. Summary of the Invention

[0008] The purpose of this invention is to provide a method and apparatus for fast indexing and retrieval based on a key-value storage engine.

[0009] The present invention aims to solve the problems existing in the prior art.

[0010] Compared with the prior art, the technical solution of the present invention and its beneficial effects are as follows:

[0011] The first aspect of this invention discloses a method for fast indexing and retrieval based on a key-value storage engine, comprising: reorganizing the key structure of a main table according to retrieval conditions, and forming a key structure of the main table based on the key fields of the retrieval conditions; storing the formed key structure of the main table in an index table, wherein the index table conforms to an LSM-tree structure design; when using a field as a query condition, if the field has an index table, then obtaining a key structure that meets the conditions by retrieving the index table, and then reorganizing the retrieved key structure in a fixed form to obtain the corresponding key structure in the main table, and then querying the required value through the key structure of the main table; when using a range as a query condition, then obtaining a key structure that meets the conditions by retrieving the index table, and then reorganizing the retrieved key structure in a fixed form to obtain the corresponding key structure in the main table, and performing a batch search in the main table to find all values ​​that meet the conditions.

[0012] Further improvements include: using range as the query condition, if there is no continuous index table based on a certain field, then the key needs to be processed using a specific algorithm, and the processed key is then compared.

[0013] As a further improvement, using range as the query condition, under the same other search conditions, keys that satisfy the range condition are consecutive in the index table; in the index table, from the first key structure that meets the range condition to the first key structure that does not meet the range condition, all key structures in between meet the range search requirements.

[0014] As a further improvement, the keyword types available for retrieval in traffic statistics are 2-byte and 4-byte integers, as well as 6-byte strings; the retrieval fields include time, MAC, IP, port, and protocol, and various retrieval conditions can be formed by combining the retrieval fields.

[0015] As a further improvement, the main table uses time as the keyword for retrieval. The key structure in the main table is: "Time field" + "Mac field" + "IP field" + "Port field" + "Protocol field".

[0016] As a further improvement, when IP is used as the main search keyword and time is also needed as a composite index, the key structure in the index sub-table is as follows: "IP field" + "Time field" + "MAC field" + "Port field" + "Protocol field".

[0017] As a further improvement, in the index table, all data is sorted by the IP field, and if the IP fields are the same, they are sorted by the time field, and so on.

[0018] The second aspect of this invention discloses a device for fast indexing and retrieval based on a key-value storage engine, comprising: a reorganization module configured to reorganize the key structure of a main table according to retrieval conditions, forming the key structure of the main table based on the key fields of the retrieval conditions; a storage module configured to store the formed key structure of the main table into an index table, the index table conforming to an LSM-tree structure design; and a query module configured to, when using a field as a query condition, if the field has an index table, retrieve the key structure that meets the conditions by retrieving the index table, then reorganize the retrieved key structure in a fixed form to obtain the corresponding key structure in the main table, and then query the required value through the key structure of the main table; the query module is further configured to, when using a range as a query condition, retrieve the key structure that meets the conditions by retrieving the index table, then reorganize the retrieved key structure in a fixed form to obtain the corresponding key structure in the main table, and perform a batch search in the main table to find all values ​​that meet the conditions.

[0019] The beneficial effects of this invention are as follows:

[0020] By extracting the index-related information from the value and placing it into the key structure, the length of the value can be reduced while achieving a high-dimensional ordered effect for the index. Taking the LSM-tree structure as an example, its biggest advantage lies in the ordered arrangement of data, thus achieving fast retrieval. Generally, within the storage engine, data is arranged byte-wise according to the key structure; therefore, the composition order of the key structure should fully consider query efficiency and the feasibility of range searches.

[0021] To achieve fast indexing and range retrieval under different query conditions, the key structure of the main table needs to be indexed. To reduce write amplification and improve read efficiency, the key structure in the index table is designed as a variant of the main table's key structure. The main table's key structure already contains the keywords used for retrieval; only the key structure needs to be reorganized according to the retrieval conditions. The newly generated key structure only needs to be stored in the index table. The index table conforms to an LSM-tree structure design, therefore the key structure in the index table is arranged in byte order.

[0022] Therefore, when querying the index table using search criteria, both fast lookups and range searches can be achieved. The key structure of the main table can be obtained through a fixed transformation method, thus eliminating the need to store the main table's key structure as the value in the index table. The index table satisfies both fast and range search requirements while minimizing write amplification.

[0023] This invention utilizes a key-value storage engine to achieve fast indexing and retrieval of network communication statistics by transforming the byte sequence of the key structure and using a masking algorithm, thereby improving efficiency and reducing memory and computational costs. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of a fast indexing and retrieval method based on a key-value storage engine provided in an embodiment of the present invention.

[0025] Figure 2 This is a schematic diagram of a device for fast indexing and retrieval based on a kv storage engine provided in an embodiment of the present invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of them. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to represent selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0027] In the description of this invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0028] Reference Figure 1As shown, a method for fast indexing and retrieval based on a key-value storage engine includes: reorganizing the key structure of the main table according to retrieval conditions, forming the key structure of the main table based on the key fields of the retrieval conditions; storing the formed key structure of the main table in an index table, the index table conforming to an LSM-tree structure design; when using a field as a query condition, if the field has an index table, then the key structure that meets the conditions is obtained by retrieving the index table, and then the retrieved key structure is reorganized in a fixed form to obtain the corresponding key structure in the main table, and then the required value is retrieved through the key structure of the main table; when using a range as a query condition, then the key structure that meets the conditions is obtained by retrieving the index table, and then the retrieved key structure is reorganized in a fixed form to obtain the corresponding key structure in the main table, and then a batch search is performed in the main table to find all values ​​that meet the conditions.

[0029] This also includes: when using a range as a query condition, if there is no continuous index table based on a certain field, then a specific algorithm needs to be used to process the key, and the processed keys are then compared. This requires traversing a certain range in the main table using a function, and then comparing the results using an algorithm.

[0030] When using a range as a query condition, if other search conditions are the same, keys that satisfy the range condition are consecutive in the index table;

[0031] In the index table, all key structures from the first key structure that meets the range condition to the first key structure that does not meet the range condition meet the range search requirements.

[0032] For example, when designing the key structure for all information about the same IP address within a certain time range, the time field should be placed after the IP address information. Therefore, keys for the same IP address are adjacent in the index table, and the key structure will be sorted according to the time field.

[0033] The keyword types available for retrieval in traffic statistics are 2-byte and 4-byte integers, and 6-byte strings;

[0034] The search fields include time, MAC address, IP address, port, and protocol. Combining these fields can create various search criteria.

[0035] The main table uses time as the keyword for retrieval. The key structure in the main table is as follows:

[0036] "Time field" + "Mac field" + "IP field" + "Port field" + "Protocol field".

[0037] When using IP address as the primary search keyword and requiring time as a composite index, the key structure of the index sub-table is as follows:

[0038] "IP field" + "Time field" + "MAC field" + "Port field" + "Protocol field".

[0039] In the index table, all data is sorted by the IP field. If the IP fields are the same, they are sorted by the time field, and so on.

[0040] The index table allows for quick retrieval of continuous data by querying IP addresses and time ranges. A simple iteration function can then be used to obtain all matching key structures. For example, one could query the communication information of an IP address 192.168.1.1 from 15:20:00 on November 23, 2022 to 15:22:23 on November 23, 2023.

[0041] First, convert the IP address into a 4-byte integer A, convert the start time 15:20:00 on November 23, 2022 into an 8-byte unsigned integer timestamp T1, and convert the end time 15:22:23 on November 23, 2023 into an 8-byte unsigned integer timestamp T2.

[0042] Secondly, when using the KV storage engine for retrieval, an iterative function is used to search the index table for all key structures located between key1(A+T1) and key2(A+T2). Since the key structures in the index table are ordered, iterative retrieval can begin once the first key structure greater than key1 is found, continuing until the first key structure greater than key2 appears, at which point all key structures within that range meet the requirements.

[0043] Finally, by changing the order of the fields in the key structure using a transformation function, the key structure in the corresponding main table can be obtained. This key structure can then be used to perform a precise query in the main table to find the corresponding value.

[0044] Reference Figure 2As shown, a device for fast indexing and retrieval based on a key-value storage engine includes: a reorganization module configured to reorganize the key structure of a main table according to retrieval conditions, forming the key structure of the main table based on the key fields of the retrieval conditions; a storage module configured to store the formed key structure of the main table into an index table, the index table conforming to an LSM-tree structure design; and a query module configured to, when using a field as a query condition, if the field has an index table, retrieve the key structure that meets the condition by searching the index table, then reorganize the retrieved key structure in a fixed form to obtain the corresponding key structure in the main table, and then query the required value through the key structure of the main table; the query module is also configured to, when using a range as a query condition, retrieve the key structure that meets the condition by searching the index table, then reorganize the retrieved key structure in a fixed form to obtain the corresponding key structure in the main table, and perform a batch search in the main table to find all values ​​that meet the conditions.

[0045] By extracting the index-related information from the value and placing it into the key structure, the length of the value can be reduced while achieving a high-dimensional ordered effect for the index. Taking the LSM-tree structure as an example, its biggest advantage lies in the ordered arrangement of data, thus achieving fast retrieval. Generally, within the storage engine, data is arranged byte-wise according to the key structure; therefore, the composition order of the key structure should fully consider query efficiency and the feasibility of range searches.

[0046] Secondly, to achieve fast indexing and range retrieval under different query conditions, the key structure of the main table needs to be indexed. To reduce write amplification and improve read efficiency, the key structure in the index table is designed as a variant of the main table's key structure. The main table's key structure already contains the keywords used for retrieval; only the keys need to be reorganized according to the search conditions, and the newly generated key structure needs to be stored in the index table. The index table conforms to an LSM-tree structure design, therefore the key structure in the index table is arranged in byte order.

[0047] Therefore, when querying the index table using search criteria, both fast lookups and range searches can be achieved. The key structure of the main table can be obtained through a fixed transformation method, thus eliminating the need to store the main table's key structure as the value in the index table. The index table satisfies both fast and range search requirements while minimizing write amplification.

[0048] The above embodiments are only used to explain the technical solutions of the present invention and are not intended to limit it. Those skilled in the art should understand that any modifications and equivalent substitutions that do not depart from the spirit and scope of the present invention should fall within the protection scope of the claims of the present invention.

Claims

1. A method for fast indexing and retrieval based on a key-value storage engine, characterized in that, include: The key structure of the main table is reorganized according to the search conditions, and the key structure of the main table is formed based on the key fields of the search conditions. The key structure of the main table is stored in the index table, which conforms to the LSM-tree structure design. When a field is used as a query condition, if the field has an index table, the key structure that meets the condition is obtained by retrieving the index table. Then, the retrieved key structure is reorganized in a fixed form to obtain the corresponding key structure in the main table. Finally, the required value is retrieved through the key structure in the main table. When using a range as the query condition, the index table is searched to obtain the key structure that meets the condition. The retrieved key structure is then reorganized in a fixed manner to obtain the corresponding key structure in the main table. A batch search is then performed in the main table to find all the values ​​that meet the condition. The primary key of the index table is formed by recombining the key fields in the key structure of the main table; By adjusting the order of key fields in the primary key of the index table using a preset transformation function, the key structure of the main table can be restored, eliminating the need to store the key structure of the main table as a value in the index table. The keyword types used for retrieval in traffic statistics are 2-byte and 4-byte integers, and 6-byte strings; the retrieval fields include time, MAC, IP, port, and protocol; for retrieval using time as a keyword in the main table, the key structure in the main table is: "time field" + "MAC field" + "IP field" + "port field" + "protocol field"; the retrieval fields can be combined to form various retrieval conditions; When using IP as the primary search keyword and also requiring time as a composite index, the key structure in the index sub-table is as follows: "IP field" + "Time field" + "MAC field" + "Port field" + "Protocol field".

2. The method for fast indexing and retrieval based on a key-value storage engine according to claim 1, characterized in that, Also includes: If a range is used as the query condition and there is no continuous index table based on a certain field, then the key needs to be processed by an algorithm, and the processed key needs to be compared.

3. The method for fast indexing and retrieval based on a key-value storage engine according to claim 1, characterized in that, When using a range as a query condition, if other search conditions are the same, keys that satisfy the range condition are consecutive in the index table; In the index table, all key structures from the first key structure that meets the range condition to the first key structure that does not meet the range condition meet the range search requirements.

4. The method for fast indexing and retrieval based on a key-value storage engine according to claim 3, characterized in that, In the index table, all data is sorted by the IP field. If the IP fields are the same, they are sorted by the time field, and so on.

5. A device for fast indexing and retrieval based on a key-value storage engine, characterized in that, include: The reorganization module is configured to reorganize the key structure of the main table according to the search conditions, and form the key structure of the main table based on the key fields of the search conditions. The storage module is configured to store the key structure of the main table into an index table, which conforms to the LSM-tree structure design. When the query module is configured to use a field as a query condition, if the field has an index table, the key structure that meets the condition is obtained by retrieving the index table. Then, the key structure obtained by the query is reorganized in a fixed form to obtain the corresponding key structure in the main table. Finally, the required value is retrieved through the key structure of the main table. The query module is also configured to use a range as the query condition. In this case, the key structure that meets the condition is obtained by searching the index table. The key structure that meets the condition is then reorganized in a fixed form to obtain the corresponding key structure in the main table. A batch search is then performed in the main table to find all the value that meets the condition. The primary key of the index table is formed by recombining the key fields in the key structure of the main table; By adjusting the order of key fields in the primary key of the index table using a preset transformation function, the key structure of the main table can be restored, eliminating the need to store the key structure of the main table as a value in the index table. The keyword types used for retrieval in traffic statistics are 2-byte and 4-byte integers, and 6-byte strings; the retrieval fields include time, MAC, IP, port, and protocol; for retrieval using time as the keyword in the main table, the key structure of the main table is: "time field" + "MAC field" + "IP field" + "port field" + "protocol field"; the retrieval fields are combined to form various retrieval conditions; when IP is used as the main retrieval keyword and time is also needed as a composite index, the key structure of the index sub-table is as follows: "IP field" + "time field" + "MAC field" + "port field" + "protocol field".

Images