A bloom filter based learning index method and system

Abstract

The application relates to a learning index method and system based on a prefix Bloom filter, and aims to solve the problem that, when computer query data does not exist in a computer database, the existing index needs to traverse the whole structure of the corresponding index or a machine learning model to determine whether the computer query data exists in the computer database and the existing position of the computer query data in the database, so that the computer data query time is too long. The application comprises the following steps: constructing a tree-shaped learning index; the learning index comprises non-leaf nodes and leaf nodes; for the non-leaf nodes, a prefix Bloom filter is used to determine whether computer data exists in a computer database; if the computer data does not exist in the computer database, a result that the computer data does not exist in the computer database is returned; if the computer data exists in the computer database, a machine learning model is used to query the position of the computer data in the computer database in the leaf nodes. The application belongs to the field of database indexes.

Description

TECHNICAL FIELD

[0001] The present application relates to an indexing method and system, in particular to a learned indexing method and system based on prefix Bloom filter, belonging to the field of database indexing. BACKGROUND

[0002] With the continuous improvement of hardware technology, the current computer system uses a main memory with a size of the order of gigabytes. When the entire computer database is stored in the main memory, assuming that the disk I / O is the main cost of database operation, the focus of researchers is then on some structures that can seriously affect the performance of computer database systems, and the indexing structure is one of the research focuses. Modern computer databases basically use traditional indexing structures such as B+ trees, and compared with search algorithms without indexes, the I / O number of B+ trees is much smaller, so B+ trees are adopted by most traditional computer database systems. However, the data volume of computer databases has now reached more than PB, and the traditional computer database index has also increased accordingly, so it is impossible to put all the indexes into the memory, because this will affect the performance of the index, and thus limit the performance of the computer database. In order to solve this problem, the concept of learned index is proposed, which combines machine learning algorithms with traditional indexing structures, and uses machine learning models to optimize / replace traditional indexing structures. Since the machine learning model only needs to store a small part of the parameters, it greatly reduces the space cost, at the same time, the learned index considers some characteristics of the data, such as data distribution rules, etc., thereby improving the performance of the index.

[0003] In the data query of the computer, the above-mentioned indexing structure has very good performance in terms of space cost and query efficiency when the data actually exists in the computer database, but when the data does not exist in the computer database, the above-mentioned learned index still needs to run the complete machine learning model or the whole index structure to determine the position of the data in the computer database, so when there are too many non-key data in the data set, the performance of the learned index is limited. Although Bloom filter can be used to determine whether the data exists, Bloom filter cannot return the actual position of the data, but only whether the data exists in the computer database. Therefore, this paper proposes a one-dimensional learned index based on Bloom filter to solve the problem of wasted time when querying data that does not exist in the computer database. SUMMARY

[0004] The application aims to solve the problem that when the computer data to be queried does not exist in the computer database, the existing index needs to traverse the entire index structure or machine learning model to determine whether the data to be queried exists in the computer database and its location in the computer database, resulting in too long computer data query time.

[0005] The technical scheme adopted by the application is:

[0006] It comprises the following steps:

[0007] S1, obtaining a computer database as a training set;

[0008] S2, constructing a tree structure learning index, the learning index comprising non-leaf nodes and leaf nodes, training the learning index using the training set, for the non-leaf nodes, using a prefix Bloom filter to determine whether the training data exists in the computer database, if not, returning a result that the data does not exist in the computer database; if so, using a machine learning model in the leaf node to query the location of the training data in the computer database, obtaining a trained learning index;

[0009] S3, inputting the computer data to be queried into the trained learning index, obtaining a result that the computer data does not exist in the computer database or the location of the computer data in the computer database.

[0010] Further, in S2, the tree structure learning index is constructed, the learning index comprising non-leaf nodes and leaf nodes, the learning index is trained using the training set, for the non-leaf nodes, a prefix Bloom filter is used to determine whether the training data exists in the computer database, if not, a result that the data does not exist in the computer database is returned; if so, a machine learning model is used in the leaf node to query the location of the training data in the computer database, obtaining a trained learning index, the specific process being:

[0011] According to the training data, the number of layers and the number of nodes of each layer of the learning index are defined, the nodes are divided into leaf nodes and non-leaf nodes, a prefix Bloom filter is arranged at each non-leaf node, a machine learning model is arranged at the leaf node, each training data in the training set is converted into binary data, a certain binary data is input into the first layer of the constructed learning index, the prefix Bloom filter of the first layer non-leaf node is used to query whether the corresponding binary data exists in the layer, if the corresponding binary data exists, the next layer is entered, the prefix Bloom filter of each non-leaf node of the next layer is used to query whether the corresponding binary data exists in the next layer; if the corresponding binary data does not exist, a result that the certain binary data does not exist in the computer database is returned, and the query is ended;

[0012] The above operations are repeated, and when the judgment result of the prefix Bloom filter of each layer of non-leaf nodes is that a certain binary data exists, the leaf node is entered, the machine learning model of the leaf node is used to query the existing position of the certain binary data in the computer database, and the position is returned.

[0013] Further, the machine learning model in S2 includes multiple regression models.

[0014] Further, in the training of the leaf node of the learning index in S2, the machine learning model automatically selects the regression model that best fits the input data of the leaf node and trains the regression model as the optimal model of the leaf node.

[0015] Further, in the training of the prefix Bloom filter of the learning index in S2, the prefix length and the hash function of the prefix Bloom filter are manually optimized according to the training data and the query efficiency.

[0016] Further, the prefix length of the prefix Bloom filter of each layer of non-leaf nodes of the learning index in S2 is sequentially increased compared to the prefix length of the prefix Bloom filter of the previous layer, that is, the prefix length of the bottom layer of the prefix Bloom filter is higher than the prefix length of the previous layer of the prefix Bloom filter.

[0017] A learning index system based on a Bloom filter, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any step of the learning index method based on a Bloom filter when executing the computer program.

[0018] A computer-readable storage medium storing a computer program, wherein the computer program is executed by a processor to implement any step of the learning index method based on a Bloom filter.

[0019] Advantages:

[0020] The application constructs a tree-structured learning index, which includes leaf nodes and non-leaf nodes. The learning index is layered according to the computer data to be queried, and the number of layers and the number of nodes of each layer of the learning index are customized. In this way, the hierarchical structure is used to divide data to reduce the problem of reduced model accuracy caused by too large data to be queried. For the leaf nodes of the learning index, a machine learning model is used to query the location of the computer data in the computer database. Since the machine learning model can consider data characteristics instead of nodes, it can better fit the data. For the non-leaf nodes, a prefix Bloom filter is used to determine whether the data to be queried exists in the computer database. The prefix Bloom filter can effectively reduce the space cost while effectively filtering data. That is, the application provides a learning index method and obtains a learning index. When querying computer data using the learning index, if the computer data exists in the computer database, the machine learning model of the leaf node of the learning index can directly output the location of the queried computer data in the computer database. If the queried computer data does not exist in the computer database, only the adjacent layers of the index where the data continuously exist need to be traversed until the queried computer data does not exist, and the result that the queried computer data does not exist in the computer database is directly returned. The entire learning index does not need to be traversed, and a large amount of query time is saved.

[0021] Since the Bloom filter has the property of 0 false negatives, when data exists in the database, the Bloom filter will not judge that the data does not exist, and therefore will not affect the existing data. However, when data does not exist in the database, a well-designed hash algorithm can make the Bloom filter return to exist with a very small probability, so the Bloom filter has good filtering properties for data that does not exist in the database. Therefore, the prefix Bloom filter and the non-leaf nodes of the learning index are integrated in the application. When the queried data does not exist in the database, only the adjacent layers where the data continuously exist need to be traversed until the data does not exist, and the result that the data does not exist in the database is directly returned. In this way, most of the data that does not exist in the database can be returned in the middle layers of the index model, and a small part of the data that is not filtered out will also be judged by the leaf node to be non-existent, saving a large amount of query time. At the same time, the use of the prefix Bloom filter can greatly reduce the space cost of the Bloom filter. Since each Bloom filter does not need to judge a long prefix, a smaller Bloom filter set can be used, thereby reducing the space cost of the Bloom filter. Therefore, the application can achieve a good trade-off between space cost and query time. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 is a learning index structure based on a Bloom filter;

[0023] Figure 2 This is a schematic diagram of the application process of the learning index based on the Bloom filter;

[0024] Figure 3 is a schematic diagram of an embodiment; DETAILED DESCRIPTION

[0025] Specific implementation method 1: Combination Figures 1-2 This embodiment describes a learning indexing method based on a Bloom filter, which includes the following steps:

[0026] S1. Obtain a computer database as a training set.

[0027] S2. Construct a tree-structured learning index. The learning index includes non-leaf nodes and leaf nodes. The learning index is trained using a training set. For non-leaf nodes, a prefix Bloom filter is used to determine whether the training data exists in the computer database. If not, a result indicating that the data does not exist in the computer database is returned. If so, a machine learning model is used in the leaf node to query the location of the training data in the computer database to obtain a trained learning index. The specific process is as follows:

[0028] The number of layers and the number of nodes in each layer of the learning index are customized according to the training data. The nodes are divided into leaf nodes and non-leaf nodes. A prefix Bloom filter is set at a non-leaf node, and a machine learning model is set at the leaf node. Each training data in the training set is converted into binary data. A certain binary data is input into the first layer (root node) of the constructed learning index. The prefix Bloom filter of the non-leaf node of the first layer is used to query whether there is corresponding binary data in this layer. If there is corresponding binary data, enter the next layer, and use the prefix Bloom filter of each non-leaf node in the next layer (second layer) to query whether there is corresponding binary data in the next layer (second layer). In the next layer, a Bloom filter with a longer prefix length is used to further determine whether the training data exists; if there is no corresponding binary data, the result that the certain binary data does not exist in the computer database is returned, and the query ends. The judgment result is that the data does not exist, which means that the data must not exist in the computer database.

[0029] Repeat the above operation. When the judgment result of the prefix Bloom filter of each layer of non-leaf nodes is that a certain binary data exists, enter the leaf node, use the machine learning model of the leaf node to query the location of a certain binary data in the computer database, and return the location; when the judgment result of any layer of non-leaf nodes with the same parent node is that a certain binary data does not exist, return the result that the certain binary data does not exist in the computer database, and end the query.

[0030] The learning index structure similar to the tree structure is constructed, the number of layers and the number of nodes of each layer of the learning index structure are customized according to the training set, and the training data is divided by using the hierarchical structure, so that the data set corresponding to each leaf node is not too small or too large. If the data set is too large, the data distribution is more complex, and then it is difficult for the regression model in the machine learning model (the machine learning model automatically uses different regression models for different computer data) to achieve or complete accurate fitting when the leaf node is fitted by using the machine learning model. At this time, if accurate fitting is to be achieved, an additional model with a larger size needs to be used, so that the space cost is large. If the data set is too small, the space is wasted.

[0031] The obtained training set is used to train the constructed learning index. For the leaf node, the most fitted regression model in the machine learning model is used to train the leaf node according to the training data of the leaf node, and the most fitted regression model is used as the model of the leaf node. Since the machine learning uses some characteristics of computer data, such as data distribution, the use of the machine learning model instead of the index has better space cost and query performance. For the non-leaf node, the prefix Bloom filter is trained by using the training data. The prefix length of the prefix Bloom filter is manually optimized according to the training data and the query efficiency, and the hash function is optimized. The optimization of the hash function of the Bloom filter can reduce the false positive rate of the Bloom filter, so that each prefix Bloom filter can achieve a balance between space cost and accuracy on a global basis. The best prefix Bloom filter is obtained, and thus the trained learning index is obtained.

[0032] The prefix Bloom filter is a kind of Bloom filter. The data used by the Bloom filter is all bits of the data, and the data used by the prefix Bloom filter is binary data. The Bloom filter is established by using some bits of the binary data. Compared with the Bloom filter, the space cost can be reduced while the accuracy is ensured, because each Bloom filter does not need to design an algorithm for a long prefix, only a small part of the data is stored, the size of the Bloom filter set can be reduced, a smaller set is used, and thus the size of the Bloom filter is reduced. The selection standard of the "some bits" is to divide the data relatively uniformly. The prefix length of the prefix Bloom filter of each non-leaf node of each layer of the learning index is sequentially increased compared with the prefix length of the prefix Bloom filter of the previous layer, that is, the prefix length (the number of bits used) of the prefix Bloom filter of the bottom layer is higher than the prefix length of the prefix Bloom filter of the previous layer.

[0033] S3, input the computer data to be queried into the trained learning index, and obtain the result that the computer data does not exist in the computer database or the position of the computer data in the computer database.

[0034] Specific implementation method two: combined with Figures 1-2To describe this embodiment, a Bloom filter-based learning indexing system described in this embodiment includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements any step of a Bloom filter-based learning indexing method.

[0035] Specific implementation method three: Combination Figures 1-2 This embodiment describes a computer-readable storage medium having a computer program stored therein. When the computer program is executed by a processor, any step of a learning indexing method based on a Bloom filter is implemented.

[0036] Example

[0037] Taking computer data as an example, the data of the computer during operation needs to be recorded in real time and stored in the computer database for quick query later. Figure 3 As shown, the specific usage of the present invention in computer data query is demonstrated. When a certain computer data to be queried is 25, the computer data 25 to be queried is first converted into binary data 11001, and the query data 11, 10, and 01 that make up the binary data 11001 are input into the learning index constructed by the present invention. The query is performed according to the prefix length of the prefix Bloom filter of the first layer (root node) of the learning index. The query result of the first layer is that the prefix 11 data exists, and the second layer is entered to query the prefix 110. The first node of the second layer determines that the prefix 110 does not exist. At this time, the second node of the second layer is entered to query, and the second layer node determines that the prefix 110 exists. At this time, the same method is used to enter the third layer node to query the prefix 1100. Since all the third layer child nodes pointed to by the second layer node that determines that the prefix 110 exists judge that the prefix 1100 does not exist, it exits directly and returns that the binary data 11001 does not exist in the computer database, and no further query is entered into the lower layer nodes. Similarly, only when all layers of a non-leaf node determine that the binary data 11001 exists, will the machine learning model in the leaf node be used to predict whether the binary data 11001 exists in the computer database and the location of the binary data 11001 in the computer database. Therefore, when the present invention queries data that does not exist in the database, it does not need to query all layers, but instead returns the result that the data does not exist in the database midway through the query, thereby having better query efficiency and space cost, saving query time.

Claims

1. A bloom filter based learning index method, characterized in that: It comprises the following steps: S1, obtaining a computer database as a training set; S2, constructing a tree-structured learning index, the learning index comprising non-leaf nodes and leaf nodes, training the learning index using the training set, for the non-leaf nodes, using a prefix Bloom filter to determine whether the training data exists in the computer database, if not, returning a result that the data does not exist in the computer database; if so, using a machine learning model in the leaf node to query the location of the training data in the computer database, obtaining the trained learning index, specifically: According to the training data, the number of layers and the number of nodes of each layer of the learning index are defined, the nodes are divided into leaf nodes and non-leaf nodes, a prefix Bloom filter is set at each non-leaf node, and a machine learning model is set at each leaf node, each training data in the training set is converted into binary data, a certain binary data is input into the first layer of the constructed learning index, the prefix Bloom filter of the first layer non-leaf node is used to query whether the corresponding binary data exists in the layer, if the corresponding binary data exists, the next layer is entered, and the prefix Bloom filter of each non-leaf node in the next layer is used to query whether the corresponding binary data exists in the next layer; if the corresponding binary data does not exist, a result that a certain binary data does not exist in the computer database is returned, and the query is ended; When the judgment results of the prefix Bloom filters of each layer of non-leaf nodes are all that a certain binary data exists, the leaf node is entered, the machine learning model of the leaf node is used to query the existing position of the certain binary data in the computer database, and the position is returned; Obtaining the trained learning index; S3, inputting the computer data to be queried into the trained learning index, obtaining a result that the computer data does not exist in the computer database or a position of the computer data in the computer database. 2.The Bloom filter based learning index method of claim 1, wherein: The machine learning model in S2 comprises a plurality of regression models. 3.The Bloom filter based learning index method of claim 2, wherein: In the training of the leaf nodes of the learning index in S2, the machine learning model automatically selects the regression model that best fits the input data of the leaf node and trains the regression model as the best model of the leaf node. 4.The Bloom filter based learning index method of claim 3, wherein: In the training of the prefix Bloom filter of the learning index in S2, the prefix length and the hash function of the prefix Bloom filter are manually optimized according to the training data and the query efficiency. 5.The Bloom filter based learning index method according to claim 4, wherein: The prefix length of the prefix Bloom filter of each layer of non-leaf nodes of the learning index in S2 is successively increased compared to the prefix length of the prefix Bloom filter of the previous layer, that is, the prefix length of the prefix Bloom filter at the bottom layer is higher than the prefix length of the prefix Bloom filter of the previous layer. 6.A Bloom filter based learning index system, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: The processor executes the computer program to implement the steps of the method of any one of claims 1-5.

7. A computer readable storage medium storing a computer program, characterized in that: The computer program is executed by the processor to implement the steps of the method of any one of claims 1-5.

Images