A database learning type index construction method and system

Abstract

The application relates to a database learning type index construction method and system. The method comprises the following steps: constructing a cumulative distribution function according to a data keyword and a data storage position; fitting the cumulative distribution function by using a machine learning model to obtain the relevance between the data keyword and the data storage position, and obtaining a learning type index; and quickly positioning the position of a key value to be queried according to the learning type index. The application can overcome the problems of high adjustment difficulty, poor self-adaptive ability and large memory space occupation of an auxiliary data structure of an existing B-tree database index algorithm, and can effectively reduce the memory occupation of the auxiliary data structure and improve the self-adaptive adjustment ability of the database index.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of key-value storage-based in-memory databases, and particularly relates to a method and system for constructing a database learning index. BACKGROUND

[0002] With the continuous development of the Internet ecosystem, large Internet companies need to store data up to the PB level, and the new business data generated every day has reached the TB level. In order to cope with such large-scale growing data storage and update requirements, distributed key-value storage databases have gradually become the first choice for large-scale data storage. Key-value storage databases usually store different key-value pairs in memory, hard disk and other storage devices in the order of key-value size. Therefore, how to quickly locate the specific storage location of a given key-value and efficiently support insertion, deletion and other functions has become a key issue determining the performance of current key-value storage databases.

[0003] Well-known key-value storage databases, such as RocksDB, Redis and PostgreSQL, basically use auxiliary data structures as indexes to speed up the database insertion, deletion, modification and query. The common method at present is to group key-value pairs into data blocks, locate a specific key-value pair in the block through binary search algorithm, and build an index with binary property through B-tree for each data block. This method needs to consume a large amount of space in memory to maintain the index, and in the face of different data loads, it often takes a lot of manual time to adjust the optimal B-tree and data block parameters, making it difficult to provide high-quality adaptive services. SUMMARY

[0004] In order to overcome the problems of high adjustment difficulty, poor adaptability and large memory space occupation of auxiliary data structures of the existing B-tree database index algorithm, the application provides a method using a learning index, which can effectively reduce the memory occupation of auxiliary data structures and improve the adaptive adjustment ability of the database index.

[0005] The object of the application is achieved by the following technical solutions:

[0006] A database learning index construction method, comprising the following steps:

[0007] Constructing a cumulative distribution function according to the data key and the data storage location;

[0008] Fitting the cumulative distribution function by using a machine learning model to obtain the relevance of the data key and the data storage location, and obtaining a learning index;

[0009] Quickly locating the position of the key-value to be queried according to the learning index.

[0010] Further, under the premise of data key-value ordered storage, a monotonic increasing relationship is presented between the key of data and the storage position, so that a cumulative distribution function (CDF) can be constructed, and the cumulative distribution function F(key) is modeled as pos=F(key) x N, where key represents the data key, pos represents the data storage position, and N represents the total data size. The cumulative distribution function itself contains two key information of data key and data storage position. The basic idea of the learning index is to use a machine learning model to fit the cumulative distribution function, obtain the relevance between the storage position and the data key value, and thus achieve the effect of quickly locating the data position.

[0011] Further, the process of fitting the cumulative distribution function is to select appropriate parameters of the cumulative distribution function, i.e., F function, through machine learning, so as to minimize the loss function. In practice, the F function can be a linear function, a feedforward neural network (FNN), etc. In the present application, the basic goal of the learning index is to fit the cumulative distribution function with as small an average error as possible.

[0012] Further, in order to enable the cumulative distribution function to be fitted using a relatively simple function, the present application performs grouping processing on the key-value data, predefines the number of data points in each group segmentSize, divides each group after reading segmentSize data points, fits a machine learning model, records the model parameters, and uses the maximum and minimum values of the data keys in the group as the demarcation points as the grouping basis. Thus, the inference speed and construction speed of the model are greatly accelerated, the model inference time is maximally reduced without seriously affecting the prediction accuracy, the query process can also be accelerated, the simple model construction is faster, the model reconstruction time is greatly reduced, and a good foundation is laid for dynamic updating.

[0013] Further, in order to accurately and quickly locate the to-be-queried key to the target group, a root query table is used as an auxiliary data structure. The root query table is based on a radix tree, and maps data keys with the same first k binary bits to the same node, so as to realize the quick grouping and positioning of data keys. Here, k is the RadixBit, i.e., the depth of the radix tree. For example, RadixBit=3 actually means that the data keys are mapped to 23=8 nodes according to the first three binary bits, and the data keys with the same first three binary bits are mapped to the same node. Thus, the data keys can be quickly grouped and positioned. 3= 8 leaf nodes. When constructing the model, the split points between two groups are used as the knots of the root query table, and an array, called knots array, is constructed with these split points, and a pointer is stored in each position of the knots array to point to their corresponding array (i.e. the array with the split point as the right interval endpoint). When inputting a data key Key to be queried, the model first locates two knots in the knots array according to the radix tree, and the property of the radix tree guarantees that the Key is between the two knots. Then, through binary search, the index of the knots array that is not less than the maximum value of the Key is found, and the index of the corresponding group in the array is obtained, and the grouping positioning process of the root query table is completed.

[0014] Further, since there is a certain error between the predicted position and the real position of the learning index, after locating the predicted position in the target group, the last step of searching is performed by using the exponential search method.

[0015] Further, the above mainly introduces the data structure used by the present application, and the complete process of index construction is introduced below by taking the grouping linear fitting as an example:

[0016] 1) The index structure is constructed from bottom to top, first ensuring that the data keys are stored in order in the array, and then reading the data keys key and their positions positon in order, and performing a training process every segmentSize data keys. Linear function or low-order polynomial function can be selected for fitting, and here the least square fitting of linear function is taken as an example. After the fitting of the group is completed, the fitting coefficients, i.e. the slope slope and the intercept intercept, are saved. And the maximum value of the data keys in this group is put into the knots array (knots array), and the pointer of the corresponding group is saved. Then, according to the first RadixBit bits of the binary representation of the maximum value, the root query table is updated. Continue to read the data until the number of remaining data points is less than segmentSize, and the last group of index model is trained.

[0017] 2) After the model construction is completed, in each query operation, first find the corresponding group of the root query table by using the key to be queried key. Get the model coefficients slope and intercept stored in the group, get the predicted position pred through the formula pred = key × slope + intercept, and finally find the accurate position pos of the data in the array from pred by using the exponential search method, and complete a query.

[0018] Further, in order to obtain better model fitting effect while saving space as much as possible, the application proposes a split method with dynamic adaptive array length, which can dynamically split the array. The split method with dynamic adaptive array length comprises the following steps:

[0019] 1) First, on the basis of completing the above model construction, tentatively merge several arrays under the same parent node at the bottom layer, replace the original several index models with an index model, to save space consumption.

[0020] 2) Then, based on the average prediction error of the index model as an index, evaluate the effect of merging, if the error is lower than the set threshold, accept the merging, otherwise, cancel the merging operation.

[0021] 3) Repeat the merging operation from bottom to top until there is no array that meets the merging condition.

[0022] Further, in order to support update operations such as insertion and deletion, a hanging external array is introduced, the inserted element is placed in the external array at the to-be-inserted position, and then the external array and the original array are integrated regularly. At each insertion operation, first perform a query operation to find the position corresponding to the to-be-inserted key (i.e. the position of the maximum value not greater than the current key in the array, which is referred to as lower_bound here). An array of a fixed size is externally connected at this position, and the to-be-inserted key-value pair is placed in it, and this array is independent of the model. At this time, the query operation also needs to add an additional step, the original model will first locate to lower_bound, if lower_bound is the key to be queried, a query operation is completed, if lower_bound is not the key to be queried, query in the corresponding external array. When the external array is full, integrate the external array and the original array, and retrain the model. When performing a deletion operation, first perform a query to find the position of the data, and also mark the position as empty, and integrate all the positions marked as empty when the model is reconstructed.

[0023] Based on the same inventive concept, the application also provides a database learning type index construction system using the above method, which comprises:

[0024] A cumulative distribution function construction module is configured to construct a cumulative distribution function according to data keys and data storage positions.

[0025] A cumulative distribution function fitting module is configured to fit the cumulative distribution function by using a machine learning model, to obtain the relevance of data keys and data storage positions, and to obtain a learning type index.

[0026] The query module is used for quickly locating a position of a key value to be queried according to the learning index.

[0027] The application provides a database learning index construction method collectively referred to as Group Spline, obtains Spline in a tree shape through dynamic adaptive segmentation of array length, thereby constructing a learning database index, can quickly locate a position of a key value to be queried, guarantees that a query result can be given after accessing limited layers of segment, and improves query speed compared with a traditional B-tree index scheme. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 is an example of root query table and an accelerated query based on the same. Model1-Model9 are abstractions of function F in the above.

[0029] Figure 2 is an example of the last step of searching in an exponential search mode.

[0030] Figure 3 is a local example of an array length dynamic adaptive segmentation training process. Segment represents a segmented data segment, and training represents a training index model.

[0031] Figure 4 is an example of inserting new key value pairs based on a hanging chain external array. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical scheme and advantages of the application more clear, the application is further described in detail below in combination with examples in the drawings. It should be understood that the specific examples described herein are only used to explain the application, and are not used to limit the application.

[0033] Figure 1 is a root query table and a query process based on the same. First, a root query table is constructed with 32 as a span, each span contains 1 function model (such as a linear model), and the maximum value and the minimum value of the data key in the group are used as left and right boundary points. For the key to be queried 162, its binary representation is 10100010, according to the RadixBit indication of the first 3 bits, it is positioned between 152 and 191 by using the radix tree, then it is determined that the key to be queried is between 152 and 178 by using the binary search mode in the node array, and it is positioned to the corresponding group by using the one-to-one correspondence between the boundary point and the array, that is, the positioning of the root query table is completed.

[0034] Figure 2 is an example process of the last step search by using exponential search. For convenience, the predicted position of the model is recorded as 0, and the offset of the accurate position of the key-value pair from the predicted position is 12. First, by increasing the exponential power step, the size difference between the key value of the exploration point and the query key value is compared to determine the search boundary. In the figure, the key value at the offset 8 is smaller than the query key value, and the key value at the offset 16 is larger than the predicted key value, so the left and right boundaries are determined as 8 and 16 respectively. After determining the left and right boundaries, the binary search method is used to determine the accurate position of the data storage.

[0035] Figure 3 is a local process of dynamic adaptive segmentation training of array length. In this process, after an index model is used for the topmost segment, the average prediction error of the model exceeds the set threshold, so the fitting effect is not good enough, and the segmentation is continued. Then, in the second layer segment, an index model is trained for the left segment to achieve the expected effect, so it is accepted. The average prediction error of the index model of the right segment still exceeds the set threshold, so the segmentation is continued.

[0036] Figure 4 shows the process of inserting a new key-value pair based on the hanging chain external array. To insert the key-value pair with the key value 79, first find the lower_bound corresponding to 79 in this segment, which is 69. The hanging chain external array corresponding to 69 still has spare space, so insert 79 into the hanging chain external array corresponding to 69 to complete the insertion process.

[0037] Specific application scenarios of the present application: in PostgreSQL and MySQL databases, the present application can be applied to replace the traditional B-tree index structure, thereby improving the query efficiency.

[0038] Table 1 is experimental data using the method of the present application:

[0039] Table 1

[0040] Average query latency / nanoseconds norm logn amzn osm wiki Group Spline 93 103 205 263 183 B-tree 473 473 471 474 483

[0041] Among them, norm is an artificial data set generated based on normal distribution, logn is an artificial data set generated based on lognormal distribution, amzn is a data set composed of part of the shopping data on Amazon, osm is a data set composed of address coordinate data provided on the Open Street Map website, and wiki is a data set composed of part of the access volume of entries on Wikipedia. All the above data sets contain one million data entries.

[0042] Based on the same inventive concept, another embodiment of the present application provides a database learning type index construction system using the above method, which comprises:

[0043] A cumulative distribution function construction module is configured to construct a cumulative distribution function according to the data key and the data storage location;

[0044] A cumulative distribution function fitting module is configured to fit the cumulative distribution function by using a machine learning model to obtain the relevance between the data key and the data storage location, and obtain a learning type index;

[0045] A query module is configured to quickly locate the position of the key value to be queried according to the learning type index.

[0046] Based on the same inventive concept, another embodiment of the present application provides an electronic device (computer, server, smart phone, etc.), which comprises a memory and a processor, the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program comprises instructions for executing each step in the method of the present application.

[0047] Based on the same inventive concept, another embodiment of the present application provides a computer readable storage medium (such as ROM / RAM, magnetic disk, optical disk), which stores a computer program, and the computer program is executed by a computer to realize each step of the method of the present application.

[0048] The specific embodiments of the present application disclosed above are intended to help understand the content of the present application and to implement the same, and those skilled in the art can understand that various substitutions, changes and modifications are possible without departing from the spirit and scope of the present application. The present application should not be limited to the content disclosed in the embodiments of the present application, and the protection scope of the present application is defined by the scope of the claims.

Claims

1. A database learning type index construction method, characterized by, The method comprises the following steps: According to the data key and the data storage location, a cumulative distribution function is constructed; The cumulative distribution function is fitted by using a machine learning model to obtain the relevance between the data key and the data storage location, and a learning index is obtained; According to the learning index, the location of the key value to be queried is quickly located; The cumulative distribution function is fitted by using a machine learning model, which comprises grouping processing of key value data; The grouping processing of key value data comprises: presetting the number of data points in each group segmentSize, dividing a group after reading segmentSize data points each time, fitting a machine learning model, recording the model parameters, and taking the maximum and minimum values of a group of data keys as the demarcation points as the grouping basis; According to the learning index, the location of the key value to be queried is quickly located, which comprises: using a root query table as an auxiliary data structure to accurately and quickly locate the key to be queried to the target group; the root query table is based on a radix tree Radix Tree, and the first k binary bits of the same data key are mapped to the same node to realize the quick grouping and positioning of the data key, wherein k is RadixBit, the depth of the radix tree; the demarcation points between two groups are used as the nodes of the root query table, and an array, called a node array, is constructed by using the demarcation points; a pointer is stored in each position of the node array; when the data key Key to be queried is input, the first two nodes in the node array are located according to the radix tree, and then a binary search is performed to find the index in the node array that is not less than the maximum value of Key, that is, the index of the corresponding group in the array is obtained, and then the root query table grouping and positioning process is completed.

2. The method of claim 1, wherein, The cumulative distribution function is modeled as pos=F(key)×N, wherein F(key) represents the cumulative distribution function, key represents the data key, pos represents the data storage location, and N represents the total data size.

3. The method of claim 2, wherein, The cumulative distribution function is fitted by using a machine learning model, which is to select appropriate parameters of the cumulative distribution function F function by machine learning to minimize the loss function.

4. The method of claim 1, wherein, An array length dynamic adaptive segmentation method is adopted to dynamically segment the array, which comprises the following steps: The arrays under the same parent node in the bottom layer are tentatively merged, and an index model is used to replace the original several index models to save space consumption; Based on the average prediction error of the index model, the effect of merging is evaluated, and if the error is lower than the set threshold, the merging is accepted, otherwise the merging operation is cancelled; The merging operation is repeated from bottom to top until there is no array that meets the merging condition.

5. The method of claim 1, wherein, The insertion operation and the deletion operation are supported by introducing the outer array, i.e. the inserted element is placed in the outer array at the position to be inserted, and then the outer array is integrated with the original array regularly; in each insertion operation, a query operation is performed first to find the position corresponding to the key to be inserted, i.e. the position of the maximum value not greater than the current key in the array, which is called lower_bound, an array of a fixed size is circumscribed to the position, and the key-value pair to be inserted is placed in the array; In the query operation, the lower_bound is located first, if the lower_bound is the key to be queried, the query operation is completed, if the lower_bound is not the key to be queried, the query is performed in the outer array corresponding to the lower_bound; when the outer array is full, the outer array and the original array are integrated, and the model is retrained; in the deletion operation, the query is performed first to find the position of the data, and the position is marked as empty, and the integration is performed on all the positions marked as empty.

6. A database learning type index construction system employing the method according to any one of claims 1 to 5, characterized by, The method comprises the following steps: A cumulative distribution function construction module is configured to construct a cumulative distribution function according to a data key and a data storage position; A cumulative distribution function fitting module is configured to fit the cumulative distribution function by using a machine learning model to obtain the relevance between the data key and the data storage position, and obtain a learning index; A query module is configured to quickly locate the position of a key-value to be queried according to the learning index.

7. An electronic device, comprising: The computer program is configured to be executed by the processor, and the computer program comprises instructions for executing the method of any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores the computer program, and the computer program is executed by the computer to realize the method of any one of claims 1-5.

Images