A learning index model oriented to update distribution

Abstract

The application discloses a learning index model facing update distribution, comprising the following steps: (1) using an overhead model to construct a key into an RMI structure; (2) uniformly dividing the range of data by using the cumulative distribution function of data at each internal node; (3) constructing a monotonous gapped unary linear regression model at each leaf node; (4) learning the update distribution of data during the construction of the index; (5) point query and insertion; and (6) model expansion and splitting. The application uses a machine learning model to replace a traditional B-tree-based database index structure, simultaneously learns the update distribution of data, utilizes the characteristics of the update distribution of data and the advantages of the machine learning model, and greatly reduces the storage overhead of the index structure and the overhead during data insertion.

Description

Technical Field

[0001] This invention belongs to the field of one-dimensional data indexing technology, specifically relating to a learning indexing model oriented towards update distribution. Background Technology

[0002] Data indexing has always been one of the core issues in the field of computer science, and handling data insertion is one of the most important issues in data indexing. With the advent of the big data era, the demand for supporting frequent data insertion and efficient querying is increasing daily. In the past few decades, one-dimensional data has generally been indexed using B-trees and their variants. However, as the amount of data increases, traditional database indexes based on B-trees often require large IO overhead and high space costs. It is meaningful and necessary to find a new index structure with lower IO and space overhead than B-trees while effectively handling frequent data insertion.

[0003] In recent years, the concept of learned indexes has been proposed. Its main idea is that an index can be viewed as a model, and a simple machine learning model can replace the widely used traditional one-dimensional data index structure, the B-tree. Using a machine learning model to index one-dimensional data effectively avoids the multiple indirect searches required by traditional indexes for each query, significantly improving the retrieval speed of one-dimensional data. Furthermore, since the machine learning model only stores model parameters and other information within the nodes, the space cost of a one-dimensional learned index is much smaller than that of a B-tree. This approach is novel and has significant research value. However, early learned indexes could only handle static queries and could not effectively handle data insertion because they stored all keys in a dense memory array, resulting in excessively high data insertion costs. Yet, data insertion is necessary in most practical applications.

[0004] Currently, B-trees and their variants are widely used one-dimensional data indexing structures. However, B-trees require significant space and time overhead. While current learned indexes effectively handle data queries, they cannot efficiently handle frequent data insertions. Therefore, there is an urgent need for a method based on a better index structure to efficiently handle frequent data insertions. Summary of the Invention

[0005] In view of the above problems, this invention provides a learning index model oriented towards update distribution. The main problem it solves is to use a simple machine learning model to replace the traditional one-dimensional data index based on B-tree, learn the data update distribution to improve the ability of the existing learning index to handle frequent data insertion, and reduce the storage overhead and query overhead of the index structure by leveraging the powerful expressive power of the machine learning model, thereby improving the index structure's ability to handle frequent insertion.

[0006] The technical solution adopted in this invention is: a learning index model oriented towards update distribution, comprising the following steps:

[0007] (1) Use the overhead model to construct the key as an RMI structure;

[0008] (2) Use the cumulative distribution function of the data to evenly divide the range of the data in each internal node to ensure that the data space interval in each internal node is the same;

[0009] (3) Construct a monotonic univariate linear regression model with gaps at each leaf node, and use the model-guided method to insert keys.

[0010] (4) Learn the data update distribution during the index building process, and reserve gaps for data insertion based on the data update distribution;

[0011] (5) Use exponential search within the leaf nodes to perform key lookups and insertions;

[0012] (6) After inserting a certain amount of data, the model is expanded or split to effectively handle subsequent insertions.

[0013] Further, the specific implementation of step (1) is as follows: First, sort all keys in ascending order, and then calculate the cost of each layer according to the distribution of keys using the cost model until the cost increases. The internal nodes of the RMI structure are divided into data ranges, and the leaf nodes are used for data fitting and prediction. The key is one-dimensional data.

[0014] Further, the specific implementation of step (2) is as follows: learn the cumulative distribution function of the data, calculate the model parameters of the internal nodes according to the cumulative distribution function of the data, so that the linear regression model can evenly distribute the data into its corresponding child nodes according to the range, input the key value into the model, and take the integer part of the model output result as the sequence number of the child node to which the key belongs.

[0015] Further, the specific implementation of step (3) is as follows: use univariate linear fitting to train the data in each leaf node, input the key value into the linear regression model, take the integer part of the model output result as the position of the key in the memory array, when the predicted position is not a gap, insert by moving a small amount of data to ensure that the error between the actual position of the key and the predicted position is small, and the gap to which it belongs is the position of the memory array where no data is stored.

[0016] Further, the specific implementation of step (4) is as follows: First, before batch loading the data, the data to be inserted is obtained according to the insertion frequency and then the updated data distribution is calculated. When batch loading the data, the internal node range is divided using the updated distribution learned in advance, and the leaf node capacity is set according to the updated distribution. A gap is reserved for the data to be inserted. If the insertion position predicted by the model is a gap and the data is arranged in an orderly manner, it can be directly inserted into the predicted position. Otherwise, since the gap reserved for the data to be inserted is calculated according to the model-guided method in step (3), the model predicts the position more accurately, and the insertion position is determined by exponential search.

[0017] Furthermore, the specific implementation of step (5) is as follows: based on the linear regression model trained in step (3) and the model-guided insertion method, the actual position of the key is close to the model-predicted position, and an exponential search is used to achieve fast search starting from the model-predicted position.

[0018] Furthermore, the specific implementation of step (6) is as follows: when data is frequently inserted into a certain leaf node, the effectiveness of the model decreases accordingly. Therefore, when the amount of data in the array corresponding to a certain leaf node reaches the expansion threshold, the leaf node model and array capacity are expanded. When the amount of data in the leaf node reaches the splitting threshold, the data in the current leaf node is retrained as an internal node, and the data is allocated to the corresponding leaf nodes according to the range.

[0019] This invention proposes a learning index model where the index key is in a one-dimensional space. It assumes that the inserted data is known before batch loading and uses a machine learning model to replace the traditional one-dimensional index structure based on B-trees. It utilizes the data update distribution to adjust the node capacity and model parameters, leveraging the powerful expressive power of the machine learning model and the characteristics of the data update distribution to greatly reduce the space and time overhead of the index structure, while efficiently handling frequent data insertions. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the overall process of the data query and update method for update distribution based on the learning index model of the present invention.

[0021] Figure 2 This diagram illustrates a horizontal comparison of the data insertion performance of the present invention with other existing traditional index structures and learned index structures. Detailed Implementation

[0022] To describe the present invention in more detail, the technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings and specific embodiments.

[0023] Example:

[0024] In this implementation, we consider the following application scenarios: For social media platform user login and registration, a database stores user account information. When a user logs into their social media account, their phone automatically prompts for their password, allowing for quick login – this constitutes a data query. This application is present on major social media platforms such as QQ, WeChat, and Weibo. When a user applies to register a social media account, this constitutes a data insertion, as users want to complete account registration as quickly as possible.

[0025] For ease of explanation, we will first use abstract language to describe this scenario. On social media platforms, user login information consists of a user account and a password. This information can be represented as a key-value pair. The user account can be used as the key to uniquely identify the user's account, and the corresponding password is the value corresponding to the key. All user accounts are called a one-dimensional key space. Therefore, this scenario can be abstracted as: given a query key, find the value corresponding to the query key and return it, or given an insertion key and its corresponding value, insert the key-value pair into the corresponding position.

[0026] To solve the aforementioned query and insertion problems, the specific implementation process of the technical solution of this invention is as follows: Figure 1 As shown, it includes the following steps:

[0027] (1) Use the overhead model to construct the key as an RMI structure.

[0028] To adapt to different data distributions, this invention uses an overhead model to achieve adaptive construction. It analyzes factors affecting index performance, including the overhead of traversing from the root node to the leaf node, the overhead of using exponential search within the leaf array, and the overhead of data sliding caused by data insertion. Based on their different degrees of influence, corresponding weights are set. During index construction, the total overhead of each node at each level is calculated until the hierarchical overhead increases continuously. Then, node merging begins from the bottom up. If the overhead saved by merging two nodes is greater than 0, it means that merging these two nodes will produce better performance. This process continues until all nodes have been traversed, and the index construction is complete.

[0029] (2) Use the cumulative distribution function of the data to evenly divide the range of the data in each internal node to ensure that the data space interval in each internal node is the same.

[0030] First, sort the data in ascending order to determine the maximum value max_key and the minimum value min_key. Then, train the internal nodes according to the cumulative distribution function of the data so that all data are evenly distributed among the child nodes according to the range.

[0031] The formula for calculating the slope of the internal node model is:

[0032]

[0033] The formula for calculating the intercept of the internal node model is:

[0034]

[0035] Finally, the formula for the internal node model is obtained:

[0036] y = a × x + b

[0037] Given a query key, we recursively search for the node containing the key using the internal node model until we reach a leaf node. The process of finding the child node of the key can be represented as follows:

[0038] pos=a×fanout×key+b×fanout

[0039] Where pos is the ID of the node to which the query key belongs in the next level, and fanout is the number of child nodes of the current node.

[0040] (3) Construct a monotonic univariate linear regression model with gaps at each leaf node, and use the model to insert keys using a model-guided approach.

[0041] After dividing the data, we fit the data for each leaf node.

[0042] The slope calculation formula for the leaf node model is:

[0043]

[0044] The formula for calculating the intercept of a leaf node model is:

[0045]

[0046] Finally, the model with the leaf nodes trained is obtained:

[0047]

[0048] (4) Learn the data update distribution during the index building process, and reserve gaps for data insertion based on the data update distribution.

[0049] Calculate the amount of data to be inserted into each leaf node (insertkeys), and scale the array according to the overall capacity and density requirements.

[0050] First, calculate the left boundary value of the leaf node:

[0051]

[0052] Where i represents the level number, and level[i-1].irb is the previous node of the current node in the current level.

[0053] The formula for calculating the right boundary value of a leaf node is:

[0054]

[0055] Where fanout represents the fanout number of the layer, i represents the layer number, a and b represent the slope and intercept respectively, and k represents the first key value to insert into the array.

[0056] Finally, the amount of data inserted into the leaf nodes, insertkeys, is calculated as follows:

[0057] insert_num_keys=irb-ilb

[0058] Here, insert_num_keys represents the amount of data inserted into the leaf nodes.

[0059] If the total amount of data in the current leaf node is less than the set capacity threshold, then the array capacity is:

[0060] capacity=(keys+insertkeys) / d nax

[0061] Otherwise, the array capacity is:

[0062] capacity = keys / d

[0063] Where capacity is the array capacity, keys is the initial data size, and d is the array density boundary. max This represents the maximum array density boundary.

[0064] Finally, adjust the model parameters according to the array capacity.

[0065] a = a × (capacity / num_keys)

[0066] b = b × (capacity / num_keys)

[0067] (5) Use exponential search within the leaf nodes to perform key lookup and insertion.

[0068] After predicting the query key position using leaf nodes, determine if the current position is the query key. If it is, directly return the payload value corresponding to the query key. If it is not the query key, use an exponential search forward or backward from the predicted position to find the first key value that is neither greater than nor less than the query key. Finally, use a binary search within a small range to search for the query key. If the query key is found, return its corresponding payload value; otherwise, it means the query key does not exist, return null, and end the query.

[0069] For the insertion key, first perform a point query to determine if the model predicts the insertion position as a gap and if the data will be sorted in ascending order after insertion into the gap. If the condition is met, insert directly into the gap and the insertion ends. If the current predicted position is not a gap, create a gap by moving some data while ensuring ascending order.

[0070] (6) After inserting a certain amount of data, the model is expanded or split to effectively handle subsequent insertions.

[0071] When a certain amount of data is inserted and the model expansion threshold is reached, the insertion is continued by expanding the array capacity and adjusting the model parameters. If data is inserted into the leaf node until the split threshold is reached, the model is triggered to split, the current leaf node becomes an internal node, all data are evenly distributed to the new leaf node according to the data range, the leaf node model is retrained, and the array capacity is adjusted.

[0072] The experimental results of this invention were tested on an Ubuntu virtual machine in the Database Technology Research Laboratory of Shenyang Aerospace University. The experiment used four datasets: the LTD dataset and LAT dataset with floating-point keys, the YCSB dataset with unsigned long integer keys, and the LNM dataset with signed long integer keys.

[0073] The experimental results of this invention are compared with those of other one-dimensional data indexing structures, including B+ trees, ALEX, and LIPP.

[0074] Figure 2 This is a horizontal comparison of the data query and insertion effects of this invention and several other methods. AveLookup Time represents the average lookup time for a point lookup of a certain index, with a value between (0, +∞). The smaller the value, the less time is consumed in executing a query and the better the effect. Ave Insert Time represents the average insertion time for a certain index, with a value between (0, +∞). The smaller the value, the less time is consumed in executing an insertion and the better the effect. Due to space limitations, only two experimental results are shown here.

[0075] The above description of the embodiments is provided to enable those skilled in the art to understand and apply the present invention. Those skilled in the art can readily make various modifications to the above embodiments and apply the general principles described herein to other embodiments without creative effort. Therefore, the present invention is not limited to the above embodiments, and any improvements and modifications made to the present invention by those skilled in the art based on the disclosure thereof should be within the scope of protection of the present invention.

Claims

1. A learning index model oriented to update distribution, characterized in that, The method comprises the following steps: (1) constructing keys into an RMI structure using an overhead model; (2) uniformly dividing the data range at each internal node using the cumulative distribution function of the data, so as to ensure that the data space interval in each internal node is the same; (3) constructing a monotonous gap unary linear regression model at each leaf node, and inserting keys using a model-oriented method based on the model; (4) learning the data update distribution during the construction of the index, and reserving a gap for the insertion of updated data according to the data update distribution; (5) using exponential search in the leaf node to query and insert keys; (6) expanding or splitting the model after a certain amount of data is inserted, so as to effectively process subsequent updates. The specific implementation of step (4) is as follows: first, before batch loading data, the update data is obtained according to the update frequency, and the update data distribution is calculated; during batch loading data, the internal node range is divided using the learned update distribution, and the leaf node capacity is set according to the update distribution, so as to reserve a gap for the insertion of data; if the insertion position predicted by the model is a gap and the data is arranged in order, the data can be directly inserted into the predicted position; otherwise, since the gap reserved for the insertion of data is calculated according to the model-oriented method of step (3), the predicted position of the model is relatively accurate, and the insertion position is determined using exponential search. 2.The learning index model of claim 1, wherein: The specific implementation of step (1) is as follows: first, sort all keys in ascending order, and then calculate the overhead of each layer using the overhead model until the overhead increases; the internal nodes of the RMI structure divide the data range, and the leaf nodes fit and predict the data; the keys are one-dimensional data. 3.The learning index model of claim 1, wherein: The specific implementation of step (2) is as follows: learn the cumulative distribution function of the data, calculate the model parameters of the internal node according to the cumulative distribution function of the data, so that the linear regression model can uniformly divide the data into its corresponding child nodes according to the range, and take the integer part of the model output result as the serial number of the child node to which the key belongs. 4.The learning index model of claim 1, wherein: The specific implementation of step (3) is as follows: train the data in each leaf node using unary linear fitting, input the key value into the linear regression model, take the integer part of the model output result as the predicted position of the model, and when the predicted position is not a gap, move a small amount of data to complete the insertion, so as to ensure that the error between the real position of the key and the predicted position is small, and the gap to which the key belongs is the position of the memory array without storing data.

5. The learning index model of claim 1, wherein: The specific implementation of step (5) is as follows: according to the linear regression model trained in step (3) and the model-oriented insertion method, the real position of the key is close to the predicted position of the model, and exponential search is used starting from the predicted position of the model to achieve fast searching. 6.The learning index model of claim 1, wherein: The specific implementation of step (6) is as follows: when data is frequently inserted in a certain leaf node, the effectiveness of the model decreases, so when the data amount of the array corresponding to the leaf node reaches the expansion threshold, the leaf node model and the array capacity are expanded. When the leaf node data volume reaches the split threshold, the data in the current leaf node is retrained as an internal node, and the data is distributed to the corresponding leaf node according to the range. 7.The learning index model of claim 1, wherein, Comprise the following specific steps: (1) using the overhead model to build the key into the RMI structure, In order to adapt to different data distribution, adaptive construction is realized by using overhead model, the factors affecting the performance of index are analyzed, including the overhead from root node to leaf node, the overhead of using exponential search in leaf array, the overhead of data sliding caused by inserting data, according to their different influence degree, the corresponding weight is set respectively, the overhead sum of each node in each layer is calculated, until the hierarchical overhead increases continuously, the node merging starts from bottom to top, until all nodes are traversed, the index construction is completed; (2) in each internal node, the cumulative distribution function of data is used to uniformly divide the range of data, so as to ensure that the data space interval in each internal node is the same, The data is sorted in ascending order, the maximum value max_key and the minimum value min_key are determined, the internal node is trained according to the cumulative distribution function of data, so that all data is evenly divided into each child node according to the range, The slope calculation formula of internal node model is: ， The intercept calculation formula of internal node model is: ， The internal node model formula is obtained: ， Given the query key key, the internal node model is used to recursively find the node where the key key is located, until the leaf node is reached, and the process of finding the child node where the key is located can be represented as follows: ， Where, pos is the number of the node to which the query key belongs in the next layer, fanout is the number of child nodes of the current node, (3) a monotonous gap unary linear regression model is constructed in each leaf node, and the model is used to insert the key based on the model guided method, After dividing the data, the fitting of each leaf node data is carried out, The slope calculation formula of leaf node model is: ， The intercept calculation formula of leaf node model is: ， The trained leaf node model is obtained: ， (4) learn the data update distribution in the process of building index, and reserve gap for inserting data according to the data update distribution, The data volume insertkeys to be inserted into each leaf node is counted, and the array is scaled according to the overall capacity and density requirements, Firstly, the left boundary value of leaf node is calculated: ， Where, i represents the layer, level[i-1].irb represents the previous node in the current node in the layer, The right boundary value calculation formula of leaf node is: ， Where, fanout represents the fanout number of the layer, i represents the layer, a and b represent the slope and intercept respectively, and k represents the first key value of the inserted array, Finally, the data volume insertkeys inserted into the leaf node is calculated as follows: ， Where, insert_num_keys is the data volume inserted into the leaf node, If the total data volume of the current leaf node is less than the set capacity threshold, the array capacity is: ， Otherwise, the array capacity is: ， where capacity is the array capacity, keys is the initial data amount, d is the array density bound, d max is the maximum array density bound, Finally, the model parameters are adjusted according to the array capacity, ， (5) exponential search is used in the leaf node to query and insert the key, After predicting the position of the query key using the leaf node, it is determined whether the current position is the query key. If it is the query key, the payload value corresponding to the query key is directly returned. If it is not the query key, the first key value not greater than or not less than the query key is found using exponential search from the predicted position, and then binary search is used to search for the query key in a small range. If the query key is found, the payload value corresponding to the query key is returned. Otherwise, it is determined that the query key does not exist, and an empty value is returned, ending the query. For an insertion key, a point query is first performed to determine whether the predicted insertion position of the model is a gap and whether the insertion into the gap satisfies the ascending arrangement. If the condition is satisfied, the insertion is directly performed into the gap, and the insertion is ended. If the current predicted position is not a gap, a gap is created by moving some data to ensure the ascending arrangement, and the insertion is completed. (6) After a certain amount of data is inserted, the model is expanded or split to effectively handle subsequent insertions. When a certain amount of data is inserted to reach the model expansion threshold, the array capacity is expanded and the model parameters are adjusted to continue processing the insertions. When data continues to be inserted in the leaf node to reach the split threshold, the model is split, the current leaf node becomes an internal node, all data is evenly divided into new leaf nodes according to the data range, the leaf node model is retrained, and the array capacity is adjusted.

Images