Efficient learning type index construction method and system for accelerated query in read-only scene

Abstract

The invention discloses an efficient learning type index construction method and system for accelerated query in a read-only scene. The method comprises the following steps: firstly, assuming that a strategy processing data set is constructed by using a distribution simplified model, grouping data keys according to k data keys in each group, taking a first key in each group as a representative key, training by using a PLA algorithm, and calculating an upper limit of the number of segments; defining a two-dimensional dynamic programming array; starting from the first representative key, fitting by using a PLA algorithm, if the fitting is successful, initializing cost information and space size, and if the fitting is failed, using a distribution remapping model strategy; then starting from the initial segment number to the upper limit, enumerating the segment number by the outer layer, enumerating segment points by the inner layer, calculating and updating the minimum space overhead, and storing a result; and finally, obtaining model information from back to front according to the calculated segmentation number, and integrating to obtain an optimal segmentation strategy. The method effectively restrains the error of the read-only learning type index, reduces the query time of the index, and has the advantages of small occupied index space, low index height and small prediction error.

Description

Technical Field

[0001] The present invention belongs to the field of databases, and in particular relates to an efficient learning index construction method and system for accelerating queries in read-only scenarios, which can be used in in-memory databases to improve the query performance of the database system. Background Art

[0002] Traditional databases rely on disks for data storage. However, with the increase in memory capacity and the decrease in the price of memory storage media, in-memory databases have emerged. Compared to traditional databases, in-memory databases can more efficiently utilize CPU cycles and memory, and improve data caching and data operation processes, resulting in faster data processing speeds.

[0003] Learned indexes can be used in in-memory databases as an index structure to improve query performance. Compared to traditional B+ tree indexes, learned indexes can leverage machine learning models to learn the distribution of data, replacing memory access with model calculations, thereby improving index performance. Based on whether they can be modified, learned indexes can be divided into two categories: updateable learned indexes and read-only learned indexes. Updateable learned indexes can be modified based on dynamic data updates, but they occupy a larger amount of space. In contrast, read-only learned indexes occupy very little space, approximately two orders of magnitude smaller than updateable learned indexes, and provide higher query performance. Therefore, in read-only scenarios, database systems tend to use read-only learned indexes.

[0004] Although it can bring about a significant improvement in query performance, read-only learning indexes still have the problem of inconsistent query performance between theory and practice. Currently, most advanced read-only learning indexes mainly focus on one of the two aspects: theoretical query performance and theoretical query time complexity. Read-only learning indexes with better query performance have higher query errors in the worst scenario. Therefore, current read-only learning indexes cannot take into account both query performance and query error at the same time. In the process of building a learning index, the learning method with each key as the granularity will result in a higher number of layers of the learning index, slowing down query performance. In addition, the data distribution of real-world data sets is relatively complex, so the learning model cannot always simplify the data distribution efficiently. The above are all problems encountered by current read-only learning indexes. Summary of the Invention

[0005] In order to solve the problem that query performance and query error of read-only learning indexes cannot be taken into account at the same time, the present invention proposes an efficient learning index construction method and system for accelerating queries in read-only scenarios, which effectively constrains the error of read-only learning indexes, reduces the query time of the index, and occupies little index space.

[0006] In order to achieve the above object of the invention, the technical solution of the present invention is as follows:

[0007] An efficient learning-based index construction method for accelerating queries in read-only scenarios includes the following steps:

[0008] By assuming that a distribution simplification model building strategy is used for the dataset, the upper limit of the number of segmentation is calculated. The distribution simplification model building strategy groups the keys in the dataset into groups of k, and takes the first key of each group as the representative key of each group to participate in the training of the dataset model;

[0009] The data is sorted and divided according to the group granularity, and the representative keys of each group are extracted and stored in the representative key array. A two-dimensional dynamic programming array is defined, and its elements store the cost information of dividing the representative keys into a specified number of segments. Starting from the first key in the representative key array, an approximate linear constraint PLA algorithm is used to attempt to linearly fit the representative keys using a distribution simplification model construction strategy. If the fitting is successful, the corresponding cost information is initialized, including the size of the model space from the beginning of the data set to the current item; if the fitting fails, the subsequent representative keys are initialized using a distribution remapping model construction strategy, and the minimum difference between adjacent representative keys is calculated and updated; the distribution remapping model construction strategy uses an expansion coefficient greater than 1 to expand the representative keys in the data set to a larger space;

[0010] Set the initial value of the number of segments, starting from this value to the upper limit of the number of segments, enumerate the number of segments of each type in the outer loop, and for each segment number, enumerate the segment points of the last segment model in the inner loop. For each segment point, calculate the model construction strategy and related costs of the last segment model, use the PLA algorithm to try to add representative keys, select the appropriate model construction strategy based on the addition results and calculate the space overhead, update the minimum space overhead of the current layer, find the minimum value and save the relevant variables and model information, save the calculation results to the two-dimensional dynamic programming array, and iterate in sequence until the enumeration of all segment numbers is completed;

[0011] From the two-dimensional dynamic programming array, based on the calculated number of data set segments, model information is retrieved from the back to the front. The previous model information is determined by the variables stored in the current model. The loop is traversed until the first layer of the array is reached. The obtained information is integrated to obtain the optimal segmentation strategy of the model.

[0012] Furthermore, the calculation method of the upper limit of the number of segmented segments specifically includes: for each representative key in the representative key array, try to use the PLA algorithm to add it to the model. If the addition is successful, continue to traverse the next representative key, otherwise clear the PLA information and increase the number of segmented segments by 1. After traversing all representative keys, the segment value obtained is the upper limit of the number of segmented segments.

[0013] Furthermore, the two-dimensional dynamic programming array is defined as dp, where dp[i][j] represents the cost information required to divide the representative key with subscript [0,j] in the representative key array into i+1 segments, and the state transition equation for calculating dp[i][j] is dp[i][j]=min(dp[i-1][jl]+train(l+1,j)), l={p,p+1,…,j-1}, where l represents the optimal segmentation point of the previous model, p represents the number of segments in the current segment, and train(l+1,j) represents the cost related to model construction on the interval [l+1,j].

[0014] Furthermore, during the iterative calculation process, the appropriate model building strategy is selected based on the joining results and the space cost is calculated, including:

[0015] Using variable l, start traversing from subscript j-1 to p, and divide the interval [0,j] into two segments: [0,l] and [l+1,j];

[0016] If the optimal segmentation model of [0, l] in the previous layer exists, then calculate the model construction strategy and model-related costs on the interval [l+1, j]. Starting from l+1, traverse to j, and try to use the PLA algorithm to add the representative key on the interval. If the representative key is added successfully, then continue with the next representative key, otherwise the representative key fails to be added; when a representative key fails to be added, stop using the PLA algorithm, and use the distributed remapping model construction strategy to build the model for the current interval [l+1, j]. Calculate the corresponding space overhead and mark the current interval using the remapping construction strategy; if all representative keys can be added to the model through the PLA algorithm, then use the distributed simplification model construction strategy for the current interval, calculate the corresponding space overhead and mark it.

[0017] Furthermore, when initializing the two-dimensional dynamic programming array, if the distribution simplification model construction strategy is successfully fitted, the space size s is initialized to j+1; if the distribution remapping model construction strategy is successfully fitted, the space size s is initialized to Gap is the minimum difference between adjacent representative keys, and key[j] represents the jth representative key.

[0018] Furthermore, the model parameters in the distribution remapping model construction strategy are determined by key constraints and parameter tuning. For a key x, the expression of the corresponding optimal model m(x) is Where s1 is the first representative key, Represents the minimum value between two adjacent representative keys, obtained by a single pass through the array of representative keys.

[0019] Furthermore, the specific method for obtaining the optimal segmentation strategy of the model is as follows: extract the corresponding model information from the dp array and count it. Assuming that the number of segments of the data set is finally calculated to be p, start from dp[p-1][n-1] and extract the corresponding model information from back to front. The information of the previous model is determined by the value of the variable l stored in the current model, that is, the model information currently extracted is on the interval [l+1,n-1]. The model information of the next layer is located in dp[p-2][l], and so on, until traversing to the first layer of the dp array, integrating the obtained information, and obtaining the optimal segmentation model.

[0020] An efficient learning-based index building system for accelerating queries in read-only scenarios, including:

[0021] A segment number upper limit determination module is used to calculate the upper limit of the segment number by assuming that a distribution simplification model construction strategy is used for the dataset. The distribution simplification model construction strategy groups the keys in the dataset into groups of k, and takes the first key of each group as the representative key of each group to participate in the training of the dataset model;

[0022] A dynamic programming array initialization module is used to sort the data and divide it by group granularity, extract the representative key of each group and store it in the representative key array, define a two-dimensional dynamic programming array, whose elements store the cost information of dividing the representative key into a specified number of segments, and use the approximate linear constraint PLA algorithm to try to perform linear fitting on the representative key using the distribution simplification model construction strategy starting from the first key in the representative key array. If the fitting is successful, the corresponding cost information is initialized, including the size of the model space from the beginning of the data set to the current item; if the fitting fails, the subsequent representative keys are initialized using the distribution remapping model construction strategy, and the minimum difference between adjacent representative keys is calculated and updated; the distribution remapping model construction strategy uses an expansion coefficient greater than 1 to expand the representative keys in the data set to a larger space;

[0023] The iterative calculation module is used to set the initial value of the number of segments, starting from this value to the upper limit of the number of segments, enumerating the number of segments of each type in the outer loop, and for each segment number, enumerating the segment points of the last segment model in the inner loop. For each segment point, the model construction strategy and related costs of the last segment model are calculated, and the PLA algorithm is used to try to add representative keys. According to the addition results, the appropriate model construction strategy is selected and the space overhead is calculated. The minimum space overhead of the current layer is updated, the minimum value is found and the relevant variables and model information are saved. The calculation results are saved in a two-dimensional dynamic programming array, and it is iterated in sequence until the enumeration of all segment numbers is completed.

[0024] The optimal strategy acquisition module is used to retrieve model information from the two-dimensional dynamic programming array from the back to the front based on the calculated number of segments of the data set. The previous segment of model information is determined by the variables stored in the current model. The module loops until it traverses the first layer of the array, integrates the obtained information, and obtains the optimal segmentation strategy of the model.

[0025] The present invention also provides an electronic device comprising: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and when the programs are executed by the processors, the steps of the efficient learning-based index construction method for accelerating queries in read-only scenarios as described above are implemented.

[0026] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-mentioned method for constructing an efficient learning-based index for accelerating queries in a read-only scenario.

[0027] Beneficial effects: The present invention fully considers the relationship between index query performance and index query error in read-only learning indexes, and constrains the query error of the learning model to be within two groups by relaxing the learning granularity to k and linking the query error ε with the granularity through the distribution simplification model construction strategy. By expanding the distribution remapping model construction strategy of the spatially adjusted mapping distribution, multiple segments are merged into one, simplifying the distribution of the data set and thus improving the query performance. Through the dynamic programming algorithm, the distribution simplification and distribution remapping model construction strategies are combined according to the actual specific data set to calculate the learning index model with the minimum number of segments, which is also the model in the optimal case. It has been proved through theory and experiment that the method of the present invention effectively adjusts and simplifies the data distribution, so that the number of models in each layer after training is reduced and the number of index layers is reduced, which reduces the number of memory accesses and thus improves the query speed. At the same time, the present invention effectively constrains the error of the read-only learning index, reduces the query time of the index, and has the advantages of small index space occupancy, low index height and small prediction error. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 Flowchart of the dynamic programming algorithm in the present invention. DETAILED DESCRIPTION

[0029] The present invention will be further described below with reference to the accompanying drawings and specific implementation examples. It should be understood that these embodiments are only intended to illustrate the present invention and are not intended to limit the scope of the present invention. After reading this invention, modifications of various equivalent forms of the present invention made by those skilled in the art all fall within the scope defined by the appended claims.

[0030] This invention provides an efficient learning-based index construction method for accelerating queries in read-only scenarios. This method relies on an optimal segmentation model construction strategy that combines distribution simplification and distribution remapping model construction strategies. This strategy relies on a dynamic programming algorithm to construct a model that minimizes the number of segments p, given a learning-based index space size M. The following first introduces the distribution simplification model and the distribution remapping model construction strategies, and then provides a specific implementation process for implementing the optimal segmentation model construction strategy that combines these two strategies using a dynamic programming algorithm.

[0031] (1) Distribution Simplification Model Construction Strategy: Relax the granularity of the learning model from 1 to k, thereby simplifying the model distribution of the dataset. This strategy groups the keys in the dataset into groups of k, takes the first key of each group as the representative key of each group, and participates in the training of the dataset model. Through the representative key of each group and the current partitioning granularity, the prediction error range of all key values in the dataset is constrained. By relaxing the granularity to k, the time complexity of model construction is reduced. In practical applications, k is generally a value greater than 1, and is taken as the error value mentioned later. In the dynamic programming algorithm, the value of k is determined by specifying the error value.

[0032] In this strategy, the keys in the dataset are grouped into groups of k, and the first key in each group is used as the representative key for the dataset model training. Specifically, the dataset is initialized to be ordered by default. Starting with the first key, each consecutive k keys in the dataset are grouped together, with the first key being the representative key. Since each group contains only data keys and no blank keys, these groups are dense and are defined as dense groups. When using the approximate linear constraint algorithm (PLA algorithm) for model training, only the representative keys extracted from each dense group are used for model training operations.

[0033] In this strategy, the prediction error range of all key values in the dataset is constrained by the representative key of each group and combined with the current partitioning granularity. Specifically, constraining the prediction error range of all key values in the dataset mainly includes two aspects, namely, constraining the error range of representative keys and the error range of non-representative keys. For representative keys, the fitted linear model is calculated by the PLA algorithm, so that the linear model can constrain the error of each representative key within the specified range. For non-representative keys, ideally, their mapping values through the model should be constrained within their corresponding groups. By adding or subtracting an error value ε and setting ε to k, the predicted group position and the actual group position can be constrained within the two groups.

[0034] This strategy reduces the time complexity of model construction by relaxing the granularity to k. Specifically, when calculating the linear model using the PLA algorithm, a linear scan of the data set is required, and the time complexity is O(n). After relaxing the granularity to k, the number of keys that need to be scanned is Therefore, the time complexity is reduced to That is, it is reduced by k times.

[0035] (2) Distribution remapping model construction strategy: map the representative keys in the model to a larger space, refit the distribution of these keys and fill some blank keys, so that these keys can be summarized by a linear model within the error constraint range, and the space occupied by the linear model is minimized. The model parameters in the distribution remapping model construction strategy are determined by key constraints and parameter tuning.

[0036] The specific implementation logic of the distributed remapping model construction strategy is as follows: for a dataset of size |D|, expand it to a larger space |D|·α (α>1). The keys in this dataset are summarized through a larger linear model, and some blank keys may be filled in. This ensures that the linear model constructed from these data keys can meet the error constraints and fit the data keys using the least space possible. Each group here contains blank keys and is therefore sparse. These groups are defined as sparse groups. Since the position of the representative key relative to each sparse group may change after expansion, the mapping value of the representative key in the model must be within its corresponding sparse group.

[0037] In this strategy, model parameters are determined by key constraints and parameter tuning, as follows: For the distribution remapping model construction strategy, its construction goal is to select appropriate parameters a and b so that m(s n )=a·(s n )+b is the smallest, where s n is the largest data key, m(s n ) is the number of groups in the entire model. i ,i∈{1,k+1,2k+1,…}, we can get m(s i+k )≥m(s i )+1, m(s i ) is expanded to get a·s i+k +b≥a·s i +b+1, so we have remember represents the minimum value between two adjacent representative keys, so And because a.s i +b ≥ 0, so b ≥ -a s i ≥-a·s1. Set a and b to their endpoint values and -a·s1, and finally derive that for a data key x, the expression of the corresponding optimal model m(x) is in It can be obtained by traversing the array of representative keys in one pass.

[0038] (3) The optimal segmentation model construction strategy that combines the distribution simplification model and distribution remapping model construction strategies: Each data key can be fitted using one of the two strategies, and the distribution simplification model construction strategy is preferentially selected. The number of segments of each possible segment is traversed, and based on the number of segments, the model used for each segment under the current segmentation strategy and the size of the space used by the model are calculated. The model size calculated in the intermediate process is saved, and the model used by the current data set is calculated through a dynamic programming algorithm when the number of segments is minimized.

[0039] In this strategy, the specific performance of the dynamic programming algorithm is as follows: first, it is assumed that the entire data set uses the distribution simplification model construction strategy, and the maximum number of segments that need to be segmented is calculated. Then, the first layer of the two-dimensional dynamic programming array is initialized, and the size of the space used by the model from the beginning of the data set to the current item is calculated. Finally, the outer loop enumerates the number of segments of each segmentation, and the inner loop enumerates the segmentation points of the last segment of the model and calculates the minimum space size of the last segment of the model. The calculated results are saved in the dynamic programming array, and it is iterated in sequence to finally obtain the optimal solution of the model.

[0040] Among them, the state transition equation of the dynamic programming algorithm is as follows: define a two-dimensional dynamic programming array dp, where dp[i][j] represents the cost information required to divide the representative key with the subscript [0,j] in the representative key array into i+1 segments, and the state transition equation for calculating dp[i][j] is dp[i][j]=min(dp[i-1][jl]+train(l+1,j)), l={p,p+1,…,j-1}, where l represents the optimal segmentation point of the previous model, p represents the number of segments in the current segment, and train(l+1,j) represents the cost related to model construction on the interval [l+1,j].

[0041] The optimal segmentation model derived by the dynamic programming algorithm has the following characteristics: the model has the smallest number of segments, each segment in the model is constructed using a distribution simplification or distribution remapping model construction strategy, and the segments of the model have no intersection with each other. The overall space occupancy of the model is not necessarily the smallest among all constructible models, but it can meet the given space and learning granularity requirements.

[0042] Reference Figure 1 The present invention implements the construction process of the learning index through the dynamic programming algorithm, which mainly includes the following three steps:

[0043] Step 1: Initialization phase of the optimal segmentation model strategy.

[0044] Specifically include:

[0045] In step 1.1, initialize the required index space size M and the query error ε, among other variables. Since the values mapped to non-representative keys by the model should be constrained to be within their corresponding groups, set ε to k to constrain the predicted and actual group positions to be within two groups. After sorting the data in the system, partition it according to the granularity k. For the entire dataset, starting with the first data key, extract the 1st, k+1st, 2k+1st, …, data keys. These data keys are the representative keys for each group and are stored in the representative key array key.

[0046] Step 1.2, define a two-dimensional dynamic programming array dp, where dp[i][j] represents the cost information required to divide the representative key with the subscript [0,j] in the representative key array into i+1 segments. The cost information includes the segmentation point j of the current layer, the optimal segmentation point l of the previous segment, the space size s used to divide the representative key corresponding to the interval [0,j] into i+1 segments, and whether the model building strategy used for the last segment [l+1,j] of the representative key corresponding to the interval [0,j] is the distribution simplification or distribution remapping model building strategy.

[0047] Step 1.3, initialize the first layer of the two-dimensional dynamic programming array dp, that is, all values of dp[0][j]. dp[0][j] is the cost information required to divide the representative key with the subscript [0,j] of the representative key array into 1 segment. Therefore, start traversing from the first key of the representative key array, and use the PLA algorithm to try to linearly fit the representative key with the distribution simplification model construction strategy. If the fitting is successful, initialize the corresponding cost information, initialize the space size s to j+1, and initialize the model construction strategy to the distribution simplification model construction strategy. When the fitting fails, all subsequent representative keys are initialized with the distribution remapping model construction strategy. During the model construction process, calculate the minimum value gap between adjacent representative keys and continuously update it. For the distribution remapping model construction strategy, the space size s in dp[0][j] is initialized to Where key[j] represents the jth representative key.

[0048] Step 1.4, calculate the upper limit of the number of segments. According to the representative key array in step 1.1, traverse it once. For each representative key in the array, try to add it to the model using the PLA algorithm. If the addition is successful, continue to traverse the next representative key. Otherwise, clear the PLA information and increase the number of segments by 1. After traversing all the representative keys, the segment value obtained is the upper limit of the number of segments. The principle of this process is to continuously try to use the PLA algorithm to add representative keys to the model. In essence, it is to fit the representative keys using the distribution simplification model construction strategy. When the entire data set uses the distribution simplification model construction strategy, the number of model segments is the largest.

[0049] Step 2: The construction phase of the optimal segmentation model strategy.

[0050] Specifically include:

[0051] Step 2.1, set the number of segments p to 2, starting from 2 and ending at upper, and enumerate the number of segments.

[0052] In step 2.2, check the segmentation of the previous layer. Extract the last element of the previous layer from the dp array and check its space usage. If a segmented model exists and the space of the segmented model meets the predetermined space requirements, then the optimal segmentation model for the current dataset has been calculated. Skip the remaining steps in step 2 and proceed to step 3.

[0053] Step 2.3, calculate the model status of the current layer.

[0054] Specifically include:

[0055] In step 2.3.1, traverse the second dimension of the current layer from index j, starting at p and traversing to n, and calculate the optimal segmentation model in the interval [0, j]. Specifically, use variable l to traverse from index j-1 to p, dividing the interval [0, j] into two segments: [0, l] and [l+1, j].

[0056] Step 2.3.2, calculate the model building strategy and model-related costs on the interval [l+1,j]. If the optimal segmentation model of [0,l] in the previous layer exists, then calculate the model building strategy and model-related costs on the interval [l+1,j]. Starting from l+1, traverse to j, and try to use the PLA algorithm to add the representative key on the interval. If the representative key is successfully added, then continue with the next representative key, otherwise the representative key fails to be added. When a representative key fails to be added, stop using the PLA algorithm, and use the distributed remapping model building strategy to build the model for the current interval [l+1,j]. Calculate the corresponding space overhead and mark the current interval using the remapping construction strategy; otherwise, all representative keys can be added to the model through the PLA algorithm, and the current interval can use the distributed simplification model building strategy to calculate the corresponding space overhead and mark it.

[0057] Step 2.3.3: Update the minimum space cost of the current layer. After completing step 2.3.2, calculate the space cost of the current model. This is the sum of the space cost of the interval [0, l] in the previous layer model and the space cost of the interval [l+1, j] calculated in step 2.3.2. Based on the results of the current layer traversal in step 2.3.1, calculate the minimum space cost of the current layer and save the corresponding variable l, the current minimum space size, and the model information used.

[0058] In step 2.3.4, record the calculation results of the current layer. Based on the calculation results in step 2.3.3, if the space overhead of the current layer is the minimum, save the results of the current layer to the dp array. Otherwise, set the current layer to be unable to train a model that meets the requirements, and also save it to the dp array.

[0059] Step 2.4: Continue the loop of step 2. Add 1 to the number of segments p and repeat step 2.2.

[0060] Step 3: Statistical phase of the optimal segmentation model strategy.

[0061] Specifically, the corresponding model information is retrieved from the dp array and counted. Assuming that the final calculated number of segments of the dataset is p, starting from dp[p-1][n-1], the corresponding model information is retrieved from the back to the front. The information of the previous model is determined by the value of the variable l stored in the current model. That is, the model information currently retrieved is in the interval [l+1,n-1]. The model information of the next layer is located in dp[p-2][l], and so on, until the first layer of the dp array is traversed. The obtained information is integrated to obtain the optimal segmentation model.

[0062] Furthermore, the performance of the efficient learning indexing method for accelerating queries in read-only scenarios described in the present invention was verified through comparative experiments. All programs in the experiment were implemented in C++, and the O3 optimization of the g++ compiler was turned on at the same time. The configuration of the test machine is as follows: Intel(R) Xeon(R) Gold 6226R CPU@2.90GHz, 32GB RAM. The comparison algorithms are read-only learning indexes RMI, PGM and RS and traditional B+ tree index BTree. The Amazon data set and different index space sizes were used for testing, and the results are shown in Table 1, where the first column represents the index space size, the first row represents the index name, and the middle content represents the query time. When the index size is close, the query time of the method of the present invention is the shortest, and the query speed is improved by 15%-120% compared with other learning indexes, and 3-5 times compared with traditional B+ tree indexes.

[0063] Table 1 Comparison of query time between the present invention and several advanced index structures

[0064] The present invention RMI PGM RS BTree 0.1MB 243ns 284ns 360ns 285ns 782ns 1MB 181ns 249ns 354ns 246ns 624ns 10MB 129ns 214ns 295ns 202ns 560ns 100MB 118ns 210ns 271ns 188ns 538ns

[0065] In summary, the present invention proposes an efficient learning indexing method for accelerating queries in read-only scenarios, which not only has the best query performance among read-only learning indexes, but also constrains the query error, and theoretically proves the effectiveness of the query error constraint. Through the distribution simplification and distribution remapping model construction strategies, the learning granularity of the learning index is optimized, and the distribution of the data is simplified. Through the optimal segmentation model strategy, combined with the dynamic programming algorithm, the number of segments of the model segment constructed by the data set is minimized, which is efficient and can also meet the specified space requirements. The method of the present invention effectively constrains the error of the read-only learning index, reduces the query time of the index, and has the advantages of small index space occupancy, low index height and small prediction error.

[0066] The present invention also provides an efficient learning index construction system for accelerating queries in read-only scenarios, comprising:

[0067] A segment number upper limit determination module is used to calculate the upper limit of the segment number by assuming that a distribution simplification model construction strategy is used for the dataset. The distribution simplification model construction strategy groups the keys in the dataset into groups of k, and takes the first key of each group as the representative key of each group to participate in the training of the dataset model;

[0068] A dynamic programming array initialization module is used to sort the data and divide it by group granularity, extract the representative key of each group and store it in the representative key array, define a two-dimensional dynamic programming array, whose elements store the cost information of dividing the representative key into a specified number of segments, and use the PLA algorithm to try to perform linear fitting on the representative key using the distribution simplification model construction strategy starting from the first key in the representative key array. If the fitting is successful, the corresponding cost information is initialized, including the size of the model space from the beginning of the data set to the current item; if the fitting fails, the subsequent representative keys are initialized using the distribution remapping model construction strategy, and the minimum difference between adjacent representative keys is calculated and updated; the distribution remapping model construction strategy uses an expansion coefficient greater than 1 to expand the representative keys in the data set to a larger space;

[0069] The iterative calculation module is used to set the initial value of the number of segments, starting from this value to the upper limit of the number of segments, enumerating the number of segments of each type in the outer loop, and for each segment number, enumerating the segment points of the last segment model in the inner loop. For each segment point, the model construction strategy and related costs of the last segment model are calculated, and the PLA algorithm is used to try to add representative keys. According to the addition results, the appropriate model construction strategy is selected and the space overhead is calculated. The minimum space overhead of the current layer is updated, the minimum value is found and the relevant variables and model information are saved. The calculation results are saved in a two-dimensional dynamic programming array, and it is iterated in sequence until the enumeration of all segment numbers is completed.

[0070] The optimal strategy acquisition module is used to retrieve model information from the two-dimensional dynamic programming array from the back to the front based on the calculated number of segments of the data set. The previous segment of model information is determined by the variables stored in the current model. The module loops until it traverses the first layer of the array, integrates the obtained information, and obtains the optimal segmentation strategy of the model.

[0071] The present invention also provides an electronic device comprising: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and when the programs are executed by the processors, the steps of the efficient learning-based index construction method for accelerating queries in read-only scenarios as described above are implemented.

[0072] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-mentioned method for constructing an efficient learning-based index for accelerating queries in a read-only scenario.

[0073] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, devices (systems), electronic devices, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0074] The present invention is described with reference to flowcharts of methods according to embodiments of the present invention. It should be understood that each process in the flowcharts and combinations of processes in the flowcharts can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts. Figure 1 A device that specifies functions in a process or multiple processes.

[0075] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A function specified in a process or multiple processes.

[0076] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 The steps of a specified function in a process or multiple processes.

[0077] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered by the scope of protection of the claims of the present invention.

Claims

1. An efficient learning-based index construction method for accelerating queries in read-only scenarios, characterized in that: The following steps are involved: By assuming that a distribution simplification model building strategy is used for the dataset, the upper limit of the number of segmentation is calculated. The distribution simplification model building strategy groups the keys in the dataset into groups of k, and takes the first key of each group as the representative key of each group to participate in the training of the dataset model; Sort the data and divide it into groups at the granularity. Extract the representative key of each group and store it in a representative key array. Define a two-dimensional dynamic programming array whose elements store the cost information of dividing the representative key into a specified number of segments. Starting from the first key in the representative key array, use the approximate linear constraint PLA algorithm to try to linearly fit the representative key using the distribution simplification model construction strategy. If the fit is successful, initialize the corresponding cost information, including the size of the model space from the beginning of the data set to the current item. After the fitting fails, the subsequent representative keys are initialized using a distribution remapping model construction strategy, and the minimum difference between adjacent representative keys is calculated and updated; the distribution remapping model construction strategy uses an expansion coefficient greater than 1 to expand the representative keys in the data set to a larger space; Set the initial value of the number of segments, starting from this value to the upper limit of the number of segments, enumerate the number of segments of each type in the outer loop, and for each segment number, enumerate the segment points of the last segment model in the inner loop. For each segment point, calculate the model construction strategy and related costs of the last segment model, use the PLA algorithm to try to add representative keys, select the appropriate model construction strategy based on the addition results and calculate the space overhead, update the minimum space overhead of the current layer, find the minimum value and save the relevant variables and model information, save the calculation results to the two-dimensional dynamic programming array, and iterate in sequence until the enumeration of all segment numbers is completed; From the two-dimensional dynamic programming array, based on the calculated number of data set segments, model information is retrieved from the back to the front. The previous model information is determined by the variables stored in the current model. The loop is traversed until the first layer of the array is reached. The obtained information is integrated to obtain the optimal segmentation strategy of the model.

2. The method according to claim 1, characterized in that The calculation method of the upper limit of the number of segmented segments specifically includes: for each representative key in the representative key array, try to add it to the model using the PLA algorithm. If the addition is successful, continue to traverse the next representative key. Otherwise, clear the PLA information and increase the number of segmented segments by 1. After traversing all representative keys, the segment value obtained is the upper limit of the number of segmented segments.

3. The method according to claim 1, characterized in that The two-dimensional dynamic programming array is defined as dp, where dp[i][j] represents the cost information required to divide the representative key with subscript [0,j] in the representative key array into i+1 segments. The state transition equation for calculating dp[i][j] is dp[i][j]=min(dp[i-1][jl]+train(l+1,j)), l={p,p+1,…,j-1}, where l represents the optimal segmentation point of the previous model, p represents the number of segments in the current segment, and train(l+1,j) represents the cost related to model construction on the interval [l+1,j].

4. The method according to claim 3, characterized in that During the iterative calculation process, the appropriate model building strategy is selected based on the joining results and the space cost is calculated, including: Using variable l, start traversing from subscript j-1 to p, and divide the interval [0,j] into two segments: [0,l] and [l+1,j]; If the optimal segmentation model of [0, l] in the previous layer exists, then calculate the model construction strategy and model-related costs on the interval [l+1, j]. Starting from l+1, traverse to j, and try to use the PLA algorithm to add the representative key on the interval. If the representative key is added successfully, then continue with the next representative key, otherwise the representative key fails to be added; when a representative key fails to be added, stop using the PLA algorithm, and use the distributed remapping model construction strategy to build the model for the current interval [l+1, j]. Calculate the corresponding space overhead and mark the current interval using the remapping construction strategy; if all representative keys can be added to the model through the PLA algorithm, then use the distributed simplification model construction strategy for the current interval, calculate the corresponding space overhead and mark it.

5. The method according to claim 1, wherein When initializing the two-dimensional dynamic programming array, if the distribution simplification model construction strategy is successfully fitted, the space size s is initialized to j+1; if the distribution remapping model construction strategy is successfully fitted, the space size s is initialized to Gap is the minimum difference between adjacent representative keys, and key[j] represents the jth representative key.

6. The method according to claim 1, characterized in that The model parameters in the distribution remapping model construction strategy are determined by key constraints and parameter tuning. For a key x, the expression of the corresponding optimal model m(x) is Where s1 is the first representative key, Represents the minimum value between two adjacent representative keys, obtained by a single pass through the array of representative keys.

7. The method according to claim 3, characterized in that The specific method for obtaining the optimal segmentation strategy of the model is as follows: extract the corresponding model information from the dp array and count it. Assuming that the final calculated number of segments of the data set is p, start from dp[p-1][n-1] and extract the corresponding model information from back to front. The information of the previous model is determined by the value of the variable l stored in the current model, that is, the model information currently extracted is on the interval [l+1,n-1]. The model information of the next layer is located in dp[p-2][l], and so on, until traversing to the first layer of the dp array, integrating the obtained information, and obtaining the optimal segmentation model.

8. An efficient learning index construction system for accelerating queries in read-only scenarios, characterized by: include: A segment number upper limit determination module is used to calculate the upper limit of the segment number by assuming that a distribution simplification model construction strategy is used for the dataset. The distribution simplification model construction strategy groups the keys in the dataset into groups of k, and takes the first key of each group as the representative key of each group to participate in the training of the dataset model; The dynamic programming array initialization module is used to sort the data and divide it into groups by granularity, extract the representative key of each group and store it in the representative key array. A two-dimensional dynamic programming array is defined, whose elements store the cost information of dividing the representative key into a specified number of segments. Starting from the first key in the representative key array, the approximate linear constraint PLA algorithm is used to attempt to perform a linear fit on the representative key using the distribution simplification model construction strategy. If the fit is successful, the corresponding cost information is initialized, including the size of the model space used from the beginning of the data set to the current item. After the fitting fails, the subsequent representative keys are initialized using a distribution remapping model construction strategy, and the minimum difference between adjacent representative keys is calculated and updated; the distribution remapping model construction strategy uses an expansion coefficient greater than 1 to expand the representative keys in the data set to a larger space; The iterative calculation module is used to set the initial value of the number of segments, starting from this value to the upper limit of the number of segments, enumerating the number of segments of each type in the outer loop, and for each segment number, enumerating the segment points of the last segment model in the inner loop. For each segment point, the model construction strategy and related costs of the last segment model are calculated, and the PLA algorithm is used to try to add representative keys. According to the addition results, the appropriate model construction strategy is selected and the space overhead is calculated. The minimum space overhead of the current layer is updated, the minimum value is found and the relevant variables and model information are saved. The calculation results are saved in a two-dimensional dynamic programming array, and it is iterated in sequence until the enumeration of all segment numbers is completed. The optimal strategy acquisition module is used to retrieve model information from the two-dimensional dynamic programming array from the back to the front based on the calculated number of segments of the data set. The previous segment of model information is determined by the variables stored in the current model. The module loops until it traverses the first layer of the array, integrates the obtained information, and obtains the optimal segmentation strategy of the model.

9. An electronic device, characterized in that: include: one or more processors; Memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and when the programs are executed by the processors, the steps of the efficient learning index construction method for accelerating queries in read-only scenarios as described in any one of claims 1-7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the efficient learning index construction method for accelerating queries in a read-only scenario are implemented as described in any one of claims 1 to 7.

Images