An efficient learning-based index construction method and system for accelerating queries in read-only scenarios

Abstract

The application discloses a high-efficiency learning index construction method and system for accelerating query in a read-only scenario. The method first assumes that a distributed simplified model construction strategy processes a data set, divides data keys into groups of k, takes the first key of each group as a representative key, trains by using a PLA algorithm, and calculates an upper limit of a segment number; then a two-dimensional dynamic programming array is defined; the PLA algorithm is used to fit from the first representative key, and if successful, initialization cost information and space size are obtained, and if failed, a distributed remapping model strategy is used; then from the initial segment number to the upper limit, the outer layer enumerates the segment number, the inner layer enumerates the segment point, the minimum space cost is calculated and updated, and the result is saved; finally, the model information is taken from the rear to the front according to the calculated segment number, and the optimal segmentation strategy is obtained by integration. The application effectively constrains the error of the read-only learning index, reduces the query time of the index, and has the advantages of small index space occupation, low index height and small prediction error.

Description

Technical Field

[0001] This invention belongs to the field of databases, and in particular relates to an efficient learning index construction method and system for accelerating queries in read-only scenarios. It can be used in in-memory databases to improve the query performance of database systems. Background Technology

[0002] Traditional databases rely on disks for data storage, but with the increase in memory capacity and the decrease in the price of memory storage media, in-memory databases have emerged. Compared with traditional databases, in-memory databases can utilize CPU cycles and memory more efficiently, and have improved data caching and data operation processes, thus enabling them to have faster data processing speeds.

[0003] Learning indexes can be used in in-memory databases to improve query performance as an index structure. Compared to traditional B+ tree indexes, learning indexes utilize machine learning models to learn the distribution of data, replacing memory access with model computation, thereby improving index performance. Based on whether they can be modified, learning indexes can be divided into two categories: updatable learning indexes and read-only learning indexes. Updatable learning indexes can be modified according to dynamic data updates, but they occupy a larger space. In contrast, read-only learning indexes occupy a very small space, approximately two orders of magnitude smaller than updatable learning indexes, and provide higher query performance. Therefore, in read-only scenarios, database systems tend to use read-only learning indexes.

[0004] While offering significant query performance improvements, read-only learning indexes still suffer from inconsistencies between theoretical and practical query performance. Currently, most advanced read-only learning indexes primarily focus on either theoretical query performance or theoretical query time complexity. A high-performance read-only learning index may exhibit high query error in worst-case scenarios, meaning that current read-only learning indexes cannot simultaneously achieve both high performance and low error. Furthermore, the key-based learning approach during index construction leads to a high number of learning index layers, slowing down query performance. Additionally, real-world datasets have complex data distributions, meaning that learning models cannot always efficiently simplify data distributions. These are all current challenges encountered by read-only learning indexes. Summary of the Invention

[0005] To address the issue of balancing query performance and query error in read-only learning indexes, this invention proposes an efficient learning index construction method and system for accelerating queries in read-only scenarios. This method effectively constrains the error of read-only learning indexes, reduces index query time, and minimizes index space usage.

[0006] To achieve the above-mentioned objectives, the technical solution of the present invention is as follows:

[0007] An efficient learning index construction method for accelerating queries in read-only scenarios includes the following steps:

[0008] By assuming that a distribution simplification model construction strategy is used on the dataset, the upper limit of the number of segments is calculated. The distribution simplification model construction strategy groups the keys in the dataset into groups of k, and takes the first key of each group as the representative key of each group to participate in the training of the dataset model.

[0009] The data is sorted and divided into groups. Representative keys for each group are extracted and stored in a representative key array. A two-dimensional dynamic programming array is defined, whose elements store the cost information of dividing the representative key into a specified number of segments. Starting from the first key in the representative key array, the approximate linear constraint PLA algorithm is used to attempt to linearly fit the representative key with a distributional simplification model construction strategy. If the fit is successful, the corresponding cost information is initialized, including the size of the model space used from the beginning of the dataset to the current item. If the fit fails, subsequent representative keys are initialized with a distributional remapping model construction strategy, and the minimum difference between adjacent representative keys is calculated and updated. The distributional remapping model construction strategy uses an expansion coefficient greater than 1 to expand the representative keys in the dataset into a larger space.

[0010] Set an initial value for the number of segments. Starting from this value, enumerate the number of segments for each type of segmentation in the outer loop. For each segment number, enumerate the segmentation point of the last segment model in the inner loop. For each segment point, calculate the model building strategy and related costs of the last segment model. Use the PLA algorithm to try to add representative keys. Select an appropriate model building strategy and calculate the space cost based on the addition results. Update the minimum space cost of the current layer. Find the minimum value and save the relevant variables and model information. Save the calculation results to a two-dimensional dynamic programming array. Iterate in this way until the enumeration of the number of segments for all segments is completed.

[0011] From the two-dimensional dynamic programming array, based on the calculated number of segments in the dataset, the model information is retrieved sequentially from the end to the beginning. The information of the previous segment is determined by the variables stored in the current model. This process is repeated until the first level of the array is reached. The retrieved information is then integrated to obtain the optimal segmentation strategy for the model.

[0012] Furthermore, the calculation method for the upper limit of the number of segments specifically includes: for each representative key in the representative key array, try to add it to the model using the PLA algorithm. If the addition is successful, continue to traverse the next representative key; otherwise, clear the PLA information and increment the number of segments by 1. After traversing all representative keys, the obtained segment value is the upper limit of the number of segments.

[0013] Furthermore, the two-dimensional dynamic programming array is defined as dp, where dp[i][j] represents the cost information required to divide the representative key with index [0,j] in the key array into i+1 segments. The state transition equation for calculating dp[i][j] is dp[i][j]=min(dp[i-1][jl]+train(l+1,j)), l={p,p+1,…,j-1}, where l represents the optimal segment point of the previous model segment, p represents the number of segments in the current segment, and train(l+1,j) represents the cost related to model construction on the interval [l+1,j].

[0014] Furthermore, during the iterative computation process, an appropriate model building strategy is selected based on the input results, and the space overhead is calculated, specifically including:

[0015] By iterating from index j-1 to p using variable l, the interval [0,j] is divided into two segments: [0,l] and [l+1,j].

[0016] If the optimal segmentation model for [0, l] in the previous layer exists, then calculate the model construction strategy and related costs for the interval [l+1, j]. Starting from l+1, iterate to j, attempting to add representative keys to this interval using the PLA algorithm. If the addition of a representative key is successful, continue to the next representative key; otherwise, the addition of a representative key fails. When any representative key fails to be added, stop using the PLA algorithm. The current interval [l+1, j] is constructed using the distributed remapping model construction strategy, its corresponding space cost is calculated, and the current interval is marked as using the remapping construction strategy. If all representative keys can be added to the model using the PLA algorithm, then the current interval is constructed using the distributed simplified model construction strategy, its corresponding space cost is calculated, and it is marked.

[0017] Furthermore, during the initialization of the two-dimensional dynamic programming array, for the case where the distributed simplification model construction strategy is successfully fitted, the space size s is initialized to j+1; for the distributed remapping model construction strategy, the space size s is initialized to... gap is the minimum difference between adjacent representative keys, and key[j] represents the j-th representative key.

[0018] Furthermore, the model parameters in the distributed remapping model construction strategy are determined through key constraints and parameter tuning. For a key x, the expression for its corresponding optimal model m(x) is: Where s1 is the first representative key. This represents the minimum value between two adjacent representative keys, which is obtained by traversing the representative key array in one pass.

[0019] Furthermore, the specific method for obtaining the optimal segmentation strategy of the model is as follows: retrieve the corresponding model information from the dp array and count it. Assuming that the final calculated number of segments of the dataset is p, start from dp[p-1][n-1] and retrieve the corresponding model information from the end to the beginning. The information of the previous segment of the model is determined by the value of the variable l stored in the current model. That is, the current retrieved information is the model information in the interval [l+1, n-1]. The model information of the next layer is located in dp[p-2][l], and so on, until the first layer of the dp array is traversed. Integrate the retrieved information to obtain the optimal segmentation model.

[0020] An efficient learning-based index building system for accelerating queries in read-only scenarios includes:

[0021] The segmentation upper limit determination module is used to calculate the upper limit of the number of segments by assuming that a distribution simplification model construction strategy is used on the dataset. The distribution simplification model construction strategy groups the keys in the dataset into groups of k, and takes the first key of each group as the representative key of each group to participate in the training of the dataset model.

[0022] The dynamic programming array initialization module is used to sort the data and divide it into groups, extract the representative keys of each group and store them in a representative key array. A two-dimensional dynamic programming array is defined, whose elements store the cost information of dividing the representative key into a specified number of segments. Starting from the first key in the representative key array, the approximate linear constraint PLA algorithm is used to attempt to linearly fit the representative key with a distributional simplification model construction strategy. If the fit is successful, the corresponding cost information is initialized, including the size of the model space used from the beginning of the dataset to the current item. If the fit fails, the subsequent representative keys are initialized with a distributional remapping model construction strategy, and the minimum difference between adjacent representative keys is calculated and updated. The distributional remapping model construction strategy uses an expansion coefficient greater than 1 to expand the representative keys in the dataset into a larger space.

[0023] The iterative calculation module is used to set the initial value of the number of segments. Starting from this value, it enumerates the number of segments for each type of segmentation in the outer loop. For each segment number, it enumerates the segmentation point of the last segment model in the inner loop. For each segment point, it calculates the model building strategy and related costs of the last segment model. It uses the PLA algorithm to try to add representative keys. Based on the addition results, it selects an appropriate model building strategy and calculates the space cost. It updates the minimum space cost of the current layer, finds the minimum value and saves the relevant variables and model information. It saves the calculation results to a two-dimensional dynamic programming array. It iterates in this way until the enumeration of the number of segments for all segments is completed.

[0024] The optimal strategy acquisition module is used to extract model information from the two-dimensional dynamic programming array according to the calculated number of segments in the dataset, sequentially from the end to the beginning. The information of the previous segment is determined by the variables stored in the current model. The process is repeated until the first level of the array is reached, and the acquired information is integrated to obtain the optimal segmentation strategy of the model.

[0025] The present invention also provides an electronic device, comprising: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, wherein when the programs are executed by the processors, they implement the steps of the efficient learning index construction method for accelerating queries in read-only scenarios as described above.

[0026] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the efficient learning index construction method for accelerating queries in read-only scenarios as described above.

[0027] Beneficial Effects: This invention fully considers the relationship between index query performance and index query error in read-only learning indexes. By relaxing the learning granularity to k and simplifying the distribution model construction strategy that links the query error ε to the granularity, the query error of the learning model is constrained to within two groups. Through a distribution remapping model construction strategy that expands the spatial adjustment mapping distribution, multiple segments are merged into one, simplifying the dataset distribution and thus improving query performance. Using a dynamic programming algorithm, the distribution simplification and distribution remapping model construction strategies are combined according to the specific dataset to calculate the learning index model with the minimum number of segments, which is also the optimal model. Theoretical and experimental results demonstrate that the method of this invention effectively adjusts and simplifies the data distribution, reducing the number of models and index layers after training. This reduces memory accesses and thus improves query speed. Simultaneously, this invention effectively constrains the error of read-only learning indexes, reduces index query time, and has the advantages of small index space occupation, low index height, and small prediction error. Attached Figure Description

[0028] Figure 1 This is a flowchart of the dynamic programming algorithm in this invention. Detailed Implementation

[0029] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading this invention, any modifications of the invention in various equivalent forms by those skilled in the art fall within the scope defined by the appended claims.

[0030] This invention provides an efficient learning index construction method for accelerating queries in read-only scenarios. This method primarily relies on an optimal segmentation model construction strategy that combines distributed simplification and distributed remapping models. This strategy mainly utilizes dynamic programming to determine how to construct a model that minimizes the number of segments p, given a learning index space size of M. The following sections first introduce the distributed simplification and distributed remapping model construction strategies, and then provide a detailed implementation process for achieving the optimal segmentation model construction strategy that combines these two strategies using dynamic programming.

[0031] (I) Simplified Model Construction Strategy: This strategy relaxes the granularity of the learning model from 1 to k, thus simplifying the model distribution across the dataset. The strategy groups the keys in the dataset into groups of k, taking the first key of each group as the representative key for training the model. By using the representative key of each group and combining it with the current granularity, the prediction error range of all key values ​​in the dataset is constrained. Relaxing the granularity to k reduces the time complexity of model construction. In practical applications, k is generally a value greater than 1, taking the error value mentioned later. In dynamic programming algorithms, the value of k is determined by specifying the error value.

[0032] In this strategy, the keys in the dataset are grouped into k groups, and the first key of each group is taken as the representative key for training the dataset model. Specifically, the dataset is initialized with an assumed order. Starting from the first key, every k consecutive keys in the dataset are grouped together, and the first key is used as the representative key. Since each group contains only data keys and no blank keys, these groups are dense and are defined as dense groups. When using the approximate linear constraint algorithm (i.e., PLA algorithm) for model training, only the representative keys extracted from each dense group are used for model training operations.

[0033] In this strategy, the prediction error range of all key values ​​in the dataset is constrained by using the representative key of each group and combining it with the current partitioning granularity. Specifically, constraining the prediction error range of all key values ​​in the dataset mainly includes two aspects: constraining the error range of representative keys and non-representative keys. For representative keys, the fitted linear model is calculated using the PLA algorithm, so that the linear model can constrain the error of each representative key within a specified range. For non-representative keys, ideally, their mapping value through the model should be constrained within their corresponding group. By adding or subtracting an error value ε and setting ε to k, the predicted group position and the true group position can be constrained within two groups.

[0034] This strategy reduces the time complexity of model construction by relaxing the granularity to k. Specifically, calculating a linear model using the PLA algorithm requires one linear scan of the dataset, with a time complexity of O(n). After relaxing the granularity to k, the number of keys that need to be scanned is... Therefore, the time complexity is reduced to That is, reduce by a factor of k.

[0035] (ii) Distribution remapping model construction strategy: Map the representative keys in the model to a larger space, refit the distribution of these keys and fill in some blank keys, so that these keys can be summarized by a linear model within an error constraint range, and the space occupied by the linear model is minimized. The model parameters in the distribution remapping model construction strategy are determined by key constraints and parameter tuning.

[0036] The specific implementation logic of the distributed remapping model construction strategy is as follows: For a dataset of size |D|, it is expanded to a larger space |D|·α (α>1). The keys in this dataset are then generalized using a larger linear model, possibly padded with some blank keys, so that the linear model constructed from these data keys can satisfy the error constraints and fit the data keys using the least amount of space possible. Each group contains blank keys and is therefore sparse; these groups are defined as sparse groups. Since the position of the representative key may change relative to each sparse group after expansion, the mapping value of the representative key in the model must be located within its corresponding sparse group.

[0037] In this strategy, model parameters are determined through key constraints and parameter tuning, as follows: For the distributed remapping model construction strategy, the construction objective is to select suitable parameters a and b such that m(s n )=a·(s n The minimum value of )+b is found where s n It is the largest data key, m(s) n ) represents the number of groups in the entire model. This is represented by the key s. i The definition of i∈{1,k+1,2k+1,…} gives m(s) i+k )≥m(s i )+1, to m(s) i Expanding the formula, we get a·s i+k +b≥a·s i +b+1, therefore we have remember This represents the minimum value between two adjacent representative keys, therefore And because of a·s i +b≥0, therefore b≥-a·s i ≥-a·s1. Take the endpoint values ​​of a and b respectively. And -a·s1, ultimately derive the expression for the optimal model m(x) for a data key x as follows: in It can be obtained by iterating through the array of representative keys in one pass.

[0038] (III) Optimal segmentation model construction strategy combining distribution simplification model and distribution remapping model: Each data segment can be fitted using one of the two strategies. The distribution simplification model construction strategy is preferred. The number of segments for each possible segmentation is traversed. Based on the number of segments, the model used for each segment under the current segmentation strategy and the space size used by the model are calculated. The model size calculated in the intermediate process is saved. The model used for the current dataset is calculated when the number of segments is minimized using a dynamic programming algorithm.

[0039] In this strategy, the dynamic programming algorithm is implemented as follows: First, it is assumed that the entire dataset is constructed using a distributed simplification model strategy, and the maximum number of segments required is calculated. Then, the first layer of the two-dimensional dynamic programming array is initialized, and the space size used by the model from the beginning of the dataset to the current item is calculated. Finally, the number of segments for each type of segmentation is enumerated in the outer loop, and the segmentation point of the last segment model is enumerated in the inner loop. The minimum space size of the last segment model is calculated, and the results are stored in the dynamic programming array. This process is repeated until the optimal solution of the model is obtained.

[0040] The state transition equation of the dynamic programming algorithm is as follows: Define a two-dimensional dynamic programming array dp, where dp[i][j] represents the cost information required to divide the representative key with index [0,j] in the representative key array into i+1 segments. The state transition equation for calculating dp[i][j] is dp[i][j]=min(dp[i-1][jl]+train(l+1,j)),l={p,p+1,…,j-1}, where l represents the optimal segment point of the previous segment, p represents the number of segments in the current segment, and train(l+1,j) represents the cost related to model construction in the interval [l+1,j].

[0041] The optimal segmented model derived by the dynamic programming algorithm has the following characteristics: the model has the smallest number of segments; each segment in the model is constructed using a distributional simplification or distributional remapping model construction strategy; and the segments of the model have no overlap with each other. The overall space occupied by the model may not be the smallest among all possible models, but it can meet the given space and learning granularity requirements.

[0042] Reference Figure 1 The construction process of a learned index, implemented using a dynamic programming algorithm, mainly includes the following three steps:

[0043] Step 1: Initialization phase of the optimal segmentation model strategy.

[0044] Specifically, it includes:

[0045] Step 1.1: Initialize variables such as the required index space size M and query error ε. Since the mapping value of non-representative keys through the model should be constrained within its corresponding group, ε is set to k. This constrains the predicted group position and the true group position within two groups. After sorting the data in the system, divide it according to granularity k. For the entire dataset, starting from the first data key, extract the 1st, k+1st, 2k+1st, ... data keys. These data keys are the representative keys of each group, and these keys are stored in the representative key array key.

[0046] Step 1.2, define a two-dimensional dynamic programming array dp, where dp[i][j] represents the cost information required to divide the representative key with index [0,j] in the representative key array into i+1 segments. The cost information includes the segmentation point j of the current layer, the optimal segmentation point l of the previous segment, the space size s used to divide the representative key corresponding to the interval [0,j] into i+1 segments, and whether the model construction strategy used for the last segment [l+1,j] of the representative key corresponding to the interval [0,j] is the distribution simplification model construction strategy or the distribution remapping model construction strategy.

[0047] Step 1.3: Initialize the first layer of the two-dimensional dynamic programming array dp, i.e., all values ​​of dp[0][j]. dp[0][j] represents the cost information required to divide the representative key with index [0,j] into one segment. Therefore, starting from the first key of the representative key array, iterate through the array and use the PLA algorithm to attempt to linearly fit the representative key using the distributional simplification model construction strategy. If the fit is successful, initialize the corresponding cost information, initialize the space size s to j+1, and initialize the model construction strategy to the distributional simplification model construction strategy. When the fit fails, initialize all subsequent representative keys using the distributional remapping model construction strategy. During the model construction process, calculate the minimum difference gap between adjacent representative keys and continuously update it. For the distributional remapping model construction strategy, the space size s in dp[0][j] is initialized to... Where key[j] represents the j-th representative key.

[0048] Step 1.4: Calculate the upper limit of the number of segments. Based on the representative key array from Step 1.1, iterate through it once. For each representative key in the array, attempt to add it to the model using the PLA algorithm. If successful, continue iterating to the next representative key; otherwise, clear the PLA information and increment the segment count by 1. After iterating through all representative keys, the resulting segment count is the upper limit of the number of segments. The principle behind this process is that continuously attempting to add representative keys to the model using the PLA algorithm essentially uses a distributed simplified model construction strategy to fit the representative keys. When the entire dataset uses the distributed simplified model construction strategy, the model will have the maximum number of segments.

[0049] Step 2: Construction phase of the optimal segmentation model strategy.

[0050] Specifically, it includes:

[0051] Step 2.1: Set the number of segments p to 2, starting from 2 and ending at most upper, and enumerate the number of segments.

[0052] Step 2.2: Check the segmentation status of the previous layer. Take the last element of the previous layer from the dp array and check its space usage. If the segmented model exists and the space of the model after segmentation meets the predetermined space requirements, then the optimal segmentation model for the current dataset has been calculated. Skip the remaining steps in Step 2 and proceed to Step 3.

[0053] Step 2.3: Calculate the model status of the current layer.

[0054] Specifically, it includes:

[0055] Step 2.3.1: Traverse the index j of the second dimension of the current layer, starting from p and traversing to n, to calculate the optimal segmentation model within the interval [0,j]. Specifically, traverse the variable l starting from index j-1 and traversing to p, dividing the interval [0,j] into two segments: [0,l] and [l+1,j].

[0056] Step 2.3.2: Calculate the model building strategy and associated costs for the interval [l+1,j]. If the optimal segmented model for [0,l] in the previous layer exists, then calculate the model building strategy and associated costs for the interval [l+1,j]. Starting from l+1, iterate to j, attempting to add the representative key to this interval using the PLA algorithm. If the representative key addition is successful, continue to the next representative key; otherwise, the addition fails. When a representative key addition fails, stop using the PLA algorithm. The current interval [l+1,j] is built using the distributed remapping model building strategy, its corresponding space cost is calculated, and the current interval is marked as using the remapping model building strategy. Otherwise, all representative keys can be added to the model using the PLA algorithm, and the current interval can use the distributed simplified model building strategy, its corresponding space cost is calculated, and it is marked.

[0057] Step 2.3.3: Update the minimum space cost of the current layer. After completing Step 2.3.2, calculate the space cost of the current model, which is the sum of the space cost of the interval [0, l] in the previous layer model and the space cost of the interval [l+1, j] calculated in Step 2.3.2. Based on the results of the current layer traversal in Step 2.3.1, calculate the minimum space cost of the current layer, and save the corresponding value of variable l, the minimum space size currently used, and the model information used.

[0058] Step 2.3.4: Record the calculation results of the current layer. Based on the calculation results in Step 2.3.3, if the space cost of the current layer has a minimum value, save the result of the current layer to the dp array; otherwise, set the current layer as unable to train a model that meets the requirements, and save it to the dp array as well.

[0059] Step 2.4: Continue the loop from Step 2. Increment the segment number p by 1, then re-execute Step 2.2.

[0060] Step 3: Statistical phase of the optimal segmentation model strategy.

[0061] Specifically, the corresponding model information is retrieved from the dp array and statistically analyzed. Assuming the final calculated number of segments in the dataset is p, starting from dp[p-1][n-1], the corresponding model information is retrieved from the end to the beginning. The information of the previous segment is determined by the value of the variable l stored in the current model. That is, the current retrieved information is the model information in the interval [l+1, n-1]. The model information of the next layer is located in dp[p-2][l], and so on, until the first layer of the dp array is traversed. By integrating the retrieved information, the optimal segmentation model can be obtained.

[0062] Furthermore, the performance of the efficient learning index method for accelerating queries in read-only scenarios described in this invention was verified through comparative experiments. All programs in the experiments were implemented in C++, with O3 optimization enabled by the g++ compiler. The test machine configuration was as follows: Intel(R) Xeon(R) Gold 6226R CPU@2.90GHz, 32GB RAM. The comparison algorithms were read-only learning indexes RMI, PGM, and RS, and the traditional B+ tree index B-tree. Tests were conducted using the Amazon dataset and different index space sizes. The results are shown in Table 1, where the first column represents the index space size, the first row represents the index name, and the middle content represents the query time. When the index sizes are similar, the method of this invention has the shortest query time, with a query speed improvement of 15%-120% compared to other learning indexes and 3-5 times compared to traditional B+ tree indexes.

[0063] Table 1 Comparison of query times between the present invention and several advanced index structures.

[0064] This invention RMI PGM RS BTree 0.1MB 243ns 284ns 360ns 285ns 782ns 1MB 181ns 249ns 354ns 246ns 624ns 10MB 129ns 214ns 295ns 202ns 560ns 100MB 118ns 210ns 271ns 188ns 538ns

[0065] In summary, this invention proposes an efficient learning index method for accelerating queries in read-only scenarios. This method achieves optimal query performance among read-only learning indexes while constraining query errors, and the effectiveness of these constraints is theoretically proven. By employing distribution simplification and distribution remapping model construction strategies, the learning granularity of the learning index is optimized, while simultaneously simplifying data distribution. Through an optimal segmentation model strategy combined with dynamic programming, the number of segments in the model constructed from the dataset is minimized, achieving high efficiency while meeting specified space requirements. This invention effectively constrains the error of read-only learning indexes, reduces index query time, and offers advantages such as small index space consumption, low index height, and small prediction error.

[0066] This invention also provides an efficient learning-based index building system for accelerating queries in read-only scenarios, comprising:

[0067] The segmentation upper limit determination module is used to calculate the upper limit of the number of segments by assuming that a distribution simplification model construction strategy is used on the dataset. The distribution simplification model construction strategy groups the keys in the dataset into groups of k, and takes the first key of each group as the representative key of each group to participate in the training of the dataset model.

[0068] The dynamic programming array initialization module is used to sort the data and divide it into groups, extract the representative keys of each group and store them in a representative key array. A two-dimensional dynamic programming array is defined, whose elements store the cost information of dividing the representative key into a specified number of segments. Starting from the first key in the representative key array, the PLA algorithm is used to attempt to linearly fit the representative key with a distributional simplification model construction strategy. If the fit is successful, the corresponding cost information is initialized, including the size of the model space used from the beginning of the dataset to the current item. If the fit fails, the subsequent representative keys are initialized with a distributional remapping model construction strategy, and the minimum difference between adjacent representative keys is calculated and updated. The distributional remapping model construction strategy uses an expansion coefficient greater than 1 to expand the representative keys in the dataset into a larger space.

[0069] The iterative calculation module is used to set the initial value of the number of segments. Starting from this value, it enumerates the number of segments for each type of segmentation in the outer loop. For each segment number, it enumerates the segmentation point of the last segment model in the inner loop. For each segment point, it calculates the model building strategy and related costs of the last segment model. It uses the PLA algorithm to try to add representative keys. Based on the addition results, it selects an appropriate model building strategy and calculates the space cost. It updates the minimum space cost of the current layer, finds the minimum value and saves the relevant variables and model information. It saves the calculation results to a two-dimensional dynamic programming array. It iterates in this way until the enumeration of the number of segments for all segments is completed.

[0070] The optimal strategy acquisition module is used to extract model information from the two-dimensional dynamic programming array according to the calculated number of segments in the dataset, sequentially from the end to the beginning. The information of the previous segment is determined by the variables stored in the current model. The process is repeated until the first level of the array is reached, and the acquired information is integrated to obtain the optimal segmentation strategy of the model.

[0071] The present invention also provides an electronic device, comprising: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, wherein when the programs are executed by the processors, they implement the steps of the efficient learning index construction method for accelerating queries in read-only scenarios as described above.

[0072] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the efficient learning index construction method for accelerating queries in read-only scenarios as described above.

[0073] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus (systems), electronic devices, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0074] This invention is described with reference to a flowchart of a method according to embodiments of the invention. It should be understood that each step in the flowchart and combinations thereof can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing device, generate instructions for implementing the process. Figure 1 A device for a function specified in one or more processes.

[0075] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 The function specified in one or more processes.

[0076] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 Steps of a specified function in one or more processes.

[0077] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. An efficient learning-based index construction method for accelerating queries in read-only scenarios, characterized in that, The method comprises the following steps: By assuming a distribution simplified model construction strategy is used on the data set, the upper limit of the number of segments is calculated, the distribution simplified model construction strategy groups the keys in the data set according to one group, and takes the first key of each group as the representative key of each group to participate in the training of the data set model; The data is sorted and divided by grouping granularity, the representative key of each group is extracted and stored in a representative key array, a two-dimensional dynamic programming array is defined, and the elements store the cost information of dividing the representative key into a specified number of segments. Starting from the first key in the representative key array, an approximate linear constraint PLA algorithm is used to attempt to construct a strategy for linear fitting of the representative key using a distribution simplification model. If the fitting is successful, the corresponding cost information is initialized, including the space size used by the model from the beginning of the data set to the current item. After the fitting fails, the subsequent representative keys are initialized using the distribution remapping model construction strategy, while the minimum difference value between adjacent representative keys is calculated and updated. The distribution remapping model construction strategy uses an expansion coefficient greater than to expand the representative key in the data set to a larger segment of space. setting an initial value of the segment number, starting from the initial value to the upper limit of the segment number, enumerating each segment number in an outer loop, for each segment number, enumerating the segment points of the last segment model in an inner loop, for each segment point, calculating the model construction strategy and the related cost of the last segment model, using the PLA algorithm to attempt to add the representative key, selecting the appropriate model construction strategy according to the addition result and calculating the space overhead, updating the minimum space overhead of the current layer, finding the minimum value and saving the related variables and model information, saving the calculation result into a two-dimensional dynamic programming array, and iteratively calculating in sequence until the enumeration of all segment numbers is completed; from the two-dimensional dynamic programming array, according to the calculated segment number of the data set, sequentially taking out the model information from the rear to the front, the previous segment model information being determined by the variables stored in the current model, and the information being integrated to obtain the optimal segment strategy of the model; The method can reduce the query time of the index, and the index space occupation is small.

2. The method of claim 1, wherein, The calculation method of the upper limit of the segment number specifically comprises: for each representative key in the representative key array, attempting to add it into the model using the PLA algorithm, if the addition is successful, continuing to traverse the next representative key, otherwise, emptying the information of the PLA, and incrementing the segment number of the segment by one . After traversing all the representative keys, the obtained segment number of the segment is the upper limit of the segment number of the segment .

3. The method of claim 1, wherein, Two-dimensional dynamic programming array is defined as ,in This indicates that the index in the key array is... The representative key is divided into The cost information required for each segment is calculated. The state transition equation is ,in This represents the optimal segmentation point of the previous model. Indicates the segment number of the current segment. Representing an interval The cost associated with building the model.

4. The method of claim 3, wherein, In the iterative calculation process, the appropriate model construction strategy is selected according to the addition result, and the space overhead is calculated, which specifically comprises: By variable From subscript Start traversing from , traverse to , divide interval and interval into two segments; If the upper layer If an optimal piecewise model exists, then the computation interval... The model building strategies and model-related costs, from Starting from point , traversing to The PLA algorithm is used to add a representative key to the current interval. If the addition is successful, the process continues with the next representative key; otherwise, the addition fails. When any representative key addition fails, the PLA algorithm is discontinued, and the current interval is closed. The model is constructed using the distributed remapping model construction strategy, its corresponding spatial cost is calculated, and the current interval is marked as using the remapping construction strategy. If all representative keys can be added to the model through the PLA algorithm, then the current interval is constructed using the distributed simplification model construction strategy, the corresponding spatial cost is calculated, and it is marked.

5. The method of claim 1, wherein, In the initialization of the two-dimensional dynamic programming array, for the case that the strategy fitting is successful for the distribution simplified model construction strategy, the space size is initialized as ; For the distribution remapping model construction strategy, the space size is initialized to , is the minimum value of the difference between adjacent representative keys, represents the first representative key.

6. The method of claim 1, wherein, The model parameters in the distribution remapping model construction strategy are determined by key constraint and parameter tuning. For a key , the expression of the corresponding optimal model is , where is the first representative key, represents the minimum value between two adjacent representative keys, which is obtained by traversing the representative key array once.

7. The method of claim 3, wherein, The specific acquisition method of the optimal segmentation strategy of the model is as follows: the corresponding model information is taken out from the array and counted, and it is assumed that the number of segments of the final calculated data set is , the corresponding model information is taken out from the array and counted, and it is assumed that the number of segments of the final calculated data set is , the corresponding model information is taken out from the array and counted, and it is assumed that the number of segments of the final calculated data set is , the corresponding model information is taken out from the array and counted, and it is assumed that the number of segments of the final calculated data set is , the corresponding model information is taken out from the array and counted, and it is assumed that the number of segments of the final calculated data set is , the corresponding model information is taken out from the array and counted, and it is assumed that the number of segments of the final calculated data set is , the corresponding model information is taken out from the array and counted, and it is assumed that the number of segments of the final calculated data set is , the corresponding model information is taken out from the array and counted, and it is assumed that the number of segments of the final calculated data set is 8. An efficient learning-based index construction system for accelerating queries under read-only scenarios, characterized in that, The system comprises: The segment number upper limit determination module is configured to calculate an upper limit of the segment number by assuming that a distribution simplification model construction strategy is used for the data set, the distribution simplification model construction strategy groups the keys in the data set into groups according to one group, takes the first key of each group as a representative key of each group, and participates in training of the data set model. a dynamic programming array initialization module, which is used for sorting data and dividing according to the grouping granularity, storing the representative keys of each group into a representative key array, defining a two-dimensional dynamic programming array, storing the cost information of dividing the representative keys into a specified segment number in the elements of the two-dimensional dynamic programming array, starting from the first key in the representative key array, using the approximate linear constraint PLA algorithm to attempt to perform linear fitting on the representative keys in the distributed simplified model construction strategy, initializing the corresponding cost information if the fitting is successful, including the space size used by the model from the data set to the current item; After the fitting fails, the subsequent representative key is initialized with a distribution remapping model construction strategy, while the minimum value of the adjacent representative key difference is calculated and updated; the distribution remapping model construction strategy uses an expansion coefficient greater than to expand the representative key in the data set to a larger space. an iterative calculation module, which is used for setting an initial value of the segment number, starting from the initial value to the upper limit of the number of segments, enumerating each segment number in an outer loop, for each segment number, enumerating the segmentation points of the last segment model in an inner loop, for each segment point, calculating the model building strategy and the related cost of the last segment model, using the PLA algorithm to attempt to add a representative key, selecting the appropriate model construction strategy according to the addition result and calculating the space overhead; updating the minimum space overhead of the current layer, finding the minimum value and saving the related variables, and model information, saving the calculation result into a two-dimensional dynamic programming array, and iteratively computing in sequence until the enumeration of all segment numbers is completed; an optimal strategy acquisition module, which is used for sequentially taking out the model information from the rear to the front from the two-dimensional dynamic programming array according to the calculated segment number of the data set, the previous segment model information being determined by the variables stored in the current model, and the obtained information being integrated to obtain the optimal segment strategy of the model; The system can reduce the query time of the index, and the index space occupation is small.

9. An electronic device, comprising: The system comprises: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the programs, when executed by the processor, implement the steps of the efficient learning-based index construction method for accelerating queries in read-only scenarios as claimed in any one of claims 1-7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer programs, when executed by the processor, implement the steps of the efficient learning-based index construction method for accelerating queries in read-only scenarios as claimed in any one of claims 1-7.

Images