A scalable learning index method and system

Abstract

The invention discloses an expandable learning index method and system, belonging to the field of computer data storage, including: sampling key-value pairs in a key-value storage system to obtain an ordered training data set; using the training data set to train and obtain Multiple linear regression models, each linear regression model is used to index the data in a data interval, and the data areas covered by each linear regression model do not overlap each other; each linear regression model according to <key，model>The key and model are the maximum data and model parameters in the range covered by the model respectively; use the hierarchical bucket structure to process newly inserted data; each hierarchical bucket structure corresponds to a data participating in training; each hierarchical bucket The structure includes a parent bucket, and the data in the parent bucket is ordered; each parent bucket corresponds to a sub-bucket, and the data in the sub-bucket is ordered, and the data in the sub-bucket is smaller than the data in the corresponding parent bucket. The invention can effectively improve the scalability of the learning index.< / key，model>

Description

technical field [0001] The invention belongs to the field of computer data storage, and more specifically relates to an expandable learning index method and system. Background technique [0002] In today's big data era, how to efficiently store and access data has become an important issue of concern to all walks of life. Computer systems usually use various index structures to efficiently store and access data according to different requirements, among which the tree index structure is an important structure to satisfy range requests. Many existing methods, such as CSS-Tree, CSB+-tree, FAST, etc., use memory, cache, or SIMD (Single Instruction Multiple Data, Single Instruction Multiple Data) optimization to make the tree structure provide fast data access, But these structures usually take up a lot of memory space. Once these structures overflow the limited memory due to the ever-increasing data, the efficiency of data access will be seriously reduced. [0003] The exist...

AI Technical Summary

Problems solved by technology

However, due to the high dependency between models, changing one model will change other models, which makes it difficult for us to add or remove a certain part of the model and data

[0005] (2) Expensive overhead: For the problem that the data cannot be inserted well due to the high degree of dependence between the models, some learning index methods use a buffer to save the newly inserted data, so that the newly inserted data will not be destroyed The original data distribution, however, will result in access to data by accessing two structures (ie, the original structure and the buffer), which greatly reduces the efficiency of data access

In order to separate and store the data in other places, the existing learning index methods either separate the data covered by multiple models, or build a data conversion table to separate and migrate the data, but both of these methods require retraining. When rebuilding, and introduce a lot of space and time overhead

[0006] Overall, existing learning indexing methods suffer from poor scalability

Images