An automatic learning indexing method and system

Abstract

The application relates to an automatic learning index method and system, in particular to a method and system for automatically recommending an optimal index according to computer data characteristics, to solve the problems that, when learning indexes of a computer are established, the number of layers of the learning indexes and corresponding models of each node need to be defined manually, the space cost of index establishment is large, the establishment process is difficult, and the establishment time is long, and the application relates to a method and system for automatically recommending an optimal index according to computer data characteristics, which comprises the following steps: constructing a learning index model, the learning index model comprising a regression model and a random forest model; training the learning index model by using a computer database as a training set; inputting a key key of training data; and outputting the position of the training data in the computer database. The application belongs to the field of database indexes.

Description

Technical Field

[0001] This invention relates to a learning indexing method and system, specifically to a method and system for automatically recommending the best index based on computer data characteristics, belonging to the field of database indexing. Background Technology

[0002] Most optimizations to existing computer index structures are based on worst-case scenarios and optimize indexes for relatively evenly read / write operations. Therefore, for special cases, such as when data in a computer dataset is arranged in 1-100M order, where the data key itself can be used as an offset, using a common B-Tree index for querying the computer data is not optimal. In fact, the construction of the B-Tree increases the query time and space complexity. B-Tree or B+Tree index structures recursively partition the space using simple conditional statements, without considering the relationship between each piece of computer data and its index key, nor utilizing known data distribution characteristics. Therefore, they have suboptimal space costs and query performance. Furthermore, since there is no known computer data distribution pattern to follow in reality, the cost of deriving a distribution model for actual computer data is too high. Additionally, the recommended index structures are designed for complex computer data, requiring large-scale neural network models with many parameters and insufficient model flexibility. To address these issues, existing technologies employ reinforcement learning methods. However, when applied to computer data queries, this method requires manual definition of the current computer dataset and the set of optional index configurations for the corresponding workload, as well as the number of layers for each index and the corresponding model for each node. This makes it impossible to recommend indexes from scratch for computer data queries, increasing the time and space complexity of index creation. Consequently, the space cost of index creation is high, the creation process is difficult, and the creation time is long. Summary of the Invention

[0003] To address the problem that existing computer learning indexes require manual definition of the number of learning index layers and the corresponding model for each node, resulting in high space costs, high difficulty, and long creation time, this invention proposes an automatic learning index method and system.

[0004] The technical solution adopted in this invention is:

[0005] It includes the following steps:

[0006] S1. Obtain a computer database as a training set;

[0007] S2. Construct a learning index model, which includes a regression model and a random forest model. Train the learning index model using the training set, input the key of the training data, and output the position of the training data in the computer database to obtain the trained learning index model.

[0008] S3. Input the computer data to be queried into the trained learning index model, and output the location of the computer data in the computer database.

[0009] Furthermore, the regression models in S2 include linear regression, multinomial regression, elastic regression, gradient boosting tree, and limit tree.

[0010] Furthermore, in S2, a learning index model is constructed, which includes a regression model and a random forest model. The learning index model is trained using a training set. The input is the key of the training data, and the output is the position of the training data in the computer database, resulting in a trained learning index model. The specific process is as follows:

[0011] The key of a training data in the training set is input into the linear regression model, multinomial regression model, elastic regression model, gradient boosting tree, and limit tree of the regression model for training. Each regression model outputs the position of the training data key in the computer database, that is, the position of the training data in the computer database.

[0012] After training, the training parameters of each regression model are recorded, and the output of each regression model is scored using a scoring function. The regression models with the highest scores are selected and associated with a certain training data to obtain each training data and its corresponding regression models with the highest scores.

[0013] The random forest model is trained using each training data point and its corresponding top-scoring regression models. The training parameters of each regression model are then optimized using the random forest model to obtain the best regression model for each training data point. Finally, the best regression model is used to locate the corresponding training data in the computer database, resulting in a well-trained learning index model.

[0014] Furthermore, in S2, during the training process of the learning index model, the computer data position E where the regression model indexes errors is recorded, the relative relationship between the correct position R and the incorrect position E is compared, and a search strategy is used to correct the regression model based on the relative relationship between the correct position R and the incorrect position E.

[0015] Furthermore, the training process of the learning index model in S2 also includes: a custom index error alarm threshold. If the index error probability exceeds the threshold, the correct index position is determined by traversing the custom area where the index error is located, based on the principle of the lowest index error rate in the area.

[0016] An automatic learning indexing system includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any step of an automatic learning indexing method.

[0017] A computer-readable storage medium storing a computer program that, when executed by a processor, implements any step of an automatic learning indexing method.

[0018] Beneficial effects:

[0019] This invention views the learning index as a non-hierarchical model structure. It constructs a learning index model using regression and random forest models. When querying computer data, from the perspective of the distribution of computer data, the regression model recommends the best regression model based on the characteristics of the computer data. The random forest model optimizes and strengthens the best regression model. Using the regression model as an index, it outputs the most accurate storage location of the computer data in the computer database.

[0020] This invention enables computer data indexing and recommendation from scratch. During the index building process, there is no need to manually define the number of learning index layers or the corresponding model for each node. Multiple indexes can be automatically constructed based on the regression model of the learned index, and the optimal index recommendation can be quickly obtained during computer data queries. This reduces the time complexity, space complexity, space cost, and difficulty of index building, shortens the index building time, and improves the query performance, query time, and overall operational efficiency of computer data. While maintaining stable runtime, this invention saves a significant amount of space resources compared to models in existing research. This makes this invention more widely applicable and more flexible than existing technologies. Attached Figure Description

[0021] Figure 1 This is a flowchart of the present invention;

[0022] Figure 2 This is a structural diagram of the present invention; Detailed Implementation

[0023] Specific implementation method one: Combining Figures 1-2 This embodiment describes an automatic learning indexing method, which includes the following steps:

[0024] S1. Obtain a computer database as a training set;

[0025] S2. Construct a learning index model, which includes a regression model and a random forest model. Train the learning index model using a training set. Input the key of the training data and output the position of the training data in the computer database to obtain the trained learning index model. The specific process is as follows:

[0026] Regression models include linear regression models, multinomial regression models, elastic regression models, gradient boosting trees, limit trees, etc.

[0027] The key of a training data point in the training set is input into various regression models, including linear regression, multinomial regression, elastic regression, gradient boosting tree, and limit tree, for training. Each regression model outputs the position (key value) of the key in the computer database, thus obtaining the position of the training data in the database. The regression models learn the distribution patterns of computer data and specifically train several high-performance regression models with broad coverage of data distribution types.

[0028] After training, the training parameters of each regression model are recorded. A scoring function is used to score the output of each regression model. The top-scoring regression models are then associated with a specific training data point, resulting in each training data point and its corresponding top-scoring regression models. The scoring function is derived from common evaluation metrics for regression models, such as MSE. The training performance of each regression model is compared based on the scoring results.

[0029] The random forest model is trained using each training data point and its corresponding top-scoring regression models. The training parameters of each regression model are then optimized using the random forest model to obtain the best regression model for each training data point. Finally, the best regression model is used to locate the corresponding training data point in the computer database, thus completing the training of the learning index model and obtaining the trained learning index model.

[0030] To some extent, computer indexes can be viewed as models. A B / B+Tree index can be seen as a model that maps computer data keys to their positions in sorted computer data; a HashMap can be seen as a model that maps computer data keys to their positions in unordered computer data; and a Bitmap index can be seen as a model that determines the existence of computer data. Optimizing computer index models can improve the efficiency of computer indexes on computer datasets. Therefore, this invention constructs a learning index model, which includes a regression model and a random forest model. This learning index model first finds the corresponding optimal regression model (optimal index) based on the features of the computer data, eliminating the need to train all regression models. Based on the optimal regression model (optimal index), the precise position of the computer data in the computer database is obtained, enabling index recommendation from scratch. The training process requires no human intervention, thus automating the establishment of the computer index model. The learning index model can directly obtain the optimal index and the accurate position in the computer database based on the input computer data.

[0031] This invention utilizes regression models in machine learning to reflect patterns in computer data, enabling the automatic construction of pattern indexes for known computer datasets at low cost. The constructed learning index uses a multi-level machine learning model to replace the traditional index structure. For static data environments, it can significantly reduce the space cost of traditional indexes and improve query performance.

[0032] The regression model itself has error correction capabilities. During training, the location E of the computer data that was incorrectly indexed by the regression model is recorded, and the relative relationship between the correct location R and the incorrect location E is compared. Through experiments on large-scale datasets, it was found that |ER|≤5. This shows that when the regression model indexes incorrectly, the correct and incorrect locations of the computer data are very close. Error correction can be achieved by using a simple search strategy based on the relative relationship between the correct location R and the incorrect location E. When the regression model indexes, this invention uses relevant experimental data to define a custom error alarm threshold for the index. When the error probability of the index exceeds the threshold, the correct location position of the index is determined by traversing and searching within a custom area of ​​the predicted error location, based on the principle of the lowest index error rate within the area.

[0033] S3. Input the computer data to be queried into the trained learning index model, and automatically select a regression model for the input data based on the learning index model, and output the position of the computer data in the computer database.

[0034] Specific Implementation Method Two: Combining Figures 1-2This embodiment describes an automatic learning indexing system, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements any step of an automatic learning indexing method.

[0035] Specific implementation method three: Combining Figures 1-2 This embodiment describes a computer-readable storage medium storing a computer program that, when executed by a processor, implements any step of an automatic learning indexing method.

Claims

1. An automatic learning indexing method, characterized in that: It includes the following steps: S1. Obtain a computer database as a training set; S2. Construct a learning index model, which includes a regression model and a random forest model. Train the learning index model using a training set. Input the key of the training data and output the location of the training data in the computer database to obtain the trained learning index model. The specific process is as follows: Regression models include linear regression, multinomial regression, elastic regression, gradient boosting tree, and limit tree; The key of a training data in the training set is input into the linear regression model, multinomial regression model, elastic regression model, gradient boosting tree, and limit tree of the regression model for training. Each regression model outputs the position of the training data key in the computer database, that is, the position of the training data in the computer database. After training, the training parameters of each regression model are recorded, and the output of each regression model is scored using a scoring function. The regression models with the highest scores are selected and associated with a certain training data to obtain each training data and its corresponding regression models with the highest scores. The random forest model is trained using each training data point and its corresponding highest-scoring regression models. The training parameters of each regression model are optimized using the random forest model to obtain the best regression model for each training data point. The best regression model is then used to find the location of the corresponding training data in the computer database to obtain the trained learning index model. S3. Input the computer data to be queried into the trained learning index model, and output the location of the computer data in the computer database.

2. The automatic learning indexing method according to claim 1, characterized in that: In S2, during the training process of the learning index model, the computer data positions E where the regression model indexes errors are recorded, the relative relationship between the correct computer data positions R and the incorrect positions E is compared, and a search strategy is used to correct the regression model based on the relative relationship between the correct positions R and the incorrect positions E.

3. The automatic learning indexing method according to claim 2, characterized in that: The training process of the learning index model in S2 also includes: a custom index error alarm threshold. If the index error probability exceeds the threshold, the correct index position is determined by traversing the custom area where the index error is located, based on the principle of the lowest index error rate in the area.

4. An automatic learning indexing system, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1-3.

5. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1-3.

Images