Load-aware multi-dimensional learning indexing model and indexing method based on disk and application

Abstract

The invention discloses a disk-based load-aware multi-dimensional learning index model, which comprises the following steps of: clustering load query by using a DBSCAN (Density-Based Spatial Clustering of Applications with Noise) algorithm; using the cluster center of the load cluster as the center of the data cluster, and using the K-means algorithm to cluster the data; constructing a minimum boundary rectangle of each data cluster, and constructing an intermediate index structure for each minimum boundary rectangle; in the leaf nodes, mapping multi-dimensional data in the leaf nodes into a one-dimensional data space by using a distance space perception dimension reduction algorithm; sorting the original data according to the numerical values after dimension reduction, and storing the original data in a disk in blocks; constructing a monotonous unary linear regression model at each leaf node, and querying and inserting keys by using the model and a method based on model prediction; an insert cache page is saved in the root node and each leaf node. Load distribution and data distribution are learned and queried at the same time, and the storage overhead, the query overhead and the overhead during data insertion of the index structure are greatly reduced.

Description

Technical Field

[0001] The present invention belongs to the technical field of multi-dimensional data indexing, and particularly relates to a disk-based load-aware multi-dimensional learning index model, an index method and an application thereof. Background Art

[0002] Data indexing, as a core research field in computer science, the processing efficiency of its query and insertion operations directly determines the overall performance of a database system. With the advent of the big data era, traditional index structures are facing severe challenges when dealing with massive data: Although multi-dimensional indexes represented by KD-trees and their variants are widely used, their inherent defects such as large IO overhead and low space efficiency are becoming increasingly prominent in high-concurrency and large-scale data scenarios. There is an urgent need to explore a new type of index structure that combines efficient query, fast insertion, and low storage overhead.

[0003] In recent years, the emerging concept of "learning index" provides an innovative solution to the above challenges. This theory abstracts the index as a machine learning model and uses lightweight algorithms such as regression models to replace traditional tree structures. Compared with the search path of KD-trees that requires multiple disk accesses, the learning index can directly locate the data position through model prediction, significantly improving the query efficiency. At the same time, the model only needs to store parameters instead of the complete structure. However, early learning indexes had obvious limitations: Their design based on static memory arrays was difficult to support efficient data updates, severely restricting their application value in actual dynamic environments. Currently, KD-trees and their variants are widely used multi-dimensional data index structures. However, the space overhead and time overhead required by KD-trees are relatively large; although the current learning indexes effectively handle data queries, they do not optimize disk-based indexes by simultaneously using load distribution and data distribution.

[0004] Therefore, there is an urgent need for a disk-based load-aware multi-dimensional index structure to efficiently process data queries and data insertions. Summary of the Invention

[0005] In view of the above problems, the present invention provides a multi-dimensional learning index model based on disk load awareness, and the main problems to be solved are: using a simple machine learning model to replace the traditional multi-dimensional data index based on KD-trees; learning the data update distribution to improve the ability of existing learning indexes to handle frequent data insertions; using the powerful expression ability of machine learning models to reduce the storage overhead and query overhead of the index structure and improve the ability of the index structure to handle frequent insertions.

[0006] The technical solution adopted by the present invention is as follows: A learning index model oriented to update distribution, including the following steps:

[0007] (1) Cluster and partition the query load based on the DBSCAN algorithm;

[0008] (2) Use the center of the load clustering as the initial clustering center of the K-means algorithm to perform clustering division on the data;

[0009] (3) For each data cluster, construct its minimum bounding rectangle, and correspondingly establish an intermediate-level index structure for each minimum bounding rectangle;

[0010] (4) In the leaf nodes of the index structure, perform distance-space-aware dimensionality reduction processing to complete the non-linear mapping of multi-dimensional data to a one-dimensional space;

[0011] (5) Perform a full sort on the one-dimensional key-value pair data set after dimensionality reduction, and then use a disk-page-aligned block storage strategy to persistently store the ordered data in blocks while maintaining block-level metadata;

[0012] (6) Construct a unary linear regression model with monotonicity constraints for each leaf node, achieve efficient key-value positioning through model prediction, and support fast insertion operations based on the predicted position;

[0013] (7) Maintain an insertion cache page in the root node and all leaf nodes of the index structure, and delay writing to the disk in a batch processing manner to significantly improve the throughput in high-insertion scenarios.

[0014] Further, the specific implementation method of step (1) is: obtain the query load space characteristics including boundary points and centroid points through statistical analysis, and then use the DBSCAN density clustering algorithm to realize the automatic classification of feature points.

[0015] Further, the specific implementation method of step (2) is: for each load cluster, calculate its geometric center point as the initial clustering center, initialize the clustering center position of the K-means algorithm based on this, and use an iterative optimization method to perform clustering division on the complete data set until the convergence condition is met.

[0016] Further, the specific implementation method of step (3) is: for each data cluster, calculate its extreme boundaries (maximum and minimum values) in each feature dimension, and construct a minimum bounding rectangle (Minimum Bounding Rectangle, MBR) based on these boundary values. This space partitioning method can intelligently decompose the global query load into local queries of multiple MBR sub-spaces, significantly improving the query processing efficiency.

[0017] Furthermore, the specific implementation of step (4) is as follows: perform spatial dimensionality reduction transformation at the leaf node level. First, calculate the centroid of the node data distribution as the pivot point, and then calculate the standardized Euclidean distance between each data object in the node and the pivot point. Through this distance mapping, the original high-dimensional feature space is transformed into a one-dimensional representation space based on relative distances, while preserving the proximity relationships in the original space. This transformation meets the requirements of distance preservation, enabling proximity queries in the multi-dimensional space to be equivalently transformed into interval queries in the one-dimensional space.

[0018] Furthermore, the specific implementation of step (5) is as follows: after the dimensionality reduction process, perform a full sorting of all multi-dimensional data records based on the one-dimensional Euclidean distance values. Subsequently, adopt the disk page alignment strategy, divide the ordered data set into logical storage units according to the disk block size, and persist the data blocks to the disk through I / O operations. This design realizes sequential write optimization, converting random writes into sequential writes.

[0019] Furthermore, the specific implementation of step (6) is as follows: inside the leaf node, maintain two key metadata for each data block. The first is the minimum Euclidean distance value of all data in the block, and the second is the corresponding disk physical address. Based on these metadata, train a univariate linear regression model for each leaf node. When a query is needed, input the Euclidean distance of the target data into this model, take the integer part of the output value as the predicted position index, find the corresponding disk address according to the index and load the data block, and finally complete the data query or insertion operation.

[0020] Furthermore, the specific implementation of step (7) is as follows: to improve the data insertion efficiency, dedicated cache pages are set at the root node and all leaf nodes. The insertion process is as follows: when the data to be inserted exceeds the data space range of the root node, it is temporarily stored in the root node cache page; when the data is correctly routed to the target leaf node, it is temporarily stored in the leaf node cache page. When the root node cache page is full, it triggers a global index reconstruction; when the leaf node cache page is full, perform batch prediction insertion based on the univariate linear regression model within the node.

[0021] Further,

[0022] A more specific implementation of step (1) is as follows:

[0023] Use the DBSCAN density clustering algorithm to perform clustering analysis on the query load, take the obtained load cluster centers as the initial data clustering centers, and then apply the K-means algorithm to complete the clustering division of the data set.

[0024] By co - optimizing the load distribution and data space distribution characteristics during the index construction process, efficient query performance optimization is achieved. This algorithm first uses computational geometry methods to extract the key feature points of the query load, including the boundary point set obtained through the convex hull algorithm and the center point set determined based on kernel density estimation. Subsequently, an improved DBSCAN density clustering algorithm is applied to intelligently cluster these feature points, and finally, the representative load cluster centers are output. Then, taking these load cluster centers as the initial data centroids, an improved K - means algorithm is used to cluster and optimize the full - volume multi - dimensional data until the data clustering is completed. This two - stage load - aware data clustering strategy effectively integrates the advantages of the load query distribution characteristics and the data distribution characteristics, improving the pruning efficiency of the index structure.

[0025] A more specific implementation method of step (2) is as follows:

[0026] Construct the minimum bounding rectangle for each data cluster, and accordingly establish an intermediate - level index structure for each minimum bounding rectangle.

[0027] After the data clustering is completed, the system needs to construct an efficient index structure for each data cluster. The specific implementation process is as follows: First, intelligently decompose the load queries across multiple data clusters and allocate them to the corresponding target data clusters. Then, construct a minimum bounding rectangle for each data cluster, and this rectangular space contains both the data point set and the associated query load. Finally, adopt an optimization method based on the cost model to construct a disk - friendly index structure for each MBR from top to bottom, including a complete hierarchical organization of intermediate nodes and leaf nodes. This process ensures that the index structure can accurately reflect the data distribution characteristics and effectively support the rapid processing of query loads.

[0028] At each internal node, independently select an optimal partitioning dimension, and use the query load distribution and data distribution to evenly partition the range of the selected dimension data space, ensuring that the data space intervals within each internal node are the same. Sort the data in ascending order according to the selected dimension, and determine the maximum value max_key and minimum value min_key of the values on the selected dimension, as well as the number of child nodes fanout of the current node.

[0029] The slope calculation formula of the internal node model is:

[0030]

[0031] The intercept calculation formula of the internal node model is:

[0032]

[0033] Obtain the internal node model formula:

[0034] y = k * x + b

[0035] Given a query key, the internal node model is used to recursively find the node where the key is located until reaching the leaf node. The process of finding the child node where the key is located can be expressed as follows:

[0036] y = k * key + b.

[0037] A more specific implementation of step (3) is as follows:

[0038] At the leaf node level, the system adopts a dimensionality reduction technique based on distance space perception to map multi-dimensional data features to a one-dimensional space representation. The specific implementation includes two key steps: First, calculate the one-dimensional projection value of each data point through the distance space perception dimensionality reduction algorithm; Second, perform a full sort on the original data according to the projection value, and divide the ordered data set into logical storage blocks according to the disk page alignment principle, and finally persist it to the storage medium in a sequential write manner;

[0039] There is no obvious sorting order for multi-dimensional data. At the leaf node, in order to save the data to the disk in order, it is necessary to reduce the dimension of the multi-dimensional data, and then sort the original multi-dimensional data according to the value after dimensionality reduction, so as to impose an order on the multi-dimensional data;

[0040] Using the distance space perception dimensionality reduction algorithm to achieve the dimensionality reduction of multi-dimensional data, first calculate the center point of the data space corresponding to the leaf node, that is, the pivot point P(p1, p2,..., pn), and then calculate the Euclidean distance from each multi-dimensional data X(x1, x2,..., xn) to the pivot point,

[0041] The calculation formula of the Euclidean distance is:

[0042]

[0043] After reducing the dimension of the multi-dimensional data using the Euclidean distance, sort the original multi-dimensional data according to the Euclidean distance. On the basis of the sorted data, divide the data according to the disk page size, and save the divided data to the specified disk page in block order. At the same time, save the minimum boundary value and the corresponding disk address in each data block in the leaf node.

[0044] A more specific implementation of step (4) is as follows:

[0045] Construct a unary linear regression model with monotonicity constraints for each leaf node, and achieve efficient key-value operations through this prediction model: for a query request, the model maps the input key to the predicted position, and then performs an exact match within a local range; for an insert operation, based on the position guidance derived from the model, write the new key-value into the target storage block in an orderly manner, and dynamically adjust the model parameters to maintain the prediction accuracy;

[0046] Sort the dimensionality-reduced data in ascending order. In the saved metadata, confirm the maximum value max_key, the maximum disk address max_addr, the minimum value min_key, and the minimum disk address min_addr, and use linear interpolation to obtain a unary linear regression model;

[0047] The slope calculation formula of the leaf node model is:

[0048]

[0049] The intercept calculation formula of the leaf node model is:

[0050]

[0051] The obtained leaf node model formula is:

[0052] y = k * x + b

[0053] Given a query key key, the process of finding the disk address where the key is located using the leaf node model is as follows:

[0054] addr = k * key + b.

[0055] A more specific implementation of step (5) is: save an insertion cache page in the root node and each leaf node to effectively handle data insertion;

[0056] The data insertion operation adopts a hierarchical processing mechanism. The specific process is as follows: First, perform a spatial range check. If the data point to be inserted exceeds the data space range of the current index, it is temporarily stored in the root node cache page, and when the root node cache page reaches the capacity threshold, a global index reconstruction is triggered; if the data point to be inserted is within the spatial range, it is located to the target leaf node through the routing mechanism of the intermediate node and stored in the leaf node cache page. When the leaf node cache page is full, the following processing is performed: First, perform dimensionality reduction and sorting on the cached multi-dimensional data; secondly, predict the target disk page positions of each multi-dimensional data point based on the linear regression model of the leaf node; finally, batch-write the multi-dimensional data to the corresponding disk pages. This mechanism significantly improves the insertion throughput through spatial filtering and batch writing.

[0057] A disk-based load-aware multi-dimensional learning index method includes the above multi-dimensional learning index model.

[0058] Application of a disk-based load-aware multi-dimensional learning index model, where the load-aware multi-dimensional learning index model is used for efficient data query and data insertion.

[0059] The present invention proposes a learning index model for multi-dimensional spatial data, which jointly optimizes the index structure by using the known query load distribution characteristics and data spatial distribution characteristics. This model uses machine learning methods to replace traditional one-dimensional index structures such as KD-trees. By intelligently analyzing the load query pattern and data distribution law, it constructs a multi-layer index architecture with spatial awareness ability. Specifically, while maintaining high query performance, the model introduces an intelligent caching mechanism to handle data insertion operations, ensuring high data insertion performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 It is a schematic diagram of the overall process of data query and insertion for the multi-dimensional learning index model based on disk load awareness of the present invention;

[0061] Figure 2 It is a schematic diagram of the horizontal comparison of data insertion performance between the present invention and other existing traditional multi-dimensional index structures as well as multi-dimensional learning index structures. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0062] In order to describe the present invention more specifically, the technical solutions of the present invention will be clearly and completely described below in conjunction with the drawings and specific embodiments.

[0063] Embodiment 1:

[0064] A disk-based load-aware multi-dimensional learning index model, and its implementation process includes the following steps:

[0065] (1) Cluster and partition the query load based on the DBSCAN algorithm;

[0066] (2) Use the center of the load cluster as the initial clustering center of the K-means algorithm to cluster and partition the data;

[0067] (3) Construct the minimum bounding rectangle for each data cluster, and accordingly establish an intermediate-level index structure for each minimum bounding rectangle;

[0068] (4) In the leaf nodes of the index structure, perform dimensionality reduction processing with distance space awareness to complete the non-linear mapping from multi-dimensional data to one-dimensional space;

[0069] (5) Perform a full sort on the one-dimensional key-value pair data set after dimensionality reduction, and then use a disk page-aligned block storage strategy to persistently store the ordered data in blocks, while maintaining block-level metadata;

[0070] (6) Construct a unary linear regression model with monotonicity constraints for each leaf node, and achieve efficient key-value positioning through model prediction, and support fast insertion operations based on the predicted position;

[0071] Maintain insertion cache pages in the root node and all leaf nodes of the index structure, and delay writing to disk in a batch processing manner, significantly improving the throughput in high-insertion scenarios.

[0072] Embodiment 2:

[0073] A disk-based load-aware multi-dimensional learning index model, and its implementation process includes the following steps:

[0074] (1) Obtain the query load space characteristics including boundary points and centroid points through statistical analysis, and then use the DBSCAN density clustering algorithm to achieve automatic classification of feature points.

[0075] (2) For each load cluster, calculate its geometric center point as the initial cluster center, initialize the cluster center position of the K-means algorithm based on this, and use the iterative optimization method to perform cluster partitioning on the complete data set until the convergence condition is met.

[0076] (3) For each data cluster, calculate its extreme value boundaries (maximum and minimum values) in each feature dimension, and construct a Minimum Bounding Rectangle (MBR) based on these boundary values. This space partitioning method can intelligently decompose the global query load into local queries of multiple MBR sub-spaces, significantly improving the query processing efficiency.

[0077] (4) Implement space dimensionality reduction transformation at the leaf node level: First, calculate the centroid of the node data distribution as the pivot point, and then calculate the standardized Euclidean distance between each data object in the node and the pivot point. Through this distance mapping, the original high-dimensional feature space is transformed into a one-dimensional representation space based on relative distance, while retaining the proximity relationship in the original space. This transformation meets the distance preservation requirement, enabling proximity queries in the multi-dimensional space to be equivalently transformed into interval queries in the one-dimensional space.

[0078] (5) After completing the dimensionality reduction process, perform a full sort on all multi-dimensional data records based on the one-dimensional Euclidean distance values; then adopt the disk page alignment strategy, divide the ordered data set into logical storage units according to the disk block size, and persist the data blocks to the disk through I / O operations. This design realizes sequential write optimization, converting random writes into sequential writes.

[0079] (6) Inside the leaf node, two key metadata are maintained for each data block: the first is the minimum Euclidean distance value of all the data within the block; the second is the corresponding physical disk address. Based on this metadata, a univariate linear regression model is trained for each leaf node. When a query is needed, the Euclidean distance of the target data is input into this model, and the integer part of the output value is taken as the predicted position index. The corresponding disk address is found according to the index, and the data block is loaded, and finally the data query or insertion operation is completed.

[0080] (7) To improve the data insertion efficiency, dedicated cache pages are set at the root node and all leaf nodes. The insertion process is as follows: when the data to be inserted exceeds the data space range of the root node, it is temporarily stored in the root node cache page; when the data is correctly routed to the target leaf node, it is temporarily stored in the leaf node cache page. After the root node cache page is full, a global index reconstruction is triggered; after the leaf node cache page is full, a batch prediction insertion is performed based on the univariate linear regression model within the node.

[0081] A disk-based load-aware multi-dimensional learning index method includes the above multi-dimensional learning index model.

[0082] Application of a disk-based load-aware multi-dimensional learning index model, where the load-aware multi-dimensional learning index model is used for efficient data query and data insertion.

[0083] Figure 2 This is a schematic diagram for the horizontal comparison of the data insertion performance between the present invention and other existing traditional multi-dimensional index structures and multi-dimensional learning index structures.

[0084] Example 3:

[0085] This embodiment considers the following application scenario: user registration and ordering on a food delivery platform. When a user applies to register for a platform account, saving the user's account, password, and nickname is a data insertion, and the user hopes to complete the account registration as soon as possible. When a user orders food, information such as merchant information, food prices, and types within a certain range is a data query.

[0086] For the convenience of description, first use abstract language to describe this scenario. On the food delivery platform, the information viewed by the user when ordering food consists of merchant location, merchant rating, merchant profile, food price, food profile, food quantity, etc. This information consists of data in multiple dimensions, and all the viewed information is called a multi-dimensional data space. Therefore, this scenario can be abstracted as given a query range, finding multi-dimensional data that meets the conditions within the query range, or given multi-dimensional data to be inserted and inserting it into the corresponding disk location.

[0087] To solve the above query and insertion problems, the specific implementation process of adopting the technical solution of the present invention is as Figure 1 shown, including the following steps:

[0088] (1) The DBSCAN density clustering algorithm is used to perform clustering analysis on the query load, and the obtained load cluster centers are used as the initial data clustering centers, and then the K-means algorithm is applied to complete the clustering division of the data set.

[0089] By jointly optimizing the load distribution and data space distribution characteristics during the index construction process, efficient query performance optimization is achieved. The algorithm first uses computational geometry methods to extract the key feature points of the query load, including the boundary point set obtained through the convex hull algorithm and the center point set determined based on kernel density estimation; then an improved DBSCAN density clustering algorithm is applied to perform intelligent clustering on these feature points, and finally the representative load cluster centers are output; then these load cluster centers are used as the initial centroids of the data, and an improved K-means algorithm is used to perform clustering optimization on the full-dimensional multi-dimensional data until the data clustering is completed. This two-stage load-aware data clustering strategy effectively integrates the advantages of the load query distribution characteristics and the data distribution characteristics, and improves the pruning efficiency of the index structure.

[0090] (2) For each data cluster, its minimum bounding rectangle is constructed, and an intermediate-level index structure is established for each minimum bounding rectangle accordingly.

[0091] After the data clustering is completed, the system needs to construct an efficient index structure for each data cluster. The specific implementation process is as follows: First, the load queries across multiple data clusters are intelligently decomposed and assigned to the corresponding target data clusters; then, a minimum bounding rectangle is constructed for each data cluster, and the rectangle space contains both the data point set and the associated query load; finally, an optimization method based on the cost model is used to construct a disk-friendly index structure for each MBR from top to bottom, including a complete hierarchical organization of intermediate nodes and leaf nodes. This process ensures that the index structure can accurately reflect the data distribution characteristics and effectively support the rapid processing of query loads.

[0092] At each internal node, an optimal partitioning dimension is independently selected, and the range of the data space of the selected dimension is evenly partitioned using the query load distribution and the data distribution, ensuring that the data space intervals within each internal node are the same. The data is sorted in ascending order according to the selected dimension, and the maximum value max_key and the minimum value min_key of the values on the selected dimension and the number of child nodes fanout of the current node are determined.

[0093] The slope calculation formula of the internal node model is:

[0094]

[0095] The intercept calculation formula of the internal node model is:

[0096]

[0097] Obtain the internal node model formula:

[0098] y = k * x + b

[0099] Given a query key, use the internal node model to recursively find the node where the key is located until reaching the leaf node. The process of finding the child node where the key is located can be expressed as follows:

[0100] y = k * key + b

[0101] (3) At the leaf node level, the system adopts a dimensionality reduction technique based on distance space perception to map multi-dimensional data features to a one-dimensional space representation. The specific implementation includes two key steps: First, calculate the one-dimensional projection value of each data point through the distance space perception dimensionality reduction algorithm; Second, perform a full sort on the original data according to this projection value, and divide the ordered data set into logical storage blocks according to the disk page alignment principle, and finally persist it to the storage medium in a sequential write manner.

[0102] There is no obvious sorting order for multi-dimensional data. At the leaf node, in order to save the data to the disk in order, it is necessary to reduce the dimension of the multi-dimensional data, and then sort the original multi-dimensional data according to the value after dimensionality reduction, thus imposing an order on the multi-dimensional data.

[0103] Use the distance space perception dimensionality reduction algorithm to achieve the dimensionality reduction of multi-dimensional data. First, calculate the center point of the data space corresponding to the leaf node, that is, the pivot point P(p1, p2,..., pn), and then calculate the Euclidean distance from each multi-dimensional data X(x1, x2,..., xn) to the pivot point. The calculation formula is:

[0104]

[0105] After reducing the dimension of multi-dimensional data using the Euclidean distance, sort the original multi-dimensional data according to the Euclidean distance. On the basis of the sorted data, divide the data according to the disk page size, and save the divided data to the specified disk page in block order. At the same time, save the minimum boundary value and the corresponding disk address in each data block in the leaf node.

[0106] (4) Construct a unary linear regression model with monotonicity constraints for each leaf node, and achieve efficient key-value operations through this prediction model: For a query request, the model maps the input key to the predicted position, and then performs an exact match within a local range; For an insert operation, based on the position guidance derived from the model, write the new key-value into the target storage block in an orderly manner, and dynamically adjust the model parameters to maintain the prediction accuracy.

[0107] Sort the data after dimensionality reduction in ascending order. In the saved metadata, confirm the maximum value max_key, the maximum disk address max_addr, the minimum value min_key, and the minimum disk address min_addr, and use linear interpolation to obtain a unary linear regression model.

[0108] The slope calculation formula of the leaf node model is:

[0109]

[0110] The intercept calculation formula of the leaf node model is:

[0111]

[0112] The obtained leaf node model formula is:

[0113] y = k * x + b

[0114] Given the query key key, the process of finding the disk address where the key is located using the leaf node model is as follows:

[0115] addr = k * key + b

[0116] (5) Save the insertion cache page in the root node and each leaf node to effectively handle data insertion.

[0117] The data insertion operation adopts a hierarchical processing mechanism. The specific process is as follows: First, perform a spatial range check. If the data point to be inserted exceeds the data space range of the current index, it is temporarily stored in the root node cache page, and a global index reconstruction is triggered when the root node cache page reaches the capacity threshold; if the data point to be inserted is within the spatial range, it is located to the target leaf node through the routing mechanism of the intermediate node and stored in the leaf node cache page. When the leaf node cache page is full, the following processing is performed: First, perform dimensionality reduction processing and sorting on the cached multi-dimensional data; secondly, predict the target disk page position of each multi-dimensional data point based on the linear regression model of the leaf node; finally, batch-write the multi-dimensional data to the corresponding disk page. This mechanism significantly improves the insertion throughput through spatial filtering and batch writing.

Claims

1. A disk-based load-aware multi-dimensional learning index model, and its implementation process includes the following steps: (1) Cluster and partition the query load based on the DBSCAN algorithm; (2) Use the center of the load cluster as the initial clustering center of the K-means algorithm to cluster and partition the data; (3) Construct the minimum bounding rectangle for each data cluster, and correspondingly establish an intermediate-level index structure for each minimum bounding rectangle; (4) In the leaf nodes of the index structure, implement distance space-aware dimensionality reduction processing to complete the non-linear mapping from multi-dimensional data to one-dimensional space; (5) Perform a full sort on the one-dimensional key-value pair data set after dimensionality reduction, and then use a disk page-aligned block storage strategy to persistently store the ordered data in blocks, while maintaining block-level metadata; (6) Construct a unary linear regression model with monotonicity constraints for each leaf node, achieve efficient key-value positioning through model prediction, and support fast insertion operations based on the predicted position; (7) Maintain an insertion cache page in the root node and all leaf nodes of the index structure, and delay writing to the disk in a batch processing manner to significantly improve the throughput in high-insertion scenarios.

2. The multi-dimensional learning index model according to claim 1, characterized in that: The specific implementation method of step (1) is: obtain the spatial characteristics of the query load through statistical analysis, including boundary points and centroid points, and then use the DBSCAN density clustering algorithm to achieve automatic classification of feature points; Preferably, a more specific implementation method of step (1) is: Use the DBSCAN density clustering algorithm to perform clustering analysis on the query load, use the obtained load cluster center as the initial clustering center of the data, and then apply the K-means algorithm to complete the clustering and partitioning of the data set, Through collaborative optimization of the load distribution and data space distribution characteristics during the index construction process, efficient query performance optimization is achieved. This algorithm first uses computational geometry methods to extract the key feature points of the query load, including the boundary point set obtained through the convex hull algorithm and the center point set determined based on kernel density estimation; then apply an improved DBSCAN density clustering algorithm to intelligently cluster these feature points, and finally output representative load cluster centers; then use these load cluster centers as the initial centroids of the data, and use an improved K-means algorithm to optimize the clustering of the full-scale multi-dimensional data until the data clustering is completed. This two-stage load-aware data clustering strategy effectively combines the advantages of the load query distribution characteristics and the data distribution characteristics, and improves the pruning efficiency of the index structure.

3. The multi-dimensional learning index model according to claim 1, wherein: The specific implementation method of step (2) is: for each load cluster, calculate its geometric center point as the initial clustering center, initialize the clustering center position of the K-means algorithm based on this, and use an iterative optimization method to cluster and partition the complete data set until the convergence condition is met, then the clustering and partitioning of the data set is completed; Preferably, a more specific implementation method of step (2) is: Construct the minimum bounding rectangle for each data cluster, and correspondingly establish an intermediate-level index structure for each minimum bounding rectangle, After data clustering is completed, the system needs to build an efficient index structure for each data cluster. The specific implementation process is as follows: First, the load queries across multiple data clusters are intelligently decomposed and allocated to the corresponding target data clusters; then, a minimum bounding rectangle is built for each data cluster, and this rectangular space contains both the data point set it belongs to and the associated query load; finally, an optimization method based on the cost model is used to build a disk-friendly index structure for each MBR from top to bottom, including a complete hierarchical organization of intermediate nodes and leaf nodes. This process ensures that the index structure can not only accurately reflect the data distribution characteristics but also effectively support the rapid processing of query loads; At each internal node, an optimal partitioning dimension is independently selected, and the range of the data space of the selected dimension is evenly partitioned using the query load distribution and data distribution, ensuring that the data space intervals within each internal node are the same. The data is sorted in ascending order according to the selected dimension, and the maximum value max_key and minimum value min_key of the values on the selected dimension, as well as the number of child nodes fanout of the current node, are determined; The slope calculation formula of the internal node model is: The intercept calculation formula of the internal node model is: The internal node model formula is obtained: y = k * x + b Given the query key key, the internal node model is used to recursively find the node where the key is located until the leaf node is reached. The process of finding the child node where the key is located can be expressed as follows: y = k * key + b.

4. The multi-dimensional learning index model according to claim 1, characterized in that: The specific implementation method of step (3) is: For each data cluster, calculate its extreme value boundaries (maximum and minimum values) on each feature dimension, and build a minimum bounding rectangle (Minimum Bounding Rectangle, MBR) based on these boundary values. This space partitioning method can intelligently decompose the global query load into local queries in multiple MBR sub-spaces, significantly improving the query processing efficiency; Preferably, a more specific implementation method of step (3) is: At the leaf node level, the system adopts a dimensionality reduction technology based on distance space perception to map multi-dimensional data features to a one-dimensional space representation. The specific implementation includes two key steps: First, calculate the one-dimensional projection value of each data point through the distance space perception dimensionality reduction algorithm; second, perform a full sort on the original data according to this projection value, and divide the ordered data set into logical storage blocks according to the disk page alignment principle, and finally persist it to the storage medium in a sequential write manner; There is no obvious sorting order for multi-dimensional data. At the leaf node, in order to save the data to the disk in order, it is necessary to reduce the dimensionality of the multi-dimensional data, and then sort the original multi-dimensional data according to the value after dimensionality reduction, thus imposing an order on the multi-dimensional data; Use the distance space perception dimensionality reduction algorithm to achieve the dimensionality reduction of multi-dimensional data. First, calculate the center point of the data space corresponding to the leaf node, that is, the pivot point P(p1, p2,..., pn), and then calculate the Euclidean distance from each multi-dimensional data X(x1, x2,..., xn) to the pivot point; The calculation formula of the Euclidean distance is: After dimensionality reduction of multi-dimensional data using the Euclidean distance, sort the original multi-dimensional data according to the Euclidean distance. Based on the sorted data, divide the data according to the disk page size, and save the divided data to the specified disk pages in block order. At the same time, save the minimum boundary value and the corresponding disk address in each data block in the leaf node.

5. The multi-dimensional learning index model according to claim 1, wherein: The specific implementation of step (4) is as follows: Perform spatial dimensionality reduction transformation at the leaf node level. First, calculate the centroid of the node data distribution as the pivot point. Subsequently, calculate the standardized Euclidean distance between each data object in the node and the pivot point. Through this distance mapping, the original high-dimensional feature space is transformed into a one-dimensional representation space based on relative distances, while preserving the proximity relationships in the original space. This transformation meets the distance preservation requirement, enabling proximity queries in the multi-dimensional space to be equivalently transformed into interval queries in the one-dimensional space. Preferably, a more specific implementation of step (4) is as follows: Construct a unary linear regression model with monotonicity constraints for each leaf node, and achieve efficient key-value operations through this prediction model. For a query request, the model maps the input key to the predicted position, and then performs an exact match within a local range. For an insert operation, based on the position guidance derived from the model, the new key-value is written into the target storage block in an orderly manner, and the model parameters are dynamically adjusted to maintain prediction accuracy. Sort the data after dimensionality reduction in ascending order. In the saved metadata, confirm the maximum value max_key, the maximum disk address max_addr, the minimum value min_key, and the minimum disk address min_addr, and use linear interpolation to obtain the unary linear regression model. The formula for calculating the slope of the leaf node model is: The formula for calculating the intercept of the leaf node model is: The formula for the leaf node model is obtained as: y = k * x + b Given a query key key, the process of finding the disk address where the key is located using the leaf node model is as follows: addr = k * key + b.

6. The multi-dimensional learning index model according to claim 1, characterized in that: The specific implementation of step (5) is as follows: After completing the dimensionality reduction process, perform a full sort on all multi-dimensional data records based on the one-dimensional Euclidean distance values. Subsequently, adopt a disk page alignment strategy to divide the ordered data set into logical storage units according to the disk block size, and persist the data blocks to the disk through I / O operations. This design realizes sequential write optimization, converting random writes into sequential writes. Preferably, a more specific implementation of step (5) is as follows: Save insert cache pages in the root node and each leaf node to effectively handle data insertion. The data insertion operation adopts a hierarchical processing mechanism, and the specific process is as follows: First, perform a spatial range check. If the data point to be inserted exceeds the data space range of the current index, it is temporarily stored in the root node cache page. When the root node cache page reaches the capacity threshold, a global index reconstruction is triggered. If the data point to be inserted is within the spatial range, it is located to the target leaf node through the routing mechanism of the intermediate node and stored in the leaf node cache page. When the leaf node cache page is full, the following processing is performed: First, dimensionality reduction processing and sorting are performed on the cached multi-dimensional data; Second, the target disk page positions of each multi-dimensional data point are predicted based on the linear regression model of the leaf node; Finally, the multi-dimensional data is batch-written to the corresponding disk page. This mechanism significantly improves the insertion throughput through spatial filtering and batch writing.

7. The multi-dimensional learning index model according to claim 1, characterized in that: The specific implementation method of step (6) is as follows: Inside the leaf node, two key metadata are maintained for each data block: The first is the minimum Euclidean distance value of all data within the block; The second is the corresponding disk physical address. Based on these metadata, a unary linear regression model is trained for each leaf node. When a query is needed, the Euclidean distance of the target data is input into the model, and the integer part of the output value is taken as the predicted position index. The corresponding disk address is found according to the index and the data block is loaded, and finally the data query or insertion operation is completed.

8. The multi-dimensional learning index model according to claim 1, wherein: The specific implementation method of step (7) is as follows: To improve the data insertion efficiency, dedicated cache pages are set at the root node and all leaf nodes. The insertion process is as follows: When the data to be inserted exceeds the data space range of the root node, it is temporarily stored in the root node cache page; When the data is correctly routed to the target leaf node, it is temporarily stored in the leaf node cache page. After the root node cache page is full, a global index reconstruction is triggered; After the leaf node cache page is full, batch prediction and insertion are performed based on the unary linear regression model within the node.

9. A disk-based load-aware multi-dimensional learning index method, characterized in that, Use the multi-dimensional learning index model according to any one of claims 1-8.

10. Application of the disk-based load-aware multi-dimensional learning index model according to any one of claims 1-8, characterized in that, The load-aware multi-dimensional learning index model is used for efficient data query and data insertion.

Images