A data storage method and system based on a scalable multidimensional learning index structure

Abstract

This invention discloses a data storage method and system based on a scalable multidimensional learning index structure, belonging to the field of computer data storage. This system fully utilizes storage characteristics to organize data into memory and disk, significantly reducing disk I / O with minimal memory usage, accelerating data search speed, and supporting data expansion. This invention can be applied to multidimensional data indexing, typically represented by geospatial data, meeting the need for rapid multidimensional key searching in the high-dimensional key space of such data.

Description

Technical Field

[0001] This invention belongs to the field of computer data storage, specifically relating to a data storage method and system based on a scalable multidimensional learning index structure. Background Technology

[0002] In today's big data era, the rapidly increasing volume of data places higher demands on the performance and capacity of storage systems, and the index structure is a crucial component. When the data volume is too large, traditional databases generally use one-dimensional index structures such as B+ trees. However, when too many indexes are created, not only is the cost of maintaining each index individually too high, but the database also needs to evaluate each index to determine the optimal strategy during queries. Moreover, multi-condition range queries cannot accurately determine the range of all query conditions, resulting in performance degradation. On the other hand, if traditional multi-dimensional indexes such as Z-Orders are used, the difference between the index order and the physical storage order in disk storage increases disk I / O, and its performance degrades sharply as the data volume increases.

[0003] In recent years, learning index structures have developed rapidly. They learn the distribution patterns of data through machine learning algorithms, achieving excellent performance with minimal storage overhead. However, existing learning indexes are not widely used because they are primarily memory-based, limiting their application with large datasets and exhibiting poor scalability. This is because data modifications alter the data distribution, requiring model reconstruction; for example, PGM models often sacrifice range queries and query efficiency to support data modifications. Furthermore, they operate within a one-dimensional key space, failing to meet the need for rapid multi-dimensional key searches using a single index structure. Summary of the Invention

[0004] To address the shortcomings and improvement needs of existing technologies, this invention provides a data storage method and system based on a scalable multidimensional learning index structure.

[0005] To achieve the objectives of this invention, the specific technical solution adopted is as follows:

[0006] In a first aspect, the present invention provides a data storage system based on a scalable multidimensional learning index structure for organizing multidimensional data into memory and disk. The data storage system includes a grid partitioning layer, a data fitting layer, a storage layer, and a computation layer.

[0007] The grid partitioning layer consists of two models. The first layer is a grid tree model, which is used to divide the high-dimensional key space of multidimensional data into multiple sub-regions with evenly distributed query load. The second layer is a multi-way partitioning tree model, which is used to further divide each sub-region obtained by the grid tree model into sub-grids with evenly distributed data space.

[0008] The data fitting layer is used to reduce the dimensionality of multidimensional data through a mapping function and construct a corresponding data fitting model according to the currently selected mode of the data storage system to achieve data location; the optional modes of the data storage system are read-only mode and read-write mode, and different data fitting models are corresponding to the two modes respectively.

[0009] The storage layer is used to organize, encode, and store the high-dimensional key space and the model data established by the grid partitioning layer and the data fitting layer. When storing, the high-dimensional key space data is stored on the disk in multiple data pages, while part of the index structure is stored in memory and the rest is stored on the disk.

[0010] The computing layer is used to provide data CRUD services to the outside world through the reserved interface of the storage layer.

[0011] As a preferred embodiment of the first aspect, the grid tree model in the grid partitioning layer is constructed as follows: First, historical query records are sampled, and the sampled query records are projected onto each dimension to obtain the distribution of queries in each dimension; then, a tree structure is constructed based on the probability density of the query distribution to divide each dimension. Each layer of the tree selects one dimension for division, and each time the key space represented by a node is divided into multiple regions. After dividing all dimensions, the query distribution in each sub-region is made as uniform as possible.

[0012] The multi-way partitioning tree model in the grid partitioning layer is a tree structure. Each node in this tree structure represents a subspace in the global multidimensional key space, and the root node represents the key space corresponding to the sub-region. During the construction of the multi-way partitioning tree model, one dimension of the subspace is partitioned each time. For the selected partitioning dimension, the cumulative distribution function of the multidimensional data in the subspace on that dimension is first obtained. Then, the subspace is uniformly partitioned according to the cumulative distribution function. The partitioned subspaces constitute the next layer of nodes in the multi-way partitioning tree model, and the lower layer nodes are iteratively partitioned.

[0013] As a preferred embodiment of the first aspect, the specific method for dimensionality reduction of multidimensional data through a mapping function in the data fitting layer is as follows: First, the subgrids obtained by the multi-way partitioning tree model are numbered. Then, a mapping function is constructed to map the multidimensional data in each subgrid, and the mapping value maintains the order of the multidimensional data. That is, for different subgrids, the smaller the number of the subgrid, the smaller the data mapping value. For the same subgrid, the smaller the data value of all dimensions within the subgrid, the smaller the data mapping value. Finally, the mapping value of all keys in one-dimensional space is calculated through the mapping function.

[0014] As a preferred embodiment of the first aspect mentioned above, the currently selected mode of the data storage system is selected by the user by sending a switching command according to their application scenario.

[0015] As a preferred embodiment of the first aspect, when the data storage system is currently selected as read-only mode, the data fitting layer establishes a piecewise linear approximation model for data fitting, and uses this data fitting model to segment all keys to accelerate the search. The piecewise linear approximation model first forms an ordered array of the mapping values ​​of all keys in one-dimensional space, and then performs piecewise fitting on the overall data distribution of the array, dividing the array into many segments. Each segment is fitted with a linear function while ensuring a certain error range. Then, for each segment, the mapping value at the beginning of the segment is extracted to form a new array. The piecewise fitting is performed recursively until a certain array can be represented by only one segment, and finally a tree structure is formed.

[0016] When the data storage system is currently selected as read / write mode, the data fitting layer establishes a data fitting model as a distribution fitting model and uses this distribution fitting model to organize the data distribution of each sub-grid. The distribution fitting model fits the mapping values ​​of each sub-grid obtained by the multi-way partitioning tree model. First, a hash function is constructed to hash these mapping values ​​into a certain interval while ensuring a certain degree of hash collision. For the hash collision mapping values, they are organized together to recursively construct a hash function, and finally a tree structure is formed.

[0017] As a preferred embodiment of the first aspect above, the storage layer stores the multidimensional data and corresponding models in different forms, as follows:

[0018] First, both the grid tree model and the multi-way partitioning tree model only store the data splitting points and the pointers corresponding to the splitting points. They are both tree structures and are stored in memory.

[0019] Secondly, in read-only mode, for all non-leaf nodes of the tree structure constructed by the piecewise linear approximation model, the linear fitting function and corresponding fitting data of each non-leaf node are also placed in memory. For each leaf node of the piecewise linear approximation model, its linear fitting function and corresponding fitting data are placed on the disk. A page size is set, and these data are stored sequentially into each page of the disk, forming a linked list structure between the pages. In read-write mode, the entire tree structure constructed by the distribution fitting model for each subgrid is first numbered using DFS order, then the adjacency list is used to represent the tree structure, and finally the tree structure is encoded and stored into multiple consecutive pages on the disk.

[0020] As a preferred embodiment of the first aspect above, the data CRUD services provided in the computing layer include single-point query, range query, data insertion, data deletion, and data modification, and data insertion, data deletion, and data modification are only supported in read-write mode and not in read-only mode.

[0021] The single-point query is used to retrieve the value corresponding to a key. The process is as follows: for a key, first find its subgrid according to the grid tree model and the multi-way partitioning tree model, and then search in the tree structure under different modes according to the mapping value of the key through the data fitting layer.

[0022] The range query is used to query all values ​​in a high-dimensional key space. The method is as follows: For the query space, find out which sub-grids it intersects with in the multi-way partitioning tree model to obtain each intersecting sub-region. Then, perform a single-point query on the two endpoints of each sub-region. These two endpoints are the two points where the values ​​of each dimension are the minimum and the values ​​of each dimension are the maximum. After finding the two endpoints, scan between the endpoints and save all data in the query space.

[0023] The method for inserting data is as follows: first, perform a single-point query to find the position to be inserted on the leaf node of the tree structure of the distribution fitting model. If the position to be inserted is empty, insert directly; otherwise, construct the next level node and then insert.

[0024] The method for deleting data is as follows: first, perform a single-point query to find the position to be deleted on the leaf node of the tree structure of the distribution fitting model; if the data to be deleted is at the position to be deleted, then delete it directly.

[0025] The method for modifying the data is as follows: First, perform a single-point query to find the position to be modified on the leaf node of the tree structure of the distribution fitting model. If the value to be modified is to be modified, the position to be modified is modified directly. If the key to be modified is to be modified, the operation is split into two parts: deleting the original data and inserting the new data at the position to be modified.

[0026] Inserting, deleting, or modifying data will all change the tree structure of the distribution fitting model. When the amount of data change meets the preset conditions, the tree structure needs to be reconstructed.

[0027] As a preferred embodiment of the first aspect above, the high-dimensional key space is composed of some attributes extracted from a multi-attribute database table, and the remaining attributes in the database table are used as the values ​​of the high-dimensional key.

[0028] As a preferred embodiment of the first aspect mentioned above, the multidimensional data is preferably geospatial data.

[0029] Secondly, this invention provides a data storage method based on a scalable multidimensional learning index structure for organizing multidimensional data into memory and disk, comprising:

[0030] S1. Construct a two-layer model to perform grid partitioning on the high-dimensional key space of multidimensional data. The first layer is a grid tree model, which is used to divide the high-dimensional key space of multidimensional data into multiple sub-regions with evenly distributed query load. The second layer is a multi-way partitioning tree model, which is used to further divide each sub-region obtained by the grid tree model into sub-grids with evenly distributed data space.

[0031] S2. Dimensionality reduction of multidimensional data is performed through a mapping function, and a corresponding data fitting model is constructed according to the currently selected mode of the data storage system to achieve data location; the optional modes of the data storage system are read-only mode and read-write mode, and different data fitting models are corresponding to the two modes respectively.

[0032] S3. Organize, encode, and store the high-dimensional key space and the model data established by the grid partitioning layer and the data fitting layer. When storing, the high-dimensional key space data is stored on the disk in multiple data pages, while part of the index structure is stored in memory and the rest is stored on the disk.

[0033] S4. Provide data creation, deletion, modification, and query services to the outside world through the reserved interface of the storage layer.

[0034] It should be noted that the technical solution adopted in the second aspect has the same inventive concept as the technical solution in the first aspect. Therefore, the preferred solution of the first aspect can also be applied to the technical solution of the second aspect.

[0035] Compared with the prior art, the present invention has the following advantages:

[0036] This invention provides a data storage method and system based on a scalable multidimensional learning index structure. This system fully utilizes storage characteristics to organize data into memory and disk, significantly reducing disk I / O with minimal memory usage, accelerating the search speed for data stored on disk, and supporting data expansion. This invention can be applied to multidimensional data indexing, typically represented by geospatial data, meeting the need for rapid multidimensional key searching in the high-dimensional key space of such data. Attached Figure Description

[0037] Figure 1 A system architecture diagram provided for embodiments of the present invention;

[0038] Figure 2 This is a graph generated from a grid tree cover set as provided in an embodiment of the present invention;

[0039] Figure 3This is a schematic diagram of a multi-way partitioning tree provided in an embodiment of the present invention;

[0040] Figure 4 This is an iterative generation diagram of the convex hull of the piecewise linear approximation model provided in an embodiment of the present invention.

[0041] Figure 5 This is a schematic diagram of the recursive construction of a piecewise linear approximation model provided in an embodiment of the present invention;

[0042] Figure 6 This is a schematic diagram illustrating the construction of the distribution fitting model provided in an embodiment of the present invention;

[0043] Figure 7 This is a flowchart of a single-point query process for the computational layer provided in an embodiment of the present invention;

[0044] Figure 8 This is a schematic diagram of a computational layer range query provided in an embodiment of the present invention. Detailed Implementation

[0045] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. Technical features in the various embodiments of the present invention can be combined accordingly without mutual conflict.

[0046] First, this invention considers a general application scenario for multidimensional learning indexes. For a database table with many attributes and numerous complex filtering conditions for different attributes, typically, indexes need to be created for frequently used attributes in the filtering conditions to speed up queries. However, during automatic database query optimization, not only are all indexes evaluated, but only one index can be selected for querying, making it impossible to fully filter data within the indexes. Furthermore, data insertion also requires maintenance of each index, increasing overhead. The multidimensional learning index structure of this invention not only ensures extremely low query and insertion performance overhead but also significantly reduces storage space usage. For attributes that appear frequently in the filtering conditions, all corresponding attributes in the database table are extracted to form a high-dimensional key space. The remaining attributes are the values ​​of these high-dimensional keys, also known as the key-based load. An index structure is then created for this high-dimensional key space.

[0047] Of course, the data in this invention is not limited to indexing multiple attributes in a database table; it can also be indexed on original high-dimensional data, such as geospatial data.

[0048] The present invention will now be described in detail through embodiments of the specific implementation of the data storage system based on a scalable multidimensional learning index structure. In a preferred embodiment of the present invention, a data storage system based on a scalable multidimensional learning index structure is provided for organizing multidimensional data into memory and disk. The architecture diagram of the data storage system is shown below. Figure 1 As shown, the high-dimensional key space is divided into sub-grids with uniform query and spatial distributions through a grid partitioning layer. The data fitting layer can then be set to either read-only or read-write mode, fitting or organizing the data distribution in each mode. Finally, the storage layer organizes, encodes, and stores the high-dimensional key space and the established model data, exposing interfaces for users to manipulate the data using the model. The specific implementation details of the grid partitioning layer, data fitting layer, storage layer, and computation layer included in this data storage system are described in detail below.

[0049] The aforementioned grid partitioning layer consists of two models. The first layer is a grid tree model, which is used to divide the high-dimensional key space of multidimensional data into multiple sub-regions with evenly distributed query loads. The second layer is a multi-way partitioning tree model, which is used to further divide each sub-region obtained by the grid tree model into sub-grids with evenly distributed data space.

[0050] In an embodiment of the present invention, the grid tree model in the grid partitioning layer divides the global key space into multiple sub-regions with uniform query load distribution. The specific construction method is as follows: First, historical query records are sampled, and the sampled query records are projected onto each dimension to obtain the distribution of queries in each dimension; then, a tree structure is constructed according to the probability density of the query distribution to partition each dimension. Each layer of the tree selects one dimension for partitioning, and each time the key space represented by a node is partitioned into multiple regions. After partitioning all dimensions, the query distribution in each sub-region is made as uniform as possible.

[0051] In an embodiment of the present invention, the multi-way partitioning tree model in the grid partitioning layer divides each sub-region into sub-grids with uniform data spatial distribution and disk storage friendliness. The specific construction method of the multi-way partitioning tree model is as follows: the multi-way partitioning tree model is a tree structure, in which each node represents a subspace in the global multidimensional key space, and the root node represents the key space corresponding to the sub-region; during the construction process of the multi-way partitioning tree model, one dimension of the subspace is split at each time. For the selected splitting dimension, the cumulative distribution function of the multidimensional data in the subspace on that dimension is first obtained, and then uniformly split according to the cumulative distribution function. The split subspaces constitute the next layer node of the multi-way partitioning tree model, and the lower layer node is iteratively split.

[0052] The aforementioned data fitting layer is used to reduce the dimensionality of multidimensional data through a mapping function, and to construct a corresponding data fitting model based on the currently selected mode of the data storage system to achieve data location; the optional modes of the data storage system are read-only mode and read-write mode, and different data fitting models are corresponding to the two modes respectively;

[0053] In an embodiment of the present invention, the specific method for dimensionality reduction of multidimensional data using a mapping function in the data fitting layer is as follows: First, the subgrids obtained by the multi-way partitioning tree model are numbered. Then, a mapping function is constructed to map the multidimensional data in each subgrid, and the mapping value maintains the order of the multidimensional data. That is, for different subgrids, the smaller the number of the subgrid, the smaller the data mapping value. For the same subgrid, the smaller the data value of all dimensions within the subgrid, the smaller the data mapping value. Finally, the mapping value of all keys in one-dimensional space is calculated using the mapping function.

[0054] It should be noted that the data storage system is divided into read-only mode and read-write mode, maximizing system performance in different usage scenarios. Users can switch between the two modes according to their application needs. The data fitting layer constructs different models in each mode, and the currently selected mode is chosen by the user by sending a switching command based on their application scenario.

[0055] In an embodiment of the present invention, when the data storage system is currently selected as read-only mode, the data fitting layer establishes a piecewise linear approximation model for data fitting, and uses this data fitting model to segment all keys to accelerate the search. The piecewise linear approximation model first forms an ordered array of the mapping values ​​of all keys in one-dimensional space, and then performs piecewise fitting on the overall data distribution of the array, dividing the array into many segments. Each segment is fitted with a linear function and a certain error range is guaranteed. Then, for each segment, the mapping value at the beginning of the segment is extracted to form a new array. The piecewise fitting is performed recursively until a certain array can be represented by only one segment, and finally a tree structure is formed.

[0056] When the data storage system is currently selected as read / write mode, the data fitting layer establishes a data fitting model as a distribution fitting model and uses this distribution fitting model to organize the data distribution of each sub-grid. The distribution fitting model fits the mapping values ​​of each sub-grid obtained by the multi-way partitioning tree model. First, a hash function is constructed to hash these mapping values ​​into a certain interval while ensuring a certain degree of hash collision. For the hash collision mapping values, they are organized together to recursively construct a hash function, and finally a tree structure is formed.

[0057] The aforementioned storage layer is used to organize, encode, and store the high-dimensional key space and the model data established by the grid partitioning layer and the data fitting layer. During storage, the high-dimensional key space data is stored on the disk in multiple data pages, while part of the index structure is stored in memory and the rest is stored on the disk.

[0058] In embodiments of the present invention, the multidimensional data and corresponding models are stored in different forms in the storage layer, as follows:

[0059] First, both the grid tree model and the multi-way partitioning tree model only store the data splitting points and the pointers corresponding to the splitting points. They are both tree structures and are stored in memory.

[0060] Secondly, in read-only mode, for all non-leaf nodes of the tree structure constructed by the piecewise linear approximation model, the linear fitting function and corresponding fitting data of each non-leaf node are also placed in memory. For each leaf node of the piecewise linear approximation model, its linear fitting function and corresponding fitting data are placed on the disk. A page size is set, and these data are stored sequentially into each page of the disk, forming a linked list structure between the pages. In read-write mode, the entire tree structure constructed by the distribution fitting model for each subgrid is first numbered using DFS order, then the adjacency list is used to represent the tree structure, and finally the tree structure is encoded and stored into multiple consecutive pages on the disk.

[0061] The aforementioned computing layer is used to provide data CRUD services to the outside world through the reserved interface of the storage layer.

[0062] In embodiments of the present invention, the data CRUD services provided in the aforementioned computing layer include single-point query, range query, data insertion, data deletion, and data modification. Furthermore, data insertion, data deletion, and data modification are only supported in read-write mode and not in read-only mode. The specific data CRUD services are as follows:

[0063] 1) Single-point query is used to retrieve the value corresponding to a key. The process is as follows: For a key, first find the subgrid where it is located according to the grid tree model and the multi-way partitioning tree model. Then, through the data fitting layer, search on the tree structure in different modes according to the mapping value of the key.

[0064] 2) Range query is used to query all values ​​in a high-dimensional key space. The process is as follows: For the query space, find out which sub-grids it intersects with in the multi-way partitioning tree model to obtain each intersecting sub-region. Then, perform a single-point query on the two endpoints of each sub-region. These two endpoints are the two points where the values ​​of each dimension are the minimum and the maximum, respectively. After finding the two endpoints, scan between the endpoints and save all data in the query space.

[0065] 3) The method for inserting data is as follows: First, perform a single-point query to find the insertion position on the leaf node of the tree structure of the distribution fitting model. If the insertion position is empty, insert directly; otherwise, construct the next level node and then insert.

[0066] 4) The procedure for deleting data is as follows: First, perform a single-point query to find the position to be deleted on the leaf node of the tree structure of the distribution fitting model. If the data to be deleted is located at the position to be deleted, then delete it directly.

[0067] 5) The procedure for modifying data is as follows: First, perform a single-point query to find the position to be modified on the leaf node of the tree structure of the distribution fitting model. If the value to be modified is to be modified, then directly modify the position to be modified. If the key to be modified is to be modified, then split it into two operations: deleting the original data and inserting the new data at the position to be modified.

[0068] 6) Inserting, deleting, or modifying data will all change the tree structure of the distribution fitting model. When the amount of data change meets the preset conditions, the tree structure needs to be reconstructed.

[0069] The present invention will now be illustrated through a specific embodiment. Figure 1 The diagram illustrates the specific construction process and working principle of a data storage system based on a scalable multidimensional learning index structure.

[0070] Example

[0071] In this embodiment, the specific construction process and usage method of the data storage system based on the scalable multidimensional learning index structure are as follows:

[0072] (1) Construct a grid tree model and use it to divide the d-dimensional bond space into sub-regions.

[0073] First, historical query records in the database are sampled. Then, the sample query range is projected onto d dimensions, and each dimension is independently segmented to obtain uniformly distributed sub-regions of the query. The specific segmentation schemes for each dimension are as follows:

[0074] Suppose we are currently partitioning the x-th dimension, and define the projection set of the query set Q onto the x-th dimension as Q. x Iterate through the query projection set Q along this dimension. x Find its left and right endpoints to obtain the maximum query range [X] l ,X r Q x Within the range [X l ,X r Query skew on Skew(Q) x ,X l ,Xr The area enclosed by the probability density function and the uniform distribution function of the query set Q within the x-dimensional query range can be defined as follows: The uniform distribution function between each cell can be changed by dividing the query range. Next, it is necessary to analyze the interval [X...] l ,X r ) partitioning to minimize Q x Query skew Skew(Q) x ,X l ,X r ).

[0075] First, divide the interval [X] l ,X r The set Q is evenly divided into 1024 continuous sub-intervals, each called a bin. Then, the set Q is traversed. x For any given query range, if it intersects with m bins, then its contribution to each of those m bins is 1 / m. That is, assuming it intersects with bin... i If they intersect, then bin i Contribution b i We need to add 1 / m. This will ultimately give us the discrete value of b. i (1<=i<=1024) is used to estimate the probability density function.

[0076] By treating each bin as the smallest segmentation unit and combining these smallest segmentation units, we can obtain the optimal segmentation scheme for the x-th dimension:

[0077] [X l ,X r ) = [bin1,bin v1 )∪[bin v1 ,bin v2 )...∪[bin vn ,bin 1024 )

[0078] Where v1, v2, ..., v n The corresponding actual coordinates are the split points of the x-th dimension. After the split, the sum of the query skew of each interval is minimized. The goal is to find such a set of split points that minimizes the sum of the query skew.

[0079] For any interval [bin] vx ,bin vy The uniform distribution function on it is defined as:

[0080]

[0081] Where b k That is, all query pairs bin kThe total contribution is in [bin] vx ,bin vy The query skew on the interval is:

[0082]

[0083] Therefore, the query skew on the x-th dimension is:

[0084]

[0085] Where v n+1 The endpoint 1024 represents the point where the x-th dimension is divided.

[0086] If a set of intervals are disjoint and their union is [bin1, bin2], then... 1024 This is called the covering set. Since a query skew of 0 for a single bin is meaningless, two adjacent bins are merged into a minimum subinterval as leaf nodes, resulting in 512 leaf nodes. These leaf nodes are then recursively merged upwards pairwise to obtain a tree. The root node of the tree represents [bin1, bin2, ..., bin3]. 1024 First, we calculate the query skewness dp[u] when the interval represented by each node u is treated as a whole. Then, we perform dynamic programming on the tree, assuming that the child nodes of node u are son1, son2, ..., son n ,but:

[0087] dp[u]=min(dp[u],(dp[son1]+dp[son2]+...+dp[son n ])×(1+n / 10))

[0088] Where n is the number of child nodes of node u. The reason why the sum of child node query skew needs to be multiplied by a coefficient (1+n / 10) during dynamic programming is to balance the query skew and the number of sub-intervals. When the query skew is not much different, the number of intervals should be minimized.

[0089] Then, a depth-first search is performed again. Each time a point u is visited, if its dp[u] is equal to the query skewness of its overall interval, it is directly backtracked, and the interval represented by u is added to the result set. The various split points in this result set are the dividing points for that dimension. For example... Figure 2 As shown, the gray-filled nodes represent sub-intervals of a result set. This result set is a cover set, and the query skew is optimal when the sub-intervals corresponding to a single gray node are not split.

[0090] (2) Construct a multi-path partitioning tree model, and perform uniform spatial partitioning for each sub-region.

[0091] A multi-path partitioning tree model is constructed by combining the partitioning strategies of kd-trees and B+ trees.

[0092] First, set a threshold α, and recursively partition the d-dimensional data. Each time, select a dimension x with the smallest variance, and uniformly partition the data according to the cumulative distribution function of the projection onto that dimension. The partitioning is performed in multiple parts, where n is the number of keys. This yields the partitioning scheme for each level. The multi-way partitioning tree model stores each partition point, and each partition point points to the next level of the model. This continues until the number of keys in a partitioned region is less than a threshold α. At this point, the node becomes a leaf node, and the key space it represents is called a sub-grid. The entire partitioning process is based on an improved kd-tree, with an improvement strategy similar to the transition from an AVL balanced tree to a B+ tree. Each partition divides the tree into multiple parts, and non-leaf nodes only store the partition point, not the actual key-value pairs, thus reducing the number of tree levels. For example... Figure 3 As shown, this represents a two-dimensional sub-region partitioning scheme. It first divides the vertical coordinate, and then further divides each sub-region on the horizontal coordinate, so that the resulting sub-grids are evenly distributed in space.

[0093] (3) Construct a mapping function F to reduce the dimensionality of multidimensional data.

[0094] To further accelerate the data search, a mapping function F is constructed to map the d-dimensional data in the subgrid obtained in step (2). The mapping is to a 1D data value f, and the mapping function needs to satisfy the condition that x1 is within the same grid. <y1,x2<y2,...,x d <y d hour,

[0095] If d-dimensional data Located in subgrid L with index i i Within, that is:

[0096]

[0097] in Submesh L i The data range in the i-th dimension, and the data range in all dimensions. The Cartesian product of L is represented by i Corresponding to the d-dimensional space range.

[0098] Defineable:

[0099]

[0100] Obviously Satisfying the condition within the same grid when x1 <y1,x2<y2,...,x d <y d hour, this condition

[0101] and Therefore, when i < j where

[0102] All the mapped values of the keys can be calculated through the mapping function F

[0103] (4) The system is divided into a read-only mode and a read-write mode

[0104] To optimize the performance of the database in different application scenarios, different models will be established for the read-only mode and the read-write mode at the bottom layer of the database index model. Users can choose different modes according to their own needs. For example, the read-only mode can be used for data analysis, while the read-write mode can be used in the OLTP scenario

[0105] (5) Establish a piecewise linear approximation model in the read-only mode

[0106] In the read-only mode, the number of keys in all sub-grids has been fixed. The mapped values of all keys are calculated through the mapping function F, and they are regarded as a whole to establish a piecewise linear approximation model. Suppose the ordered array composed of all mapped values is f, and the mapped value of a certain key is f[i]. The piecewise linear approximation model can predict its position i in the array through f[i]

[0107] First, regard all (f[i], i) as points on the plane. Now, a piecewise linear function needs to be established to fit these points, and the error threshold is set to ε. Let u[i] = i - ε, v[i] = i + ε, then the slope k and intercept b of each line segment need to be found to satisfy

[0108] u[i] <= k * f[i] + b <= v[i]

[0109] where f[i], u[i], and v[i] are all known. Transforming it gives

[0110]

[0111] This actually becomes a linear programming problem. General linear programming problems can be solved using the simplex algorithm or network flow graph construction, but these methods have high complexity. However, the special point here is that because the array f is ordered and increasing and greater than 0, (-f[i]) < 0 always holds true, and its slope becomes increasingly steep. Plotting these two inequalities on a plane results in two parallel half-planes. Substituting multiple f[i], u[i], v[i] into the inequalities and plotting them on the plane, if they intersect, they form a convex hull, and any edge in the convex hull has an edge parallel to it. Furthermore, since the slope becomes increasingly steep with each insertion of a new f[i], we know that the rightmost point of the convex hull is the lowest point, and the leftmost point is the highest point. Let the rightmost endpoint of the convex hull be R, and the leftmost endpoint be L.

[0112] Starting from (f[0],0), add two lines sequentially, each time adding the corresponding two lines. Now, we need to find the intersection points of each added line with the original convex hull to obtain the new convex hull. Let the newly added right line be Xr and the left line be Xl.

[0113] Connect the vertices L and R of the convex hull with a straight line. First, scan the vertices of the convex hull in the right half-plane of line LR, starting from point R. If a vertex of the original convex hull is outside the half-plane corresponding to Xr, delete that vertex. Otherwise, if this is the first scan (i.e., no points have been deleted before), it means the entire convex hull is within the half-plane corresponding to Xr, and no operation is performed. If this is not the first scan, the intersection of the line connecting the previously deleted point and the point scanned this time with Xr can be calculated, and this intersection point can be added to the convex hull. Similarly, the intersection points of Xr with the convex hull in the left half-plane of LR, and the intersection points of Xl with the convex hulls in both the left and right half-planes of line LR, can be calculated to obtain the new convex hull generated in this iteration. Figure 4 As shown in the figure, the solid line represents the original convex hull, and the dashed line represents the new convex hull generated by adding two half-planes. The region enclosed by the convex hull is the possible value of the slope and intercept of the linear function segment. If there is no intersection point in the current iteration, it means that k and b do not simultaneously satisfy the range of the original convex hull and the half-plane containing Xr and Xl in the current iteration. In this case, the coordinates of the R point of the original convex hull are taken as k and b, representing one segment of the piecewise function, while Xr and Xl in the current iteration are added to an empty coordinate system to start the construction of the next segment of the piecewise function.

[0114] Since each point in the convex hull is added and deleted at most once, the overall time complexity of constructing a piecewise linear function is O(n).

[0115] Next, extract the starting point of each segment of the piecewise linear function and use it as a new ordered array f. Recursively construct the piecewise linear function following the steps above until only one segment remains, thus obtaining the piecewise linear approximation model. For example... Figure 5As shown, the bottom layer is the original mapping value array, where the first 6 numbers can be fitted with a linear function within the allowable error range, and the last 3 numbers can also be fitted with another linear function. Each starting point is combined into a new array to construct the next layer of piecewise linear functions.

[0116] (6) Establish a distribution fitting model under read and write mode.

[0117] All sub-mesh are modeled separately, for one of the sub-mesh L i The mapping values ​​of all keys in a single subgrid are calculated using the mapping function F, forming an ordered array f, where f contains n numbers.

[0118] Define a hash mapping function M:

[0119]

[0120] The range of function M is [0, len-1), where len is a constant representing how many positions a node has in the tree. Obviously, multiple f[i] may be mapped to the same position through function M. The conflict degree t is defined as the maximum number of numbers that function M can map to the same position. Now we need to find a set of k, b such that the conflict degree t is minimized, where k>0.

[0121] It is clear that the function M is monotonically increasing, and the array f to be mapped is also ordered and increasing. Therefore, M needs to satisfy the following properties:

[0122] Property 1: The number of f[i] mapped to 0 and len-1 is less than or equal to t, and f is increasing, that is:

[0123]

[0124] Subtracting the two equations, we get:

[0125]

[0126] Property 2: The mapping value of f[i] is different from the mapping value of f[i+t]. Property 1 has already guaranteed that f[i] mapped to 0 and len-1 satisfies the condition. For other mapping values, the base-taking symbol in the mapping function needs to be removed. It can be seen that if the difference between f[i] and f[i+t] before base-taking is greater than 1, then the mapping values ​​after base-taking will definitely be different, that is:

[0127]

[0128] Combining the expressions for properties 1 and 2 above, we get:

[0129]

[0130] Because f increases, the left side of the equation increases with t, and the right side decreases with t. That is, if the condition in the equation is already satisfied, increasing t will still satisfy it. We can iterate through i from smallest to largest, and increase t when the condition in the equation is not satisfied. We can find the minimum t in O(n) complexity. Substituting t back into the expression of property 1, we can find the corresponding k and b. At this point, we have obtained the function M.

[0131] When there is no conflict at a position, the multidimensional key and its corresponding payload are placed in the corresponding position. When multiple numbers conflict, these conflicting numbers are grouped into a new array f, and the function M is recursively constructed from them to obtain the distribution fitting model. This model is a tree structure, where each node has a parameter len indicating the number of positions it has. Each position is either empty, storing a key-value pair if there is no conflict, or storing a pointer to the root node of the subtree if there is a conflict. Figure 6 As shown, the black box indicates a conflict at that location, and the storage pointer points to the subtree.

[0132] (7) Storage layer design.

[0133] To reduce disk I / O overhead, some data with low storage overhead needs to be placed in memory, while data with high storage overhead is encoded and placed on disk. The grid tree model in step (1) and the multi-way partitioning tree model in step (2) are both tree structures, storing only the split points and pointers of each dimension in memory.

[0134] In read-only mode, in step (5), the key, slope, and intercept of the piecewise linear approximation model for each segment of the non-leaf nodes are stored in memory. The piecewise function of the leaf nodes, the key of the data itself, and the load value corresponding to the key are stored on disk. Each segment is divided into a page. If there are multiple segments, they are evenly divided. All pages are linked into a linked list according to their minimum mapping value. For the first page of each segment, the low address is a fixed-size area used to store the slope and intercept of the piecewise function. The following areas are evenly divided areas used to store key-value pairs. If the load value is too large and causes an overflow, the overflowing load is placed in the overflow area, and a pointer is stored in the corresponding area of ​​the original page to point to the actual data location.

[0135] In read / write mode, the distribution fitting model corresponding to each subgrid in step (6) is encoded, and the encoded data is stored on the disk as a page. If the space exceeds the size of one page, multiple pages are used for storage, and these pages are linked into a linked list. Specific encoding scheme: First, the nodes of the distribution fitting model are numbered using DFS order. Then, each node has a fixed length. For each position of a node, if it is empty, it is marked with 0; if it contains non-conflicting data, it is marked with 1 and stored as a key-value pair; if it conflicts, it is marked with 2 and a pointer to the node's number is stored. Afterward, for each node, its number, the marked array, and the key-value pair are stored in the page.

[0136] (8) Single point query.

[0137] Single-point query flowchart as follows Figure 7 As shown. First, the subgrid corresponding to a multidimensional data key is quickly located using the grid tree model in step (1) and the multi-way partitioning tree model in step (2). Then, the mapping value f of the key is obtained through the mapping function F in step (3).

[0138] In read-only mode, starting from the root of the piecewise linear approximation model tree in step (5), the index of the mapping value f is calculated according to the piecewise function. Then, binary search is used to correct within the error threshold ε to obtain the piecewise function of the next layer. Recursively, the pages in which f is located on the disk are finally obtained. These pages are read into memory, and the actual location of the data is located through its piecewise function and binary search correction, thereby reading the payload value corresponding to the key.

[0139] In read / write mode, the page corresponding to the subgrid is first loaded into memory, and then the recursion starts from the root node of the distribution fitting model in (6). The actual location of the data is quickly located according to the function M of each layer of the subgrid, so as to read the load value corresponding to the key.

[0140] (9) Range query.

[0141] A diagram illustrating the range query is shown below. Figure 8 As shown. First, based on the grid tree model in step (1) and the multi-way partitioning tree model in step (2), find all sub-regions that intersect with the range. Then, for each sub-region, first take out its two endpoints, which are the two points where the values ​​of each dimension are the minimum and the maximum, respectively. Then, use the single-point query method in step (8) to read the positions of these two endpoints. Because the mapping function F satisfies the condition that when x1 <y1,x2<y2,...,x d <y d hour, Therefore, the mapping value f of all queried points lies between the mapping values ​​of these two endpoints. Whether it's the read-only mode modeling all mapping values ​​uniformly or the read-write mode modeling a single subgrid, the order of the mapping value f is preserved. All keys and values ​​between the two endpoints can be read directly, and it can be determined whether they are within the query range. If they are, the payload value is added to the result set.

[0142] (10) Insertion, deletion and modification.

[0143] Only read-write mode supports insert, delete, and modify operations.

[0144] For the insertion operation, the position to be inserted is first found through the single-point query in step (8). If the position is empty, it is stored directly; otherwise, the data at the original position and the newly added data are organized together to construct a new function M. If for a node, the number of newly inserted data in its corresponding subtree is twice the number of data in the last reconstruction, then all the data in the subtree is taken out for reconstruction. The reconstruction method is the same as the method of recursively constructing function M in the read-write mode in (6).

[0145] For the deletion operation, the location of the data to be deleted is found through the single-point query in step (8), and the deletion is performed directly. If the size of the subtree corresponding to a node is already half of the number of data in the last reconstruction, then all the data in the subtree is taken out for reconstruction. The reconstruction method is the same as the recursive construction function M in the read-write mode in (6).

[0146] For modification operations, the location of the data to be modified is found through the single-point query in step (8). If the load value is modified, it is modified directly. If the key is modified, it is split into deleting the original data and inserting the new data. The above corresponding operations can be performed.

[0147] It should be noted that the hierarchical structure and process flow of the systems and methods described above in this invention can be implemented as software functional units in the form of logical instructions in memory. When this software is sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention.

[0148] It should also be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the division of steps or levels in the systems and methods described above is merely a logical functional division. In actual implementation, there may be other division methods. For example, multiple steps or levels may be combined or integrated together, and a single step or level may also be split.

[0149] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention. Therefore, all technical solutions obtained through equivalent substitution or transformation fall within the protection scope of the present invention.

Claims

1. A data storage system based on a scalable multidimensional learning index structure, used to organize multidimensional data into memory and disk, characterized in that, The data storage system includes a grid partitioning layer, a data fitting layer, a storage layer, and a computation layer; The grid partitioning layer consists of two models. The first layer is a grid tree model, which is used to divide the high-dimensional key space of multidimensional data into multiple sub-regions with evenly distributed query load. The second layer is a multi-way partitioning tree model, which is used to further divide each sub-region obtained by the grid tree model into sub-grids with evenly distributed data space. The data fitting layer is used to reduce the dimensionality of multidimensional data through a mapping function and construct a corresponding data fitting model according to the currently selected mode of the data storage system to achieve data location; the optional modes of the data storage system are read-only mode and read-write mode, and different data fitting models are corresponding to the two modes respectively. The storage layer is used to organize, encode, and store the high-dimensional key space and the model data established by the grid partitioning layer and the data fitting layer. When storing, the high-dimensional key space data is stored on the disk in multiple data pages, while part of the index structure is stored in memory and the rest is stored on the disk. The computing layer is used to provide data CRUD services to the outside world through the reserved interface of the storage layer; When the data storage system is currently selected as read-only mode, the data fitting layer establishes a piecewise linear approximation model for data fitting, and uses this data fitting model to segment all keys to accelerate the search. The piecewise linear approximation model first forms an ordered array of the mapping values ​​of all keys in one-dimensional space, and then performs piecewise fitting on the overall data distribution of the array, dividing the array into many segments. Each segment is fitted with a linear function while ensuring a certain error range. Then, for each segment, the mapping value at the beginning of the segment is extracted to form a new array. The piecewise fitting is performed recursively until a certain array can be represented by only one segment, and finally a tree structure is formed. When the data storage system is currently selected as read / write mode, the data fitting layer establishes a data fitting model as a distribution fitting model and uses this distribution fitting model to organize the data distribution of each sub-grid. The distribution fitting model fits the mapping values ​​of each sub-grid obtained by the multi-way partitioning tree model. First, a hash function is constructed to hash these mapping values ​​into a certain interval while ensuring a certain degree of hash collision. For the hash collision mapping values, they are organized together to recursively construct a hash function, and finally a tree structure is formed.

2. The data storage system based on a scalable multidimensional learning index structure as described in claim 1, characterized in that, The grid tree model in the grid partitioning layer is constructed as follows: First, historical query records are sampled, and the sampled query records are projected onto each dimension to obtain the distribution of queries in each dimension; then, a tree structure is constructed based on the probability density of the query distribution to divide each dimension. Each layer of the tree selects one dimension for division, and each time the key space represented by a node is divided into multiple regions. After dividing all dimensions, the query distribution in each sub-region is made as uniform as possible. The multi-way partitioning tree model in the mesh partitioning layer is a tree structure. Each node in this tree structure represents a subspace in the global multidimensional key space, and the root node represents the key space corresponding to the sub-region. In the process of constructing a multi-way partitioning tree model, one dimension of the subspace is split at each step. For the selected splitting dimension, the cumulative distribution function of the multidimensional data in the subspace on that dimension is first obtained. Then, the subspace is split uniformly according to the cumulative distribution function. The split subspaces constitute the next layer node of the multi-way partitioning tree model. The lower layer node is iteratively split.

3. The data storage system based on a scalable multidimensional learning index structure as described in claim 1, characterized in that, In the data fitting layer, the specific method for dimensionality reduction of multidimensional data using a mapping function is as follows: First, the subgrids obtained by the multi-way partitioning tree model are numbered. Then, a mapping function is constructed to map the multidimensional data in each subgrid, and the mapping value maintains the order of the multidimensional data. That is, for different subgrids, the smaller the number of the subgrid, the smaller the data mapping value. For the same subgrid, the smaller the data value of all dimensions within the subgrid, the smaller the data mapping value. Finally, the mapping function is used to calculate the mapping value of all keys in one-dimensional space.

4. The data storage system based on a scalable multidimensional learning index structure as described in claim 1, characterized in that, The currently selected mode of the data storage system is chosen by the user by sending a switching command according to their application scenario.

5. The data storage system based on a scalable multidimensional learning index structure as described in claim 1, characterized in that, In the storage layer, the multidimensional data and corresponding models are stored in different forms, as detailed below: First, both the grid tree model and the multi-way partitioning tree model only store the data splitting points and the pointers corresponding to the splitting points. They are both tree structures and are stored in memory. Secondly, in read-only mode, for all non-leaf nodes of the tree structure constructed by the piecewise linear approximation model, the linear fitting function and corresponding fitting data of each non-leaf node are also placed in memory. For each leaf node of the piecewise linear approximation model, its linear fitting function and corresponding fitting data are placed on the disk. A page size is set, and these data are stored sequentially into each page of the disk, forming a linked list structure between the pages. In read-write mode, the entire tree structure constructed by the distribution fitting model for each subgrid is first numbered using DFS order, then the adjacency list is used to represent the tree structure, and finally the tree structure is encoded and stored into multiple consecutive pages on the disk.

6. The data storage system based on a scalable multidimensional learning index structure as described in claim 1, characterized in that, The data CRUD services provided in the computing layer include single-point query, range query, data insertion, data deletion, and data modification. However, data insertion, data deletion, and data modification are only supported in read-write mode and are not supported in read-only mode. The single-point query is used to retrieve the value corresponding to a key. The process is as follows: for a key, first find its subgrid according to the grid tree model and the multi-way partitioning tree model, and then search in the tree structure under different modes according to the mapping value of the key through the data fitting layer. The range query is used to query all values ​​in a high-dimensional key space. The method is as follows: For the query space, find out which sub-grids it intersects with in the multi-way partitioning tree model to obtain each intersecting sub-region. Then, perform a single-point query on the two endpoints of each sub-region. These two endpoints are the two points where the values ​​of each dimension are the minimum and the maximum, respectively. After finding the two endpoints, scan between the endpoints and save all data in the query space. The method for inserting data is as follows: first, perform a single-point query to find the position to be inserted on the leaf node of the tree structure of the distribution fitting model. If the position to be inserted is empty, insert directly; otherwise, construct the next level node and then insert. The method for deleting data is as follows: first, perform a single-point query to find the position to be deleted on the leaf node of the tree structure of the distribution fitting model; if the data to be deleted is at the position to be deleted, then delete it directly. The method for modifying the data is as follows: First, perform a single-point query to find the position to be modified on the leaf node of the tree structure of the distribution fitting model. If the value to be modified is to be modified, the position to be modified is modified directly. If the key to be modified is to be modified, the operation is split into two parts: deleting the original data and inserting the new data at the position to be modified. Inserting, deleting, or modifying data will all change the tree structure of the distribution fitting model. When the amount of data change meets the preset conditions, the tree structure needs to be reconstructed.

7. The data storage system based on a scalable multidimensional learning index structure as described in claim 1, characterized in that, The high-dimensional key space is composed of some attributes extracted from the multi-attribute database table, with the remaining attributes in the database table serving as the values ​​of the high-dimensional key.

8. The data storage system based on a scalable multidimensional learning index structure as described in claim 1, characterized in that, The multidimensional data is geospatial data.

9. A data storage method based on a scalable multidimensional learning index structure, used to organize multidimensional data into memory and disk, characterized in that, include: S1. Construct a two-layer model to perform grid partitioning on the high-dimensional key space of multidimensional data. The first layer is a grid tree model, which is used to divide the high-dimensional key space of multidimensional data into multiple sub-regions with evenly distributed query load. The second layer is a multi-way partitioning tree model, which is used to further divide each sub-region obtained by the grid tree model into sub-grids with evenly distributed data space. S2. Dimensionality reduction of multidimensional data is performed through a mapping function, and a corresponding data fitting model is constructed according to the currently selected mode of the data storage system to achieve data location; the optional modes of the data storage system are read-only mode and read-write mode, and different data fitting models are corresponding to the two modes respectively. S3. Organize, encode, and store the high-dimensional key space and the model data established by the grid partitioning layer and the data fitting layer. When storing, the high-dimensional key space data is stored on the disk in multiple data pages, while part of the index structure is stored in memory and the rest is stored on the disk. S4. Provide data CRUD services to the outside world through the reserved interface of the storage layer; When the data storage system is currently selected as read-only mode, the data fitting layer establishes a piecewise linear approximation model for data fitting, and uses this data fitting model to segment all keys to accelerate the search. The piecewise linear approximation model first forms an ordered array of the mapping values ​​of all keys in one-dimensional space, and then performs piecewise fitting on the overall data distribution of the array, dividing the array into many segments. Each segment is fitted with a linear function while ensuring a certain error range. Then, for each segment, the mapping value at the beginning of the segment is extracted to form a new array. The piecewise fitting is performed recursively until a certain array can be represented by only one segment, and finally a tree structure is formed. When the data storage system is currently selected as read / write mode, the data fitting layer establishes a data fitting model as a distribution fitting model and uses this distribution fitting model to organize the data distribution of each sub-grid. The distribution fitting model fits the mapping values ​​of each sub-grid obtained by the multi-way partitioning tree model. First, a hash function is constructed to hash these mapping values ​​into a certain interval while ensuring a certain degree of hash collision. For the hash collision mapping values, they are organized together to recursively construct a hash function, and finally a tree structure is formed.

Images