An Automatic Database Partitioning Method and System Based on Depth Map Compression Algorithm

Abstract

This invention discloses an automatic database partitioning method and system based on a depth graph compression algorithm. The method includes: acquiring the workload and dataset of query statements in the input database, as well as the query relationships of relational tables; constructing a graph model for selecting partitioning columns using the extracted workload features, data features, and query relationships; extracting the self-features and subgraph structure information of each data column node in the graph model using a depth graph compression model; inputting the self-features and subgraph structure information of each data column node into a partitioning column selection model to obtain the selected partitioning columns; and performing table partitioning of the database based on the partitioning columns. This invention can effectively estimate the quality of database partitioning.

Description

Technical Field

[0001] This invention relates to the field of information retrieval technology, and in particular to an automatic database partitioning method and system based on a depth map compression algorithm. Background Technology

[0002] Given a set of tables, database partitioning aims to select a partitioning key (single-column or multi-column) for each table and use that key to distribute rows into different partitions. Database partitioning is crucial in distributed databases to meet query performance requirements (e.g., high throughput, low latency).

[0003] However, database partitioning is an NP-hard problem, and selecting a partitioning key from numerous columns (e.g., millions of columns in a commercial database) to optimize query performance is extremely costly. Traditional exhaustive methods (such as dynamic programming) enumerate every possible column combination and select the highest-performing combination as the partitioning key, which is very time-consuming and difficult to use in practice. Heuristic methods search for the maximum spanning tree of the tables on the schema graph, but fail to obtain high-quality partitions because they primarily consider foreign key constraints between tables and cannot capture correlations between columns (e.g., equi-joins), significantly impacting query performance.

[0004] Database partitioning based on deep reinforcement learning (DRL) also has some limitations. First, it fails to capture rich data features (e.g., data distribution), column features (e.g., join selectivity), and column relevance, which significantly diminishes the advantages of using columns as partitioning keys. Second, it struggles to adapt to different scenarios (e.g., different queries and datasets) because it only supports a limited set of query templates and cannot capture data features. Third, they require evaluating the quality of the partitioning strategy. The approach of partitioning first and then evaluating partition quality is time-consuming. For example, training a reinforcement learning model can take over 13 hours. Furthermore, simple cost models cannot effectively estimate the quality of database partitions, resulting in poor partition quality. Summary of the Invention

[0005] The present invention aims to at least partially solve one of the technical problems in the related art.

[0006] Therefore, the purpose of this invention is to propose an automatic database partitioning method based on a depth graph compression algorithm, which can improve query performance by selecting a partitioning strategy and effectively estimate the quality of database partitions.

[0007] Another objective of this invention is to propose an automatic database partitioning system based on a depth map compression algorithm.

[0008] To achieve the above objectives, this invention proposes an automatic database partitioning method based on a depth map compression algorithm, comprising:

[0009] Obtain the workload, dataset, and query relationships of the input database's query statements;

[0010] A graph model for selecting partitioning columns is constructed using the extracted workload features, data features, and the query relationships.

[0011] The depth graph compression model is used to extract the individual features and subgraph structure information of each data column node in the graph model;

[0012] The unique characteristics and subgraph structure information of each data column node are input into the partition column selection model to obtain the selected partition column, and the database table partitioning is performed based on the partition column.

[0013] The automatic database partitioning method based on the depth map compression algorithm according to embodiments of the present invention may also have the following additional technical features:

[0014] Furthermore, in one embodiment of the present invention, the query performance of the database under the partitioning strategy based on the selected partitioning column is evaluated using a trained evaluation model to obtain the query efficiency under different partitioning strategies. The method further includes:

[0015] Offline training: Obtain system logs from different databases to generate a first data row graph, use the evaluation model to estimate the performance of the first data row graph, and update the graph weights in the evaluation model based on the performance estimation results and the loss value of gradient descent to train the initial evaluation model;

[0016] Online training: The selected partition column is input into the initial evaluation model to generate a second data row graph. Based on the second data row graph, the latency and throughput are estimated. The network weights in the initial evaluation model are optimized according to the latency and throughput estimation results to obtain a trained evaluation model.

[0017] Furthermore, in one embodiment of the present invention, the step of constructing a graph model for selecting partitioning columns using the extracted workload features and data features and the query relationship includes:

[0018] Calculate the query characteristics and data characteristics of each data column node in the relation table; and,

[0019] Based on the relevant data columns and call status in the query statement, edges are constructed between the corresponding data column nodes to build a graph model.

[0020] Furthermore, in one embodiment of the present invention, the step of extracting the self-features and subgraph structure information of each data column node of the graph model using a depth graph compression model includes:

[0021] The depth graph compression model is used to perform feature selection and compression on the subgraph structure of each data column node in the graph model to obtain the individual features and subgraph structure information of each data column node; and,

[0022] The subgraph structure information is mapped to a low-dimensional feature vector through forward propagation to obtain the mapping result.

[0023] Furthermore, in one embodiment of the present invention, the step of inputting the self-features and subgraph structure information of each data column node into the partition column selection model to obtain the selected partition column includes:

[0024] The self-features and subgraph structure information of each data column node are analyzed using correlation analysis to obtain a vertex set with preset conditions; wherein, the vertex set with preset conditions includes vertices of each data column node whose correlation with other nodes is greater than a first preset threshold and whose compression density is greater than a second preset threshold.

[0025] The partition column is obtained based on the vertex set.

[0026] To achieve the above objectives, another aspect of the present invention proposes an automatic database partitioning system based on a depth map compression algorithm, comprising:

[0027] The data collection module is used to obtain the workload and dataset of the query statements in the input database, as well as the query relationships of the relational tables;

[0028] The data preprocessing module is used to construct a graph model for selecting partitioning columns using the extracted workload features and data features as well as the query relationships;

[0029] The depth graph compression module is used to extract the self-features and subgraph structure information of each data column node of the graph model using the depth graph compression model;

[0030] The partition column selection module is used to input the characteristics and subgraph structure information of each data column node into the partition column selection model to obtain the selected partition column, and to perform table partitioning of the database based on the partition column.

[0031] The automatic database partitioning method and system based on the depth graph compression algorithm of this invention can improve query performance by selecting a partitioning strategy and can effectively estimate the quality of database partitions.

[0032] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0033] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0034] Figure 1 This is a flowchart of an automatic database partitioning method based on a depth map compression algorithm according to an embodiment of the present invention;

[0035] Figure 2 This is a schematic diagram of the overall framework according to an embodiment of the present invention;

[0036] Figure 3 A schematic diagram illustrating an example of a data column chart constructed according to an embodiment of the present invention;

[0037] Figure 4 This is a framework diagram of the depth map compression model according to an embodiment of the present invention;

[0038] Figure 5 This is a schematic diagram of an automatic database partitioning system based on a depth map compression algorithm according to an embodiment of the present invention. Detailed Implementation

[0039] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0040] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0041] The automatic database partitioning method and system based on the depth map compression algorithm proposed according to embodiments of the present invention are described below with reference to the accompanying drawings.

[0042] The overall framework of the present invention is as follows Figure 2 As shown, database partitioning includes data preprocessing, depth graph compression, partition column selection, and performance evaluation. Data preprocessing sequentially includes node generation and edge generation. Node generation extracts query and data features relevant to partition selection. Edge generation constructs a graph model using the query relationships between data columns. Depth graph compression parses the graph model generated based on data columns into a feature set for each node. Partition column selection selects partition columns based on the subgraph features of each data column. Therefore, for the currently submitted query statement and relational tables, the appropriate partition columns are automatically selected through the column graph model, resulting in optimal query performance after database partitioning.

[0043] Figure 1 This is a flowchart of an automatic database partitioning method based on a depth map compression algorithm according to an embodiment of the present invention.

[0044] like Figure 1 As shown, the method includes, but is not limited to, the following steps:

[0045] S1, obtain the workload and dataset of the query statement in the input database, as well as the query relationships of the relational tables.

[0046] As we can understand, table partitioning refers to distributing rows of a table across different nodes based on certain columns (partitioning columns) and a partitioning function (such as range partitioning). Database partitioning refers to selecting appropriate partitioning columns for all tables in the database to optimize query performance after partitioning. A column graph is an undirected, cyclic graph with columns as vertices and the query relationships between columns as edges. The function of a database partitioning system is to automatically select appropriate partitioning columns based on the currently submitted query and relational tables, thereby optimizing query performance after database partitioning.

[0047] The embodiments of the present invention first define the relevant concepts of query performance, table partitioning, database partitioning, and query relationships between columns. Preferably, in the present invention, a workload and dataset consisting of a batch of parallel query statements can be selected.

[0048] S2 uses the extracted workload features, data features, and query relationships to construct a graph model for selecting partitioning columns.

[0049] It is understood that the embodiments of the present invention use data column information to construct a graph model for selecting partition columns. The method used is to construct edges between the corresponding data column nodes based on the columns involved in the query statement and the cases in which they are called together (e.g., joint connection), representing the access relationship between data columns.

[0050] Specifically, the present invention needs to consider node generation and edge generation during data preprocessing.

[0051] First, node generation involves creating node features based on the data in the data columns and query characteristics. This invention models the data columns of each table and their relationships as a graphical model. Nodes are the columns involved in the query, and this invention encodes the features of each node into a vector containing six dimensions (e.g., ...). Figure 3As shown): (1) Table number: indicates the table to which the data column belongs; (2) Table size: the number of rows in the table to which this column belongs; (3) Row selectivity: the number of distinct values ​​in the column; (4) Row length: the maximum number of bytes in a row in this table; (5) Scan operation frequency: the number of selection operations involving this column; (6) Aggregation operation frequency: the number of aggregation operations involving this column. Furthermore, even if different nodes have similar column characteristics, their importance to database partitioning may differ. For example, in Figure 3 In the graph, p_partkey and p_name come from the same table. However, p_name has only one joined column, while p_partkey has seven directly joined columns, and partitioning by using joined columns yields a greater partitioning advantage. Furthermore, importance-related features are scattered throughout the graph. To address this issue, this invention proposes an undirected graph-based attention model to compute node importance based on graph structure features. Finally, given a node c... j This invention will assign node importance a(c) j Multiply by vector V(c) j The six dimensions of ) are used to obtain the weighted vertex vector, namely V(c j )=a(c j )V(c j For example, for the l_suppkey column, this invention calculates the local feature vector as V(l_suppkey) and calculates the joint feature vector of the seven connected columns; and uses these two vectors to calculate a scalar value of 0.13 as the vertex weight. Then, this invention calculates the weighted vertex vector as V(l_suppkey) = 0.13V(l_suppkey) = [130, 1950, 3.25, 47.45, 1.56, 0.39].

[0052] Second, edge generation generates edge weights based on the query relationships between data columns. This invention considers two main features to calculate edge weights. (1) Frequency of join predicates. The frequency of join predicates between two columns is the number of queries that include this join predicate. This is important for database partitioning because if two columns are joined frequently, this invention can save more remote joins by using the two columns as partition keys. (2) Cardinality of join predicates. The cardinality of join predicates is also important because cardinality estimates the size of the result set and can reflect execution costs. Note that existing cardinality estimation methods cannot be applied here because they rely on the query plan, which cannot be obtained before partitioning. Therefore, this invention samples rows from the table and estimates the cardinality by executing predicates on the sampled rows. For example, this invention samples 1% of the rows from 10G of data and obtains 5 results by executing predicates on the sampled rows. Then, this invention approximates the cardinality of the predicate as 5 / 1% = 500.

[0053] S3 utilizes a depth graph compression model to extract the unique features and subgraph structure information of each data column node in the graph model.

[0054] Specifically, in this embodiment of the invention, the graph model generated based on the data column is parsed into a feature set of each node. The method used is to use a depth graph compression model to select and compress features for the k-hop subgraph structure of each node, and to map the k-hop subgraph structure information to a low-dimensional feature vector through forward propagation.

[0055] The original data row graph is compressed, mapping the high-dimensional graph structured features of the original load graph to an abstract feature space strongly correlated with performance characteristics. For example... Figure 4 As shown, the depth graph compression network first extracts the k-hop neighbor information of each node through stacked graph convolutional layers, denoted as D. i =|E i T E :,i |H l-1 Where i represents node v i E i T Represents node v i out-degree neighbor node, E :,i Represents node v i H is the in-degree neighbor node. l-1 This represents the compression matrix obtained after convolution of the first (l-1) layers of graphs. Then, a nonlinear rectified layer is used to further compress the node features, denoted as H. l =β l (D -1 / 2 ED -1 / 2 W l H l-1 ), where β represents the nonlinear rectifier layer, E represents the edge matrix, D represents the neighbor matrix of the node, and W l Let W represent the network parameters of the l-th layer. This invention updates the network parameters W using the loss value calculated from the training bias, thereby efficiently learning a compressed network structure based on the target performance. Furthermore, this invention finds that the neighbor matrix of a node has a high dimension (each bit corresponds to a two-dimensional neighbor matrix), and this invention needs to reduce the dimension of the compressed matrix H to reduce computational overhead. Therefore, this invention approximates the network's weight matrix W and neighbor matrix H using Chebyshev's truncated expansion, denoted as W = W * H. Where l represents the first l layers, θ′ k The weights of the k-th layer are represented by the Chebyshev polynomial T. k (V) is defined as T during iteration. k (V)=2T k-1 (V)-T k-2 (V), where T oChebyshev polynomial t(x) = 1, T1(x) = x. k (V) has been shown to effectively reduce the matrix dimension in forward propagation, thereby saving compression time. As shown in Table 1:

[0056] Table 1

[0057]

[0058] S4: Input the characteristics and subgraph structure information of each data column node into the partition column selection model to obtain the selected partition column, and perform table partitioning of the database based on the partition column.

[0059] It is understood that, in this embodiment of the invention, the partitioning column is selected based on the subgraph features of each data column. The method used is to use relevance analysis to parse the combination of vertices with high relevance to other nodes and high compression density from the compressed graph model as the partitioning column, so as to optimize query performance.

[0060] It is understandable that data column encoding can be divided into two categories: one is the node information on the graph, where each node corresponds to a data column, including the sequence number of the relation table to which it belongs, the size of the relation table, the number of distinct values, the number of bytes in the data row, and the read / write frequency; the other is the edge information between nodes, which mainly refers to the connection relationship in the query statement.

[0061] Specifically, the partitioning column is selected based on the compressed node features. This invention utilizes Taylor decomposition to extract the importance of each node from the compressed vector and selects the partitioning column based on the importance of each vertex. Similar to training a neural network, node importance corresponds to the gradient calculated when calculating the backpropagation loss value of a graph propagation network, and its absolute value reflects the importance of each neuron. This invention writes the embedding vector as X = {x1, x2, ..., xn} and represents the total correlation as R = ∑ x∈X R(x). Assuming the graph compression network has L layers, this invention initializes the partitioning column selection network with L layers, and the units in the two models are in one-to-one correspondence. For example, in Figure 4 In this paper, the graph compression network has a graph compression layer and a ReLU layer, therefore L=2. In the input layer, this invention represents the importance of each unit as 1, i.e. It represents the importance of columns that have not yet been selected. Furthermore, for other hidden layers l, the importance of any unit i in the next layer (l-1) is the sum of the importance of all neurons from unit i to layer l. Using Taylor's theorem, this invention knows that any continuous function can be represented as an infinite polynomial whose efficiency is the derivative of that function. Therefore, in key selection networks, this invention calculates the derivative of the output function with respect to the input variables and uses the derivative of each input variable as the importance of the corresponding unit.

[0062] Furthermore, this invention can utilize a performance evaluation model to evaluate query performance under a selected partitioning strategy. The performance of load execution is evaluated based on the selected strategy. This invention first generates a k-partition graph model, where nodes are data partitions, edges are the connectivity relationships between sampled rows, and k represents the number of partitions (number of nodes). Then, this invention uses graph representation learning to map the entire graph to query performance. The model of this invention supports changes in k because it reflects the partitioning differences in the graph and can be adapted to different graph structures through graph compression. To model the partitioning strategy, this invention first samples rows from the table using hierarchical sampling to reflect the data distribution of a finite number of rows. Then, based on the selected key and database partitioning capabilities, the sampled rows are distributed across k partitions. Next, this invention calculates edge weights and node weights according to the methods discussed in the data preprocessing module. Note that the execution costs of connections and cross-partitions within a partition can be very different; this invention uses graph embedding to learn the differences between trainable graph weights. Thus, this invention generates a k-partition sample graph representing the query and data distribution across k partitions. Therefore, this invention needs to learn the actual weight distribution of edges within the same partition and the actual weight distribution of edges across different partitions. Secondly, this invention cannot directly map the k-partition sample graph to query performance, as this is affected by factors such as configuration (e.g., buffer size, maximum concurrency) and scheduling order. This invention uses deep learning to identify the query performance variations caused by different partitioning strategies.

[0063] Furthermore, this invention can collect historical data to train a depth graph compression model, a partitioning column selection model, and a performance evaluation model, and then deploy the trained models into the system. First, the evaluation model is trained using historical queries and actual performance against different partitioning strategies (offline training). Then, this invention uses the evaluation model to assess the performance of the new partitioning strategy recommended by the partitioning column selection model, and fine-tunes the evaluation model based on the actual performance after partitioning (online training). Since performance metrics may contain a lot of noise caused by non-partitioning factors (such as different connection orders and random mutexes), this invention uses a highly robust loss function, Least Mean Square (LMLS). LMLS reduces the side effects of noise by gradually reducing the impact of large errors (taking the logarithm) and calculating the average of all training samples. In the offline training phase: This invention first trains the performance evaluation model using historical samples {Q, D, C, k, P}. Unlike the training data for the partitioning column selection model, this invention collects system logs from different databases to generate different data row graphs and obtain actual performance data. During training the evaluation model, this invention uses the evaluation model to estimate the performance of different data row graphs and updates the graph weights in the evaluation model based on the loss value of gradient descent. The model converges when the loss value does not change significantly (e.g., the error does not exceed a threshold after more than 20 iterations). Online training: For a new partitioning request, the evaluation model takes as input the partitioning columns selected by the partitioning column selection model and generates a new data row graph. The evaluation model then estimates latency and throughput based on the data row graph. If the reward calculated based on latency and throughput is high enough, this invention partitions the table using the selected columns and obtains the actual performance on these partitions to optimize the network weights in the evaluation model.

[0064] The automatic database partitioning method based on the depth map compression algorithm according to embodiments of the present invention can effectively estimate the quality of database partitions.

[0065] To achieve the above embodiments, such as Figure 5 As shown, this embodiment also provides an automatic database partitioning system 10 based on a depth map compression algorithm. The system 10 includes: a data collection module 100, a data preprocessing module 200, a depth map compression module 300, and a partition column selection module 400.

[0066] The data collection module 100 is used to obtain the workload and dataset of the query statements in the input database, as well as the query relationships of the relational tables;

[0067] The data preprocessing module 200 is used to construct a graph model for selecting partitioning columns by utilizing the extracted workload features, data features, and query relationships;

[0068] The depth graph compression module 300 is used to extract the self-features and subgraph structure information of each data column node of the graph model using the depth graph compression model;

[0069] The partition column selection module 400 is used to input the characteristics and subgraph structure information of each data column node into the partition column selection model to obtain the selected partition column, and to perform table partitioning of the database based on the partition column.

[0070] Furthermore, following the partition column selection module 400, the system 10 also includes a performance evaluation model, used to evaluate the query performance of the database under the partitioning strategy based on the partition column selection using the trained evaluation model to obtain the query efficiency under different partitioning strategies, including:

[0071] Offline training: Obtain system logs from different databases to generate the first data row graph, use the evaluation model to estimate the performance of the first data row graph, and update the graph weights in the evaluation model based on the performance estimation results and the loss value of gradient descent to train the initial evaluation model;

[0072] Online training: Input the selected partition column into the initial evaluation model to generate a second data row graph. Estimate latency and throughput based on the second data row graph. Optimize the network weights in the initial evaluation model based on the latency and throughput estimation results to obtain the trained evaluation model.

[0073] Furthermore, the data preprocessing module 200 includes a data column encoding module and a graph generation module; wherein,

[0074] The data column encoding module is used to calculate the query features and data features of each data column node in the relational table; and,

[0075] The graph generation module is used to construct edges between corresponding data column nodes based on the relevant data columns and call status in the query statement to build a graph model.

[0076] Furthermore, the depth map compression module 300 is also used for:

[0077] Feature selection and compression are performed on the subgraph structure of each data column node in the graph model to obtain the self-feature and subgraph structure information of each data column node. The subgraph structure information is then mapped to a low-dimensional feature vector through forward propagation to obtain the mapping result.

[0078] Furthermore, the column selection module 400 is also used for:

[0079] The relevance analysis is used to analyze the self-features and subgraph structure information of each data column node to obtain a vertex set with preset conditions; wherein, the vertex set with preset conditions includes vertices of each data column node whose relevance to other nodes is greater than a first preset threshold and whose compression density is greater than a second preset threshold.

[0080] The partitioning column is obtained based on the vertex set.

[0081] The automatic database partitioning system based on the depth map compression algorithm according to embodiments of the present invention can effectively estimate the quality of database partitions.

[0082] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0083] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0084] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. An automatic database partitioning method based on a depth map compression algorithm, characterized in that, Includes the following steps: Obtain the workload, dataset, and query relationships of the input database's query statements; A graph model for selecting partitioning columns is constructed using the extracted workload features, data features, and the query relationship. The graph model uses data columns as nodes, and the node features include table size, row selectivity, scan operation frequency, and aggregation operation frequency. Edges are constructed between nodes based on the join predicates between data columns in the query statement, and the edge weights are determined based on the frequency of occurrence of the join predicates and the cardinality of the predicates. The depth graph compression model is used to extract the self-features and subgraph structure information of each data column node of the graph model; wherein, the depth graph compression model extracts the features of the k-hop subgraph structure of each node based on graph convolution, and maps the subgraph structure information into a low-dimensional feature vector through forward propagation; The self-features and subgraph structure information of each data column node are input into the partition column selection model to obtain the selected partition column. Based on the low-dimensional feature vector, correlation analysis is used to obtain a vertex set with preset conditions. The vertex set with preset conditions includes vertices where the correlation between each data column node and other nodes is greater than a first preset threshold and the compression density is greater than a second preset threshold. The database is then partitioned based on the partition column.

2. The method according to claim 1, characterized in that, The method further includes evaluating the database query performance under a partitioning strategy based on the selected partitioning column using a trained evaluation model to obtain the query efficiency under different partitioning strategies. Offline training: Obtain system logs from different databases to generate a first data row graph, use the evaluation model to estimate the performance of the first data row graph, and update the graph weights in the evaluation model based on the performance estimation results and the loss value of gradient descent to train the initial evaluation model; Online training: The selected partition column is input into the initial evaluation model to generate a second data row graph. Based on the second data row graph, the latency and throughput are estimated. The network weights in the initial evaluation model are optimized according to the latency and throughput estimation results to obtain a trained evaluation model.

3. The method according to claim 2, characterized in that, The step of constructing a graph model for selecting partitioning columns using extracted workload features, data features, and query relationships includes: Calculate the query characteristics and data characteristics of each data column node in the relation table; and, Based on the relevant data columns and call status in the query statement, edges are constructed between the corresponding data column nodes to build a graph model.

4. The method according to claim 3, characterized in that, The extraction of the intrinsic features and subgraph structure information of each data column node of the graph model using the depth graph compression model includes: The depth graph compression model is used to perform feature selection and compression on the subgraph structure of each data column node in the graph model to obtain the individual features and subgraph structure information of each data column node; and, The subgraph structure information is mapped to a low-dimensional feature vector through forward propagation to obtain the mapping result.

5. The method according to claim 4, characterized in that, The step of inputting the self-features and subgraph structure information of each data column node into the partition column selection model to obtain the selected partition column includes: The partition column is obtained based on the vertex set.

6. An automatic database partitioning system based on a depth map compression algorithm, characterized in that, include: The data collection module is used to obtain the workload and dataset of the query statements in the input database, as well as the query relationships of the relational tables; The data preprocessing module is used to construct a graph model for selecting partitioning columns using the extracted workload features, data features, and the query relationship; wherein, the graph model uses data columns as nodes, and the node features include table size, row selectivity, scan operation frequency, and aggregation operation frequency, and edges are constructed between nodes according to the connection predicates between data columns in the query statement, and the edge weights are determined based on the occurrence frequency and cardinality of the connection predicates; The depth graph compression module is used to extract the self-features and subgraph structure information of each data column node of the graph model using the depth graph compression model; wherein, the depth graph compression model extracts features of the k-hop subgraph structure of each node based on graph convolution, and maps the subgraph structure information into a low-dimensional feature vector through forward propagation; The partition column selection module is used to input the self-features and subgraph structure information of each data column node into the partition column selection model to obtain the selected partition column. Specifically, based on the low-dimensional feature vector, correlation analysis is used to obtain a vertex set with preset conditions based on the self-features and subgraph structure information of each data column node. The vertex set with preset conditions includes vertices where the correlation between each data column node and other nodes is greater than a first preset threshold and the compression density is greater than a second preset threshold. The database is then partitioned based on the partition column.

7. The system according to claim 6, characterized in that, Following the partition column selection module, the system further includes a performance evaluation model for evaluating the database query performance under the partitioning strategy based on the partition column selection, thereby obtaining query efficiency under different partitioning strategies. This includes: Offline training: Obtain system logs from different databases to generate a first data row graph, use the evaluation model to estimate the performance of the first data row graph, and update the graph weights in the evaluation model based on the performance estimation results and the loss value of gradient descent to train the initial evaluation model; Online training: The selected partition column is input into the initial evaluation model to generate a second data row graph. Based on the second data row graph, the latency and throughput are estimated. The network weights in the initial evaluation model are optimized according to the latency and throughput estimation results to obtain a trained evaluation model.

8. The system according to claim 7, characterized in that, The data preprocessing module includes a data column encoding module and a graph generation module; wherein, The data column encoding module is used to calculate the query features and data features of each data column node in the relation table; and, The graph generation module is used to construct edges between corresponding data column nodes to build a graph model based on the relevant data columns and call status in the query statement.

9. The system according to claim 8, characterized in that, The depth map compression module is also used for: Feature selection and compression are performed on the subgraph structure of each data column node in the graph model to obtain the self-feature and subgraph structure information of each data column node, and the subgraph structure information is mapped to a low-dimensional feature vector through forward propagation to obtain the mapping result.

10. The system according to claim 9, characterized in that, The column selection module is also used for: The partition column is obtained based on the vertex set.

Images