Domain adaptive database cardinality estimation method and apparatus
By employing a domain-adaptive database cardinality estimation method, and utilizing pre-training and adversarial training of feature extractors and domain classifiers, domain-invariant feature representations are generated. This addresses the issue of decreased cardinality estimation accuracy caused by workload drift and improves query execution efficiency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SOUTH CENTRAL UNIVERSITY FOR NATIONALITIES
- Filing Date
- 2026-02-10
- Publication Date
- 2026-06-05
Smart Images
Figure CN122152871A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of database technology, and in particular to a domain-adaptive database cardinality estimation method and device. Background Technology
[0002] Cardinality estimation is a crucial function of database query optimizers. By encoding the user's query into a query feature vector and then feeding it into a database cardinality estimation model, the number of results a query can return can be predicted. The accuracy of this prediction directly impacts the efficiency of the database query optimizer in selecting an execution plan; low prediction accuracy can lead to inefficient query execution.
[0003] In practical applications, database cardinality estimation models often experience a significant drop in accuracy when the workload changes. This is because the query patterns used during model training ("source domain workload") differ from the actual query patterns used in use ("target domain workload"), resulting in workload drift. Current methods typically address this issue by retraining or fine-tuning the model.
[0004] However, since retraining or fine-tuning a model requires a large amount of labeled data, and frequent fine-tuning or retraining consumes a lot of computing resources, and there is still a problem that training delays cause cardinality estimation to be unable to adapt quickly to changes in workload, there is an urgent need for a cardinality estimation method to meet the cardinality estimation requirements under complex and variable workloads. Summary of the Invention
[0005] The main objective of this application is to provide a domain-adaptive database cardinality estimation method and device, which aims to solve the technical problem that existing cardinality estimation models are unable to address the decrease in cardinality estimation accuracy when the workload shifts.
[0006] To achieve the above objectives, this application proposes a domain-adaptive database cardinality estimation method. The method is applied to a device deployed with a cardinality estimation model, which includes a feature extractor, a cardinality estimator, and a domain classifier. The feature extractor is connected to both the cardinality estimator and the domain classifier. The method includes: Obtain query training data, which includes source domain training data and target domain training data; During the pre-training phase, the source domain training data is input into the feature extractor, the cardinality prediction value is obtained through the cardinality estimator, and the model parameters of the cardinality estimation model are updated based on the cardinality prediction value and the source domain training data. During the adversarial training phase, the source domain training data and the target domain training data are respectively input into the feature extractor after the pre-training phase. The domain classifier obtains the domain probability distribution value, and the model parameters of the cardinality estimation model after the pre-training phase are updated based on the domain probability distribution value, the source domain training data, and the target domain training data. When the model parameters meet the preset parameter optimization conditions, the trained cardinality estimation model is obtained; Upon receiving the current query statement, the current query statement is input into the trained cardinality estimation model to obtain the cardinality estimation result.
[0007] In one embodiment, the source domain training data includes several training query statements. The step of inputting the source domain training data into the feature extractor and obtaining the cardinality prediction value through the cardinality estimator includes: The feature extractor extracts features from the training query statements in the source domain training data to obtain a query feature vector, which is the source domain query feature vector. The cardinality predictor is obtained by performing numerical mapping based on the query feature vector using the cardinality estimator.
[0008] In one embodiment, the step of updating the model parameters of the cardinality estimation model based on the predicted cardinality value and the source domain training data includes: In the source domain training data, obtain the cardinality true value corresponding to the training query statement; The estimated base loss value is determined according to the predicted base value and the actual base value using a preset prediction loss function. Based on the strategy of minimizing the cardinality estimation loss value, the parameters of the feature extractor and the cardinality estimator in the cardinality estimation model are updated according to the cardinality estimation loss value.
[0009] In one embodiment, both the source domain training data and the target domain training data include several training query statements. The step of inputting the source domain training data and the target domain training data into the feature extractor after the pre-training stage, and obtaining the domain probability distribution value through the domain classifier, includes: The feature extractor extracts features from the training query statements in the source domain training data or the target domain training data to obtain query feature vectors, which are source domain query feature vectors or target domain query feature vectors. The domain classifier performs probability mapping based on the query feature vector to obtain the domain probability distribution value.
[0010] In one embodiment, the step of updating the model parameters of the cardinality estimation model after the pre-training phase based on the domain probability distribution value, the source domain training data, and the target domain training data includes: The real domain attribute corresponding to the training query statement is determined based on the source domain training data and the target domain training data. The domain classification loss value is determined according to the domain probability distribution value and the real domain attributes using a preset adversarial loss function; Based on the strategy of minimizing the domain classification loss value and the gradient reversal strategy, the parameters of the feature extractor and the domain classifier in the cardinality estimation model after the pre-training phase are updated according to the domain classification loss value.
[0011] In one embodiment, the step of updating the parameters of the feature extractor and the domain classifier in the cardinality estimation model after the pre-training phase based on the strategy of minimizing the domain classification loss value and the gradient reversal strategy includes: Freeze the cardinality estimator in the cardinality estimation model after the pre-training phase ends; Based on the strategy of minimizing the domain classification loss value, the parameters of the domain classifier in the cardinality estimation model after the pre-training phase are updated according to the domain classification loss value; Based on the gradient reversal strategy, the feature extractor in the cardinality estimation model after the pre-training phase is updated according to the domain classification loss value.
[0012] In one embodiment, after the step of updating the parameters of the feature extractor and the domain classifier in the cardinality estimation model after the pre-training phase based on the domain classification loss value, the method further includes: Determine the source domain feature vector and the target domain feature vector from all the query feature vectors respectively; The source domain feature vector and the target domain feature vector are used to determine the source domain-target domain distribution difference loss value according to a preset distribution distance metric loss function; Based on the strategy of minimizing the source-target domain distribution difference loss value, the feature extractor in the cardinality estimation model after the adversarial training phase is updated according to the source-target domain distribution difference loss value.
[0013] In one embodiment, the step of extracting features from the training query statement using the feature extractor includes: The training query statement is decomposed using the feature extractor to obtain a table set, a join set, and a predicate set. The training query statement and database pattern are graph encoded based on a graph neural network to obtain the graph encoding result. Based on the graph encoding results, determine the table feature vector, connection condition feature vector, and predicate feature vector corresponding to the table set, the connection set, and the predicate set, respectively. The query feature vector is obtained by concatenating the table feature vector, the connection condition feature vector, and the predicate feature vector. The query feature vector is either the source domain query feature vector or the target domain query feature vector.
[0014] In one embodiment, the step of obtaining the trained cardinality estimation model when the model parameters satisfy preset parameter optimization conditions includes: When the updated model parameters are the same as those obtained from a preset number of iterations, a trained cardinality estimation model is obtained. Validate the trained cardinality estimation model on the test dataset, and when the validation is successful, execute the step of inputting the current query statement into the trained cardinality estimation model to obtain the cardinality estimation result when the current query statement is received.
[0015] Furthermore, to achieve the above objectives, this application also proposes a domain-adaptive database cardinality estimation device, the device comprising: a memory, a processor, and a domain-adaptive database cardinality estimation program stored in the memory and executable on the processor, the domain-adaptive database cardinality estimation program being configured to implement the steps of the domain-adaptive database cardinality estimation method as described above.
[0016] This application discloses a domain-adaptive database cardinality estimation method. The method is applied to a device deployed with a cardinality estimation model. The cardinality estimation model includes a feature extractor and a cardinality estimator and a domain classifier connected to it. The method includes: acquiring query training data, which includes source domain training data and target domain training data; in a pre-training phase, inputting the source domain training data into the feature extractor, obtaining cardinality prediction values through the cardinality estimator, and updating the model parameters of the cardinality estimation model based on the cardinality prediction values and the source domain training data; in an adversarial training phase, inputting the source domain training data and the target domain training data into the feature extractor after the pre-training phase, obtaining domain probability distribution values through the domain classifier, and updating the model parameters of the cardinality estimation model after the pre-training phase based on the domain probability distribution values, the source domain training data, and the target domain training data; obtaining a trained cardinality estimation model when the model parameters meet preset parameter optimization conditions; and upon receiving a current query, inputting the current query into the trained cardinality estimation model to obtain a cardinality estimation result.
[0017] This application explicitly divides the training process into two phases: pre-training and adversarial training. This allows the model to be optimized step-by-step and with specific focus. In the pre-training phase, the model learns accurate cardinality estimation capabilities from source domain data, laying a solid foundation for subsequent domain adaptation. In the adversarial training phase, adversarial training using a domain classifier effectively drives the feature extractor to learn and generate domain-invariant features that are highly discriminative to both the source and target domains. This ensures the model's cardinality estimation performance in the source domain while significantly improving the accuracy of cardinality estimation under workload-shifting scenarios. Attached Figure Description
[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0020] Figure 1 This is a flowchart illustrating the first embodiment of the adaptive database cardinality estimation method in the field of this application. Figure 2 This is a schematic diagram of the network framework of the cardinality estimation model in this application; Figure 3This is a flowchart illustrating a second embodiment of an adaptive database cardinality estimation method in the field of this application. Figure 4 This is a schematic diagram illustrating the entire training process of the cardinality estimation model in this application; Figure 5 This is a flowchart illustrating a third embodiment of an adaptive database cardinality estimation method in the field of this application. Figure 6 This is a schematic diagram of the feature extraction process of the feature extractor in this application; Figure 7 This is a schematic diagram of the structure of an adaptive database cardinality estimation device in the field of this application.
[0021] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation
[0022] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.
[0023] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.
[0024] This application provides a domain-adaptive database cardinality estimation method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating a first embodiment of an adaptive database cardinality estimation method in the field of this application. In this embodiment, the method includes steps S10 to S50: Step S10: Obtain query statement training data, which includes source domain training data and target domain training data.
[0025] It should be noted that the executing entity in this embodiment can be a computing electronic device with data processing, network communication, and program execution functions, such as a desktop computer, a mainframe computer, and a server cluster. This embodiment does not limit this. The following uses a domain-adaptive database cardinality estimation device (hereinafter referred to as "device") as an example to describe the various embodiments of this application.
[0026] The device can be equipped with a cardinality estimation model, which can be obtained by adding a domain adaptor to an existing cardinality estimation model enhanced with query features. Therefore, the cardinality estimation model can include a feature extractor, a cardinality estimator, and a domain classifier, with the feature extractor connected to both the cardinality estimator and the domain classifier.
[0027] This can be used as a reference. Figure 2 The network architecture of the cardinality estimation model in this application is described. Figure 2 This is a schematic diagram of the network framework of the cardinality estimation model in this application.
[0028] Depend on Figure 2 It can be seen that the original query feature-enhanced cardinality estimation model (GNMSCN) includes: a feature extractor (F) and a cardinality estimator (C); while this cardinality estimation model (DAN) 2 M 2 -GNMSCN) also includes Domain Adaptor (DA). Figure 2 In this context, the shared-weight feature extractor (F) can be regarded as the same feature extractor (F), and then the feature extractor (F) is connected to the cardinality estimator (C) and the domain classifier (D) respectively.
[0029] It should also be noted that the training data for the query statement can be the data input to the cardinality estimation model during model training and model validation, and can include both source domain training data and target domain training data. Specifically, the source domain training data can be labeled source domain workload data. , which includes A labeled query, i.e., a query-cardinality pair The target domain training data can be unlabeled target domain workload data. , which includes A single unlabeled query. This means that both the source domain training data and the target domain training data can contain several SQL query statements.
[0030] Step S20: In the pre-training stage, the source domain training data is input into the feature extractor, the cardinality prediction value is obtained through the cardinality estimator, and the model parameters of the cardinality estimation model are updated according to the cardinality prediction value and the source domain training data.
[0031] It should be noted that this pre-training phase can be the phase where the training model performs the cardinality estimation task. During the pre-training phase, the feature extractor and cardinality estimator can be trained using only the source domain training data.
[0032] The feature extractor encodes SQL queries, effectively capturing their semantic features and database schema structure to generate high-quality query feature vectors. The cardinality estimator predicts cardinality based on these feature vectors, optimizing both the feature extractor and cardinality estimator by minimizing the loss function.
[0033] To further illustrate the model training process in the pre-training phase, step S20 also includes: steps S201~S205: Step S201: The feature extractor extracts features from the training query statements in the source domain training data to obtain a query feature vector, which is the source domain query feature vector.
[0034] It should be understood that this feature extractor can be the query featureization module in the original query feature enhancement cardinality estimation model (GNMSCN). This query featureization module can accurately capture the relationship between the query and the pattern item through the graph neural network (GGNN), thereby enhancing the model's ability to understand the SQL query statement and outputting the query feature vector of the SQL query statement.
[0035] During the pre-training phase, since the training queries are all derived from the source domain training data, the resulting query feature vectors are source domain query feature vectors, which can be represented as follows: .
[0036] Step S202: The cardinality estimator performs numerical mapping based on the query feature vector to obtain the predicted cardinality value.
[0037] It should be understood that this cardinality estimator can be the cardinality estimation module in the original query feature-enhanced cardinality estimation model (GNMSCN), which is the core component of the cardinality estimation model. Its main function is to complete the cardinality prediction task based on the query feature vector output by the feature extractor.
[0038] It should be noted that the cardinality estimator can capture the mapping relationship between the query feature vector and the cardinality label, providing high-precision prediction output for the cardinality estimation task. It can be composed of two fully connected layers: the first layer further extracts higher-order semantic features through nonlinear transformation, and the second layer uses the sigmoid activation function to map the features to the interval, and finally obtains the cardinality prediction value through inverse normalization. .
[0039] Step S203: Obtain the cardinality true value corresponding to the training query statement from the source domain training data.
[0040] Understandably, since the training queries all come from the source domain training data, each training query can correspond to a true label, i.e., the true cardinality value. .
[0041] Step S204: Determine the estimated base loss value according to the predicted base value and the actual base value using a preset prediction loss function.
[0042] Step S205: Based on the strategy of minimizing the cardinality estimation loss value, update the parameters of the feature extractor and the cardinality estimator in the cardinality estimation model according to the cardinality estimation loss value.
[0043] It should be noted that the preset prediction loss function can be the q-error loss function. q-error is an accuracy evaluation metric that measures the accuracy of cardinality estimation. The q-error loss function... It can be represented as follows:
[0044] in, N s This indicates the number of training query samples in the source domain training data, compared with the true cardinality. and predicted value This ensures that the model has high cardinality estimation accuracy in the source domain.
[0045] It should be understood that the cardinality estimation loss value can be based on the q-error loss function described above. The calculated scalar value quantifies the degree of error in the model's predictions. The more accurate the model's predictions, the smaller the loss value; the greater the prediction error, the greater the loss value.
[0046] It should also be noted that the strategy of minimizing the cardinality estimation loss can be implemented using the gradient descent algorithm or by using the Adam optimizer.
[0047] In a practical implementation, the Adam optimizer can be used to calculate the gradient of the cardinality estimation loss with respect to all parameters of the model. Then, the parameters of the feature extractor and the cardinality estimator can be updated along the direction of decreasing gradient, thereby continuously reducing the cardinality estimation loss.
[0048] Step S30: In the adversarial training phase, the source domain training data and the target domain training data are respectively input into the feature extractor after the pre-training phase, the domain probability distribution value is obtained through the domain classifier, and the model parameters of the cardinality estimation model after the pre-training phase are updated according to the domain probability distribution value, the source domain training data and the target domain training data.
[0049] It should be noted that this adversarial training phase allows the feature extractor in the training model to learn domain-invariant feature representations between the source and target domains. During this phase, a mixture of source and target domain training data can be used to train both the feature extractor and the domain classifier.
[0050] Feature extractors can also encode SQL queries to effectively capture their semantic features and database schema structure, generating high-quality query feature vectors. Domain classifiers, on the other hand, can take the domain probability distribution of the query feature vectors as input, thus distinguishing whether the query feature vectors originate from the source or target domain.
[0051] In addition, the gradient inversion layer (GRL) can be used to invert the gradient direction of the neighborhood classifier during backpropagation, thereby prompting the feature extractor to generate neighborhood-invariant feature representations, making it difficult for the neighborhood classifier to distinguish query feature vectors belonging to the source and target domains.
[0052] In the specific implementation, for any SQL query statement, after the feature extractor extracts the query feature vector and inputs it into the domain classifier, the domain classifier can output the corresponding domain probability distribution value. Then, the specific domain to which the SQL query statement belongs is determined, that is, whether the SQL query statement comes from the source domain training data or the target domain training data. Then, a domain classification loss value is calculated. Finally, the parameters of the domain classifier are updated by using the strategy of minimizing the domain classification loss value, and GRL is used to enable the feature extractor to perform adversarial training to generate domain-invariant features that confuse the judgment of the domain classifier.
[0053] Step S40: When the model parameters meet the preset parameter optimization conditions, the trained cardinality estimation model is obtained.
[0054] It should be noted that the aforementioned preset parameter optimization submission can be based on the condition that the model's loss function no longer decreases or reaches the predetermined training epochs. For example, when the updated model parameters are the model parameters obtained after a preset number of iterations, a trained cardinality estimation model can be obtained.
[0055] Furthermore, to verify the performance of the trained model, the cardinality estimation model can be tested on a test dataset, and the q-error metric can be used to evaluate its cardinality estimation performance. The test dataset can consist of a combination of several SQL queries from the source domain training data and several SQL queries from the target domain training data annotated by the user.
[0056] In the specific implementation, when the model parameters are the model parameters obtained from a preset number of iterations, a trained cardinality estimation model is obtained; then, the trained cardinality estimation model is verified on a test dataset, and if the verification is successful, the feasibility of the trained cardinality estimation model is determined to proceed with subsequent steps.
[0057] Step S50: Upon receiving the current query statement, input the current query statement into the trained cardinality estimation model to obtain the cardinality estimation result.
[0058] It should be noted that, since the cardinality estimation model trained above is a cardinality estimation model with domain adaptability, when a new SQL query statement, i.e. the current query statement, is input, the trained cardinality estimation model can output a more accurate cardinality estimation result.
[0059] This embodiment trains the feature extractor and cardinality estimator using source domain training data, ensuring the model's basic cardinality estimation capability. It also combines target domain training data to achieve adversarial training, that is, by using a gradient inversion layer to optimize the feature extractor in adversarial training. This effectively drives the feature extractor to learn and generate domain-invariant features that are highly discriminative to both the source and target domains, thereby significantly improving the accuracy of cardinality estimation in workload drift scenarios.
[0060] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 3 , Figure 3 This is a flowchart illustrating a second embodiment of an adaptive database cardinality estimation method in the field of this application.
[0061] In this embodiment, to specifically illustrate the model training in the adversarial training phase, step S30 includes: steps S301~S305: Step S301: The feature extractor extracts features from the training query statements in the source domain training data or the target domain training data to obtain query feature vectors, wherein the query feature vectors are source domain query feature vectors or target domain query feature vectors.
[0062] It should be noted that this feature extractor can be the query featureization module in the original query feature enhancement cardinality estimation model (GNMSCN). This query featureization module can accurately capture the relationship between the query and the pattern item through GGNN, thereby enhancing the model's ability to understand SQL query statements and outputting the query feature vector of the SQL query statement.
[0063] During the pre-training phase, since the training query statements come from a mixture of source domain training data and target domain training data, the corresponding obtained query feature vector is the source domain query feature vector. or target domain query vector .
[0064] Step S302: Obtain the domain probability distribution value by performing probability mapping based on the query feature vector using the domain classifier.
[0065] It should be understood that the domain classifier can be a new module added to the original query feature enhanced cardinality estimation model (GNMSCN). The domain classifier can incorporate unsupervised domain adaptation techniques, and its main function is to reduce the difference between the source domain and target domain workloads in the generated query feature vectors.
[0066] It should be noted that the domain classifier can take the aforementioned generated query feature vector as input and consists of two fully connected layers. The intermediate layer uses the ReLU activation function to enhance non-linear expressiveness, and the output layer uses the LogSoftmax activation function to output the probability distribution of whether the query feature vector belongs to the source or target domain. or .
[0067] Step S303: Determine the real domain attribute corresponding to the training query statement based on the source domain training data and the target domain training data.
[0068] It should be noted that, based on the source of the training query statement, it can be determined that the training query statement is a query from the source domain. Or a query of the target domain .
[0069] Step S304: Determine the domain classification loss value according to the domain probability distribution value and the real domain attributes using a preset adversarial loss function.
[0070] Step S305: Based on the strategy of minimizing the domain classification loss value and the gradient reversal strategy, update the parameters of the feature extractor and the domain classifier in the cardinality estimation model after the pre-training phase according to the domain classification loss value.
[0071] It should be noted that this preset adversarial loss function can be the domain classifier loss function. The loss function of the classifier in this domain It can be represented as follows:
[0072] in, N s This indicates the number of samples of training query statements in the source domain training data. N t This indicates the number of training query statements in the target domain training data. and This represents a query that uses the source and target domains.
[0073] It should be understood that the domain classification loss value can be based on the domain classifier loss function described above. The calculated scalar value quantifies the degree of error in the domain classifier's classification. The more accurate the model's classification, the smaller the loss value; the larger the classification error, the larger the loss value.
[0074] It should also be noted that the strategy of minimizing the domain classification loss value can be a strategy that uses the gradient of the loss value to update the parameters of the domain classifier itself; while the gradient reversal strategy can be to reverse the gradient from the domain classifier when backpropagating to the feature extractor, thereby prompting the feature extractor to generate a domain-invariant query feature vector, making it difficult for the domain classifier to distinguish the query feature vectors from the source domain and the target domain.
[0075] In a specific implementation, the cardinality estimator in the cardinality estimation model after the pre-training phase can be frozen first; then, based on the strategy of minimizing the domain classification loss value, the parameters of the domain classifier in the cardinality estimation model after the pre-training phase are updated according to the domain classification loss value; and then, based on the gradient reversal strategy, the parameters of the feature extractor in the cardinality estimation model after the pre-training phase are updated according to the domain classification loss value.
[0076] Furthermore, to further enhance the feature distribution alignment capability of the feature extractor under workload drift scenarios, a distance metric can be introduced as an explicit constraint mechanism. Therefore, after step S30, the following steps are also included: S310~S330: Step S310: Determine the source domain feature vector and the target domain feature vector from all the query feature vectors respectively.
[0077] Step S320: Determine the source domain-target domain distribution difference loss value according to the source domain feature vector and the target domain feature vector using a preset distribution distance metric loss function.
[0078] Step S330: Based on the strategy of minimizing the source domain-target domain distribution difference loss value, update the model parameters of the feature extractor in the cardinality estimation model after the adversarial training phase ends according to the source domain-target domain distribution difference loss value.
[0079] This should be understood as the feature extractor outputting feature representations of the source and target domain queries, i.e., the source domain query feature vector. or target domain query vector Subsequently, MK-MMD can be used to measure its distribution distance in the high-dimensional regenerating kernel Hilbert space, and the differences in feature distribution at different granularities can be captured by combining multiple kernel functions.
[0080] Specifically, MK-MMD is a probability distribution distance metric based on the reproducing kernel Hilbert space. This method evaluates the difference in feature distributions between the source and target domains by calculating the maximum mean difference across multiple Gaussian kernel functions with different bandwidths. Its loss function is the preset distribution distance metric loss function. It can be expressed as follows:
[0081] in, This represents the feature map that maps data to the Hilbert space, while H k It is determined by the kernel function k Induced Hilbert space.
[0082] It should be understood that the source-target domain distribution difference loss value can be based on the aforementioned preset distribution distance metric loss function. The calculated scalar value quantifies the source domain query feature vector. or target domain query vector Difference degree.
[0083] The strategy of minimizing the domain classification loss value can be a strategy that normally uses the gradient of the loss value to update the parameters of the feature extractor itself. This is achieved by minimizing a preset distribution distance metric loss function. It can constrain the feature extractor to generate feature representations with domain invariance, thereby achieving alignment of feature distributions between the source and target domains.
[0084] It should be noted that by complementing the advantages of MK-MMD and the feature extractor, the original semantic feature extraction capability of the feature extractor is retained, while the domain adaptability is enhanced by explicit distribution alignment constraints. Through the learning of domain-invariant features and distribution alignment optimization, a more adaptive feature representation is provided for subsequent cardinality estimation.
[0085] Furthermore, when training a cardinality estimation model in practical applications, a total loss function for the cardinality estimation model can be determined, and a dynamic optimization strategy can be adopted in the specific training process to achieve a balance between domain adaptation and cardinality estimation accuracy.
[0086] The total loss function L can be expressed as:
[0087] In the formula, To pre-determine the prediction loss function, To pre-define the adversarial loss function, The loss function is a preset distribution distance metric. and It is a hyperparameter that controls the weights between loss functions.
[0088] Specifically, in the pre-training phase, firstly... and Set to 0, by minimizing Optimize F and C to ensure high cardinality estimation accuracy of the model in the source domain. During the adversarial adaptive training phase, progressively adjust the hyperparameters. Freeze C and optimize F and D, where D is achieved by minimizing the preset adversarial loss function. To distinguish between source and target domain features, F undergoes adversarial training using GRL to generate domain-invariant features that confuse D's judgments. During the feature distribution alignment stage, hyperparameters are fixed. Gradual adjustment of hyperparameters By minimizing the pre-defined distribution distance metric loss function Optimize F.
[0089] Based on the above dynamic optimization strategy, the model can learn domain-invariant features during the entire training process. These features are then refined to further reduce the distribution differences between the source and target domains, thus effectively ensuring the accuracy of cardinality estimation in workload drift scenarios.
[0090] In addition, you can refer to this place. Figure 4 The complete training process of the cardinality estimation model in this application is described. Figure 4 This is a schematic diagram illustrating the entire training process of the cardinality estimation model in this application.
[0091] Depend on Figure 4 As can be seen, firstly, F generates a query feature vector based on the training data (including labeled source domain workload data and unlabeled target domain workload data) of the input query statement; Next, in the pre-training phase, C optimizes the initial parameters by minimizing the cardinality estimation loss. Subsequently, in the adversarial training phase, D and F are updated by minimizing the domain classification loss, where D is minimized to distinguish the feature sources, and F is trained adversarially using GRL. At the same time, in the feature distribution alignment phase, the source domain-target domain distribution difference loss is minimized to further optimize F, thereby achieving cross-domain alignment of the query feature vector.
[0092] After calculating the gradient error based on the loss value and updating the model parameters (weights and biases) according to the gradient error, the trained cardinality estimation model is obtained when the training epochs meet the maximum number of iterations.
[0093] Finally, the performance of the trained cardinality estimation model is evaluated on the test set (from labeled source domain workload data and unlabeled target domain workload data) to obtain a cardinality estimation model that can be applied to real-world scenarios.
[0094] In the training process described above, the cardinality estimation model achieves end-to-end adversarial training and feature distribution alignment: that is, by introducing the MK-MMD distance metric to quantify the difference in feature distribution between the source and target domain workloads and to promote convergence of the two domain distributions; by using a gradient inversion layer to optimize the query feature extractor in adversarial training, the source domain model is prompted to learn domain-invariant feature representations, which significantly improves the accuracy of cardinality estimation in workload drift scenarios.
[0095] Based on the first and second embodiments of this application, in the third embodiment of this application, the content that is the same as or similar to that in embodiments one and two above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 5 , Figure 5 This is a flowchart illustrating a third embodiment of an adaptive database cardinality estimation method in the field of this application.
[0096] In this embodiment, to specifically illustrate how the feature extractor performs feature extraction on the training query statement, the method further includes: steps S01~S04: Step S01: Decompose the training query statement using the feature extractor to obtain a table set, a join set, and a predicate set.
[0097] This can be used as a reference. Figure 6 The specific process of the feature extractor extracting the query feature vector is explained. Figure 6 This is a schematic diagram of the feature extraction process of the feature extractor in this application.
[0098] Figure 6 The training query data input to the feature extractor can come from labeled or unlabeled workload data, depending on the training phase. Furthermore, Figure 6 The processing of the query feature vector output by the feature extractor by the cardinality estimator and the domain classifier can be referred to the aforementioned embodiments, and will not be elaborated upon in this embodiment.
[0099] Depend on Figure 6 As can be seen, in the feature extractor, the input training query statement can first be decomposed into three types of elements: table name, join relationship and predicate, and then the table set, join set and predicate set are obtained accordingly.
[0100] Step S02: Perform graph encoding on the training query statement and database pattern based on the graph neural network to obtain the graph encoding result.
[0101] Step S03: Determine the table feature vector, connection condition feature vector, and predicate feature vector corresponding to the table set, the connection set, and the predicate set, respectively, based on the graph encoding result.
[0102] It should be noted that a graph neural network (GGNN) can be used to model the entire database schema (tables, columns, primary and foreign key relationships) and the current query statement together into a graph structure. In this graph structure, nodes can be tables or columns, while edges can represent connections or predicate references.
[0103] Simultaneously, database statistics can be introduced, that is, the connection bitmap can be used as additional domain knowledge input to GGNN, so that primary and foreign key associations and data distribution information are also embedded in the node representation.
[0104] Based on the graph structure described above, each node (such as a table or column) can obtain a new vector representation that contains its contextual information, namely the table feature vector, the join condition feature vector, and the predicate feature vector. This replaces the One-Hot vector in the original query-driven model (MSCN model) and can better capture the semantic relationships between pattern items and between pattern items and queries.
[0105] Step S04: Concatenate the table feature vector, the connection condition feature vector, and the predicate feature vector to obtain the query feature vector, which is either the source domain query feature vector or the target domain query feature vector.
[0106] It should be understood that independent MLPs can be designed for the aforementioned set of tables, joins, and predicates. Each MLP performs a nonlinear transformation on each element in the corresponding set to generate an element-wise transformation representation; Next, the element transformation representations of each set can be compressed into a single vector through average pooling operations to obtain the final feature vector of the table set, the final feature vector of the join set, and the final feature vector of the predicate set. Finally, the final feature vectors of the table set, the join set, and the predicate set can be concatenated to generate the query feature vector. This query feature vector can then be used as input to the subsequent cardinality estimator and domain classifier.
[0107] In this embodiment, the feature extractor can enhance its understanding of database patterns and data distribution by fusing pattern terms and connection bitmaps, thereby efficiently extracting semantic features of training query statements and providing high-quality feature vector inputs for subsequent accurate cardinality estimation and domain classification.
[0108] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the domain-adaptive database cardinality estimation method in this application. Any simple transformations based on this technical concept are within the protection scope of this application.
[0109] This application also provides a domain-adaptive database cardinality estimation device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the domain-adaptive database cardinality estimation method in Embodiment 1 above.
[0110] The following is for reference. Figure 7 , Figure 7 This is a schematic diagram of the structure of a domain-adaptive database cardinality estimation device for the present application. The domain-adaptive database cardinality estimation device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 7 The domain-adaptive database cardinality estimation device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.
[0111] like Figure 7As shown, the domain-adaptive database cardinality estimation device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 1002 or a program loaded from storage device 1003 into random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the domain-adaptive database cardinality estimation device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the domain-adaptive database cardinality estimation device to communicate wirelessly or wiredly with other devices to exchange data. Although a domain-adaptive database cardinality estimation device with various systems is shown in the figure, it should be understood that it is not required to implement or possess all the systems shown. More or fewer systems can be implemented alternatively.
[0112] The domain-adaptive database cardinality estimation device provided in this application, employing the domain-adaptive database cardinality estimation method in the above embodiments, can solve the technical problem of domain-adaptive database cardinality estimation. Compared with the prior art, the beneficial effects of the domain-adaptive database cardinality estimation device provided in this application are the same as those of the domain-adaptive database cardinality estimation method provided in the above embodiments, and other technical features in this domain-adaptive database cardinality estimation device are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.
[0113] The sequence numbers of the above embodiments of the present invention are merely for description and do not represent the superiority or inferiority of the embodiments. They are only some embodiments of this application and are not intended to limit the scope of this application. All equivalent structural transformations made under the technical concept of this application and based on the content of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included within the protection scope of this application.
Claims
1. A domain-adaptive database cardinality estimation method, characterized in that, The method is applied to a device deployed with a cardinality estimation model, the cardinality estimation model including: a feature extractor, a cardinality estimator, and a domain classifier, wherein the feature extractor is connected to the cardinality estimator and the domain classifier, respectively, and the method includes: Obtain query training data, which includes source domain training data and target domain training data; During the pre-training phase, the source domain training data is input into the feature extractor, the cardinality prediction value is obtained through the cardinality estimator, and the model parameters of the cardinality estimation model are updated based on the cardinality prediction value and the source domain training data. During the adversarial training phase, the source domain training data and the target domain training data are respectively input into the feature extractor after the pre-training phase. The domain classifier obtains the domain probability distribution value, and the model parameters of the cardinality estimation model after the pre-training phase are updated based on the domain probability distribution value, the source domain training data, and the target domain training data. When the model parameters meet the preset parameter optimization conditions, the trained cardinality estimation model is obtained; Upon receiving the current query statement, the current query statement is input into the trained cardinality estimation model to obtain the cardinality estimation result.
2. The method as described in claim 1, characterized in that, The source domain training data includes several training query statements. The step of inputting the source domain training data into the feature extractor and obtaining the cardinality prediction value through the cardinality estimator includes: The feature extractor extracts features from the training query statements in the source domain training data to obtain a query feature vector, which is the source domain query feature vector. The cardinality predictor is obtained by performing numerical mapping based on the query feature vector using the cardinality estimator.
3. The method as described in claim 2, characterized in that, The step of updating the model parameters of the cardinality estimation model based on the predicted cardinality value and the source domain training data includes: In the source domain training data, obtain the cardinality true value corresponding to the training query statement; The estimated base loss value is determined according to the predicted base value and the actual base value using a preset prediction loss function. Based on the strategy of minimizing the cardinality estimation loss value, the parameters of the feature extractor and the cardinality estimator in the cardinality estimation model are updated according to the cardinality estimation loss value.
4. The method as described in claim 1, characterized in that, Both the source domain training data and the target domain training data include several training query statements. The step of inputting the source domain training data and the target domain training data into the feature extractor after the pre-training stage, and obtaining the domain probability distribution value through the domain classifier, includes: The feature extractor extracts features from the training query statements in the source domain training data or the target domain training data to obtain query feature vectors, which are source domain query feature vectors or target domain query feature vectors. The domain classifier performs probability mapping based on the query feature vector to obtain the domain probability distribution value.
5. The method as described in claim 4, characterized in that, The step of updating the model parameters of the cardinality estimation model after the pre-training phase based on the domain probability distribution value, the source domain training data, and the target domain training data includes: The real domain attribute corresponding to the training query statement is determined based on the source domain training data and the target domain training data. The domain classification loss value is determined according to the domain probability distribution value and the real domain attributes using a preset adversarial loss function; Based on the strategy of minimizing the domain classification loss value and the gradient reversal strategy, the parameters of the feature extractor and the domain classifier in the cardinality estimation model after the pre-training phase are updated according to the domain classification loss value.
6. The method as described in claim 5, characterized in that, The step of updating the parameters of the feature extractor and the domain classifier in the cardinality estimation model after the pre-training phase based on the strategy of minimizing the domain classification loss value and the gradient reversal strategy includes: Freeze the cardinality estimator in the cardinality estimation model after the pre-training phase ends; Based on the strategy of minimizing the domain classification loss value, the parameters of the domain classifier in the cardinality estimation model after the pre-training phase are updated according to the domain classification loss value; Based on the gradient reversal strategy, the feature extractor in the cardinality estimation model after the pre-training phase is updated according to the domain classification loss value.
7. The method as described in claim 5, characterized in that, After the step of updating the parameters of the feature extractor and the domain classifier in the cardinality estimation model after the pre-training phase based on the domain classification loss value, the method further includes: Determine the source domain feature vector and the target domain feature vector from all the query feature vectors respectively; The source domain feature vector and the target domain feature vector are used to determine the source domain-target domain distribution difference loss value according to a preset distribution distance metric loss function; Based on the strategy of minimizing the source-target domain distribution difference loss value, the feature extractor in the cardinality estimation model after the adversarial training phase is updated according to the source-target domain distribution difference loss value.
8. The method as described in claim 2 or 4, characterized in that, The steps of extracting features from the training query statement using the feature extractor include: The training query statement is decomposed using the feature extractor to obtain a table set, a join set, and a predicate set. The training query statement and database pattern are graph encoded based on a graph neural network to obtain the graph encoding result. Based on the graph encoding results, determine the table feature vector, connection condition feature vector, and predicate feature vector corresponding to the table set, the connection set, and the predicate set, respectively. The query feature vector is obtained by concatenating the table feature vector, the connection condition feature vector, and the predicate feature vector. The query feature vector is either the source domain query feature vector or the target domain query feature vector.
9. The method as described in claim 1, characterized in that, The step of obtaining the trained cardinality estimation model when the model parameters meet the preset parameter optimization conditions includes: When the updated model parameters are the same as those obtained from a preset number of iterations, a trained cardinality estimation model is obtained. Validate the trained cardinality estimation model on the test dataset, and when the validation is successful, execute the step of inputting the current query statement into the trained cardinality estimation model to obtain the cardinality estimation result when the current query statement is received.
10. A domain-adaptive database cardinality estimation device, characterized in that, The device includes: a memory, a processor, and a domain-adaptive database cardinality estimation program stored in the memory and executable on the processor, the domain-adaptive database cardinality estimation program being configured to implement the steps of the domain-adaptive database cardinality estimation method as described in any one of claims 1 to 9.