A single-tower model multi-table selection method based on triplet loss
By using a single-tower BERT model based on Triplet Loss, the problems of model input token overflow and poor generalization in multi-table operations are solved, achieving accurate table selection for structured data and improving the model's selection ability in multi-table scenarios.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- COMPUTER INNOVATION TECH RES INST OF ZHEJIANG UNIV
- Filing Date
- 2023-11-20
- Publication Date
- 2026-07-03
AI Technical Summary
Existing technologies face problems such as token overflow or poor generalization when processing structured data and performing multi-table operations, leading to inaccurate table selection.
A single-tower BERT model based on Triplet Loss is adopted. By constructing a triplet dataset and training with weighted loss, the model's feature representation of table creation statements and questions is enhanced. Pooling operation is used to obtain feature vectors, and similarity measurement is used to select relevant table data.
It enables accurate multi-table selection of structured tabular data in most scenarios, improving the model's generalization ability and robustness.
Smart Images
Figure CN117743393B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a multi-table selection method, specifically a multi-table selection method for a single-tower model based on Triplet Loss. Background Technology
[0002] In today's information age, the explosion of massive text data has led to the emergence of large-scale models like Chatgpt in our lives. Researchers are constantly fine-tuning these models to adapt them to tasks in different domains. Processing structured data has always been tedious and complex, and the emergence of large-scale models seems to offer a solution. Many researchers have attempted to generate instructions for processing structured data, such as SQL statements, by interacting with these models. However, these large-scale models of structured data all face the problem of multi-table operations. Sometimes a user's question may involve multiple tables, and how to get the large model to select the relevant tables from the numerous tables in the database becomes a challenge. Inputting the question along with the table creation statements for all tables in the database into the large model, allowing it to autonomously select tables and generate corresponding instructions, is one approach. However, if a database contains hundreds or thousands of tables, this can lead to token overflow or catastrophic forgetting of input. Therefore, how to filter relevant structured table data based on the user's question has become an urgent problem to solve.
[0003] Currently, approaches to handling structured data involving multiple tables generally fall into two categories: one is to input all relevant table creation information along with the question into the model, allowing the model to handle table selection and command generation in one step. The proposed Text-to-SQL approach inputs all table creation statement information into the model. The other approach trains a model with good generalization ability for sentence semantic features, performing relevance matching between table creation statements and questions. The proposed Table-TextEncoding approach trains a model to encode information from the table data, thereby matching relevant questions for table selection. However, neither of these methods can achieve accurate table selection in most scenarios. The first approach may cause token overflow issues in the model's input, while the second approach results in a model that generalizes well to table information but is not as robust to question feature representation, leading to poor matching in most cases. Summary of the Invention
[0004] To address the problems existing in the background technology, the present invention provides a method for selecting multiple tables in a single-tower model based on Triplet Loss.
[0005] The technical solution adopted in this invention is:
[0006] The method for multi-table selection based on the Triplet Loss single-tower model of the present invention includes:
[0007] 1) Construct a triplet dataset based on the database.
[0008] 2) Establish a single-tower BERT (Bidirectional Encoder Representation from Transformers) model, and use the triplet dataset to perform weighted loss joint training on the single-tower BERT model to obtain a single-tower BERT model with better table construction information expression and problem statement generalization features after training.
[0009] 3) Input the table construction statement information of several structured table data into the trained single-tower BERT model. The trained single-tower BERT model outputs the encoded information of each table construction statement information, and constructs a vector feature library for the structured table data; input the problem into the trained single-tower BERT model, and the trained single-tower BERT model outputs the feature vector of the problem.
[0010] 4) Perform similarity measurement on the vector feature library of the structured table data and the feature vector of the problem, and select several structured table data with higher similarity to achieve multi-table selection.
[0011] In the step 1), the table construction statement information DDL (Data Define Language), the first type of questions, and the second type of questions of the structured table data in a database are constructed into a structured table dataset. The first type of questions includes M related questions A related to the table construction statement information DDL, and the second type of questions includes N unrelated questions B unrelated to the table construction statement information DDL; the table construction statement information DDL in the structured table dataset is respectively bound to each related question A. For the bound table construction statement information DDL and each related question A, randomly select K unrelated questions B in the structured table dataset and respectively bind them to the bound table construction statement information DDL and the related question A, where K < N. Finally, obtain M×K samples containing the table construction statement information DDL, the related question A, and the unrelated question B, regarded as three statements with specific semantic information, and thus construct a triplet dataset.
[0012] In specific implementation, five unrelated questions B in the second type of questions can be selected, so that the sample data is expanded to five times the original, realizing sample enhancement.
[0013] In the step 2), the single-tower BERT model includes a Bert pre-training model and a pooling operation connected in sequence. Use the pre-trained Bert model as the feature encoder, and add a pooling operation to the output to obtain the average word embedding of the sentence vector output by the model as the expression of the feature vector.
[0014] In step 2), the triple dataset is used to jointly train the single-tower BERT model with weighted loss. Specifically, the triple dataset is input into the single-tower BERT model, and the weighted loss is calculated to obtain the loss value of the single-tower BERT model. Training stops when the loss value reaches the minimum, and the trained single-tower BERT model is obtained.
[0015] For each sample in the triplet dataset, the weighted calculation of the loss value of the single-tower BERT model is as follows:
[0016] Step 2.1) Input the sample into the single-tower BERT model. The single-tower BERT model outputs the table creation statement information (DDL), the feature vectors of related question A and unrelated question B in the sample.
[0017] Step 2.2) Concatenate the feature vectors of the table creation statement (DDL) with the feature vectors of the relevant problem A and the unrelated problem B respectively to perform classification calculations and obtain the binary classification loss.
[0018] Step 2.3) Perform distance measurement between the feature vectors of the table creation statement information DDL and the feature vectors of related problem A and unrelated problem B respectively, and calculate the Triplet loss to obtain the Triplet loss.
[0019] Step 2.4) The classification loss and Triplet loss are weighted and calculated to obtain the loss value of the single-tower BERT model.
[0020] The weighted calculation of classification loss and Triplet loss is as follows:
[0021] loss = γ·loss binary +(1-γ)·loss triplet
[0022]
[0023] loss triplet = d(T, A) - d(T, B) + margin
[0024] Where loss represents the loss value of the single-tower BERT model; γ represents the balanced binary classification loss. binary and Triplet loss triplet The balance factor; y and These represent the actual value and the predicted value, respectively; d() represents the distance function, which can be Euclidean distance, cosine distance, Manhattan distance, etc.; T, A, and B represent the table creation statement information (DDL), relevant question A, and irrelevant question B in the sample, respectively; margin represents the difference between the distance between the table creation statement information (DDL) and the relevant question A, and the distance between the table creation statement information (DDL) and the irrelevant question B. In practice, the distance between questions related to the table should be less than the distance between questions unrelated to the table, and the distance difference should be controlled at around 0.2.
[0025] In step 4), the similarity between the vector feature library of the structured table data and the feature vector of the question is measured using a distance function. Then, the structured table data is sorted according to the similarity values obtained after measuring each structured table data. Finally, several structured table data with similarity values higher than the similarity threshold are selected to complete the multi-table selection.
[0026] The beneficial effects of this invention are:
[0027] The method of this invention trains the table creation information and the question by binding the BERT model, and finally obtains a single-tower model that is robust to both table information generalization and question generalization. Based on the single-tower model, when performing joint operations on multiple tables in structured tabular data in the field of artificial intelligence, the relevant table data involved can be selected according to the question, and the table selection can be accurately achieved in most scenarios. Attached Figure Description
[0028] Figure 1 This is a flowchart of the table selection based on Triplet loss in this invention;
[0029] Figure 2 This is the core structural diagram of the single-tower network model of this invention;
[0030] Figure 3 This is a schematic diagram of the Triplet loss of the present invention. Detailed Implementation
[0031] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0032] like Figure 1 As shown, the method for selecting multiple tables in a single-tower model based on Triplet Loss according to the present invention includes:
[0033] 1) Construct a triplet dataset based on the database.
[0034] In step 1), the table creation statement information DDL of the structured table data in a database, the first type of questions, and the second type of questions are constructed into a structured table data set. The first type of questions includes M relevant questions A related to the table creation statement information DDL, and the second type of questions includes N irrelevant questions B unrelated to the table creation statement information DDL. The table creation statement information DDL in the structured table data set is respectively bound to each relevant question A. For the bound table creation statement information DDL and each relevant question A, K irrelevant questions B are randomly selected in the structured table data set and are respectively bound to the bound table creation statement information DDL and the relevant question A, where K < N. Finally, M × K samples containing the table creation statement information DDL, relevant questions A, and irrelevant questions B are obtained, regarded as three statements with specific semantic information, and thus constructed into a triple data set.
[0035] During specific implementation, five irrelevant questions B in the second type of questions can be selected, so that the sample data is expanded to five times the original, realizing sample enhancement.
[0036] The category related to the table creation statement information DDL is set to 1, and the unrelated category is set to 0. The specific data set form is shown in Table 1.
[0037] Table 1
[0038]
[0039]
[0040] 2) Establish a single - tower BERT model, and use the triple data set to jointly train the single - tower BERT model with weighted loss to obtain a single - tower BERT model with better expression of table creation information and generalization features of question statements after training.
[0041] In step 2), the single - tower BERT model includes a Bert pre - trained model and a pooling operation connected in sequence. The pre - trained Bert model is used as a feature encoder, and a pooling operation is added to the output to obtain the average word embedding of the sentence vector output by the model as the expression of the feature vector.
[0042] Such as Figure 2As shown, the BERT model is a type of Sentence-Transformer architecture. A pooling operation is added to the output of BERT / RoBERTa to obtain a fixed-size sentence embedding. Three pooling strategies were tried in the implementation: using CLS-tagged outputs (the original BERT classification method, taking the vector of the first token of the sentence), calculating the average of all output vectors (MEAN strategy), and calculating the maximum value of the word embeddings in the output vector (MAX strategy). Ultimately, the MEAN strategy was found to be the most effective. The training dataset used was the CSpider dataset. Table creation information and corresponding table-related questions were parsed using a syntax tree. Then, five questions were randomly selected from the remaining unrelated questions, thus expanding the dataset fivefold.
[0043] In step 2), the triplet dataset is used to jointly train the single-tower BERT model with weighted loss. Specifically, the triplet dataset is input into the single-tower BERT model, and the weighted loss is calculated to obtain the loss value of the single-tower BERT model. Training stops when the loss value reaches the minimum, and the trained single-tower BERT model is obtained.
[0044] For each sample in the triplet dataset, the weighted calculation of the loss value of the single-tower BERT model is as follows:
[0045] Step 2.1) Input the sample into the single-tower BERT model. The single-tower BERT model outputs the table creation statement information (DDL), the feature vectors of related question A and unrelated question B in the sample.
[0046] Step 2.2) Concatenate the feature vectors of the table creation statement (DDL) with the feature vectors of the relevant problem A and the unrelated problem B respectively to perform classification calculations and obtain the binary classification loss.
[0047] Step 2.3) Perform distance measurement between the feature vectors of the table creation statement information DDL and the feature vectors of related problem A and unrelated problem B respectively, and calculate the Triplet loss to obtain the Triplet loss.
[0048] Step 2.4) The classification loss and Triplet loss are weighted and calculated to obtain the loss value of the single-tower BERT model.
[0049] The weighted calculation of classification loss and Triplet loss is as follows:
[0050] loss = γ·loss binary +(1-γ)·loss triplet
[0051]
[0052] loss triplet = d(T, A) - d(T, B) + margin
[0053] Where loss represents the loss value of the single-tower BERT model; γ represents the balanced binary classification loss. binary and Triplet loss triplet The balance factor; y and These represent the actual value and the predicted value, respectively; d() represents the distance function, which can be Euclidean distance, cosine distance, Manhattan distance, etc.; T, A, and B represent the table creation statement information (DDL), relevant question A, and irrelevant question B in the sample, respectively; margin represents the difference between the distance between the table creation statement information (DDL) and the relevant question A, and the distance between the table creation statement information (DDL) and the irrelevant question B. In practice, the distance between questions related to the table should be less than the distance between questions unrelated to the table, and the distance difference should be controlled at around 0.2.
[0054] Softmax binary classification is the loss function that determines whether the two questions in a triplet are related to the table's information, defined as loSs. binary The triplet loss is the difference between the similarity measure between the table's construction information and question A, and the similarity measure between the table's construction information and question B. The loss is defined as... triplet .
[0055] like Figure 3 As shown, the Triplet loss principle is as follows: questions related to structured tabular data are closer to the table's construction information, while questions unrelated to the table are farther away. However, to ensure a certain distance difference between related and unrelated questions and avoid equal distances, a hyperparameter `margin` is introduced to guarantee that positive example questions (question A) and negative example questions (question B) will not be equal. Consider an extreme case where the similarity metrics of questions related to the table and those unrelated to the table are equal. In this case, the Triplet loss can still be optimized to 0, but the distance between these two similarity metrics is not significantly different, making it difficult to quantify the difference between the two types of questions during subsequent inference. Therefore, setting the `margin` to a constant greater than 0 avoids this situation.
[0056] 3) Input the table creation statement information of several structured table data into the trained single-tower BERT model. The trained single-tower BERT model outputs the encoding information of each table creation statement, and constructs a vector feature library of structured table data. Input the question into the trained single-tower BERT model. The trained single-tower BERT model outputs the feature vector of the question.
[0057] 4) Measure the similarity between the vector feature library of structured tabular data and the feature vector of the question, and select several structured tabular data with high similarity to achieve multi-table selection.
[0058] In step 4), the similarity between the vector feature library of the structured table data and the feature vector of the question is measured using a distance function. Then, the structured table data is sorted according to the similarity values obtained after measuring each structured table data. Finally, several structured table data with similarity values higher than the similarity threshold are selected to complete the multi-table selection.
[0059] The single-tower model for table selection refers to the following: Assuming there is a problem C and N tables of data in the data warehouse, the model is used to infer all the table creation information (DDL) of the N tables to obtain feature vectors of the N table creation information. Problem C is then inferred using the model to obtain a feature vector C. Then, the feature vector C is used to perform N similarity calculations with the feature vectors of the N tables. By controlling the similarity threshold, the top n tables are selected.
[0060] In steps 3) and 4), the trained model infers the table creation information of all tables to obtain the table creation information embedding vectors of all tables. When selecting a table, the question is inferred again to obtain the feature expression vector of the question. Then, the similarity between the question and the table creation information embedding vectors of all tables is calculated using a distance metric function. All tables with similarity values greater than the threshold are selected by controlling the similarity threshold (the threshold is set to 0.85). Thus, the multi-table selection process is completed.
Claims
1. A method for selecting multiple tables in a single-tower model based on Triplet Loss, characterized in that, Including: 1) Constructing a triple dataset based on a database; 2) Establishing a single-tower BERT model, and using the triple dataset for weighted loss joint training of the single-tower BERT model to obtain a trained single-tower BERT model; 3) Inputting the table creation statement information of several structured table data into the trained single-tower BERT model. The trained single-tower BERT model outputs the encoded information of each table creation statement information, and constructs a vector feature library for the structured table data; Inputting the question into the trained single-tower BERT model, and the trained single-tower BERT model outputs the feature vector of the question; 4) Measuring the similarity between the vector feature library of the structured table data and the feature vector of the question, and selecting several structured table data with higher similarity to achieve multi-table selection; In the step 1), the table creation statement information DDL of the structured table data in a database, the first type of questions and the second type of questions are constructed into a structured table dataset. The first type of questions includes M related questions A related to the table creation statement information DDL, and the second type of questions includes N unrelated questions B unrelated to the table creation statement information DDL; Binding the table creation statement information DDL in the structured table dataset with each related question A respectively. For the bound table creation statement information DDL and each related question A, randomly select K unrelated questions B in the structured table dataset and bind them with the bound table creation statement information DDL and related question A respectively, where K < N. Finally, obtain M×K samples containing the table creation statement information DDL, related question A and unrelated question B, and thus construct a triple dataset; In the step 2), the single-tower BERT model includes a Bert pre-training model and a pooling operation connected in sequence; In the step 2), using the triple dataset for weighted loss joint training of the single-tower BERT model, specifically inputting the triple dataset into the single-tower BERT model, calculating the weighted loss to obtain the loss value of the single-tower BERT model, and stopping training until the loss value reaches the minimum to obtain a trained single-tower BERT model; For each sample in the triple dataset, the weighted calculation of the loss value of the single-tower BERT model is specifically as follows: Step 2.1) Inputting the sample into the single-tower BERT model, and the single-tower BERT model outputs the feature vectors of the table creation statement information DDL, related question A and unrelated question B in the sample respectively; Step 2.2) Concatenating the feature vector of the table creation statement information DDL with the feature vectors of the related question A and the unrelated question B respectively for classification calculation to obtain a binary classification loss; Step 2.3) Measuring the distance between the feature vector of the table creation statement information DDL and the feature vectors of the related question A and the unrelated question B respectively for Triplet loss calculation to obtain a Triplet loss; Step 2.4) Calculating the weighted sum of the classification loss and the Triplet loss to obtain the loss value of the single-tower BERT model.
2. The method for selecting multiple tables in a single-tower model based on Triplet Loss according to claim 1, characterized in that: The specific weighted calculation of the classification loss and the Triplet loss is as follows: in, This represents the loss value of a single-tower BERT model; Represents balanced binary classification loss and Triplet loss The balance factor; and These represent the actual value and the predicted value, respectively. Represents the distance function; , and These represent the table creation statement information (DDL), relevant question A, and irrelevant question B in the sample, respectively. This represents the difference between the distance between the table creation statement information (DDL) and the related question A, and between the distance between the table creation statement information (DDL) and the unrelated question B.
3. The method for selecting multiple tables in a single-tower model based on Triplet Loss according to claim 1, characterized in that: In step 4), the similarity between the vector feature library of the structured table data and the feature vector of the question is measured using a distance function. Then, the structured table data is sorted according to the similarity values obtained after measuring each structured table data. Finally, several structured table data with similarity values higher than the similarity threshold are selected to complete the multi-table selection.