A database table retrieval method and system

Abstract

The application discloses a database table retrieval method and system, and relates to the technical field of data analysis.The method comprises the following steps: obtaining data generation table description data of a target database; obtaining a query statement input by a user, and encoding to obtain a query vector; encoding to obtain table description vectors, and comparing the similarity of the table description vectors with the query vector one by one, screening and constructing a preliminary screening candidate set; inputting the table description data corresponding to the query vector and the target table description vector into a plurality of preset scoring models to sequentially perform multidimensional scoring, and constructing a multidimensional feature vector; inputting the multidimensional feature vector into a preset decision tree to obtain a table sorting score, determining the target table description data corresponding to the query statement, and retrieving and outputting the original data corresponding to the target table description data in the target database.The application solves the technical problem that the traditional inverted index in the prior art is affected by mode ambiguity and structural noise, semantic retrieval is prone to result generalization, and irrelevant tables are introduced.

Description

Technical Field

[0001] This invention relates to the field of data analysis technology, and in particular to a database table retrieval method and system. Background Technology

[0002] Currently, enterprise databases are rapidly expanding in size and complexity. These databases typically contain tens of thousands of tables, each with hundreds of fields, and are continuously updated as business processes evolve. In this environment, table retrieval, as a fundamental prerequisite for downstream applications such as NL2SQL and business intelligence, faces extremely high challenges in achieving efficient and accurate retrieval, especially in scenarios with numerous new tables and no query logs.

[0003] Traditional keyword-based and inverted index-based retrieval methods (such as Elasticsearch) rely on table names, column names, and annotation matching. While fast and highly interpretable, they perform poorly in enterprise scenarios. The effectiveness of keyword matching is weakened by schema ambiguity (simplified names, domain-specific names, non-standardized names) and structural noise (common column names, placeholder fields). This problem is even more pronounced when new tables lack logs or manual annotations. While semantic retrieval based on embeddings can overcome keyword limitations, it is prone to result generalization and the introduction of irrelevant tables when schema information is ambiguous, incomplete, or complex. Summary of the Invention

[0004] This invention provides a database table retrieval method and system to solve the technical problems of traditional inverted indexes being unsatisfactory in enterprise data tables due to the influence of schema ambiguity and structural noise, and semantic retrieval based on embedding being prone to generalization of results and introduction of irrelevant tables when schema information is ambiguous or incomplete.

[0005] To address the aforementioned technical problems, embodiments of the present invention provide a database table retrieval method, including: Obtain data from the target database, extract the data from the target database, and generate a table describing the data; Obtain the query statement input by the user, perform semantic encoding on the query statement, and obtain a query vector; The table description data is semantically encoded to obtain table description vectors. The similarity of each table description vector is compared with the query vector to filter out target table description vectors and construct an initial candidate set. The query vector and the table description data corresponding to the target table description vector are input into several preset scoring models to perform multi-dimensional scoring in sequence, and multi-dimensional feature vectors are constructed based on the scoring results; wherein, each multi-dimensional feature vector corresponds to a table description data. The multi-dimensional feature vector is input into a preset decision tree to obtain a table sorting score. Based on the table sorting score, the target table description data corresponding to the query statement is determined, and the original data corresponding to the target table description data in the target database is retrieved and output.

[0006] As a preferred embodiment, the step of acquiring data from the target database and extracting the data from the target database to generate tabular descriptive data specifically includes: Retrieve data from the target database; Extract data from the target database to obtain data tables corresponding to all data in the target database. Then, using a preset large language model, generate table summaries and column descriptions for all data tables in sequence. Use the table summary and column descriptions corresponding to each data table as table description data.

[0007] As a preferred embodiment, after providing the table summary and column descriptions corresponding to each data table as table description data, the method further includes: Based on a preset generation control algorithm, the table description data is scored for quality, and the table description data with a quality score greater than a preset value is output to obtain the final table description data.

[0008] As a preferred embodiment, the quality scoring of the table description data based on the preset generation control algorithm specifically includes: The column data of all tables in the target database are parsed sequentially to obtain column information; Extract all data tables from the target database to obtain table primary key information, table foreign key information, and table names; Based on the primary key information, foreign key information, table name, and column information of the table, construct the key columns corresponding to each data table; Based on the key columns and the table description data, the percentage score for each data table is calculated.

[0009] As a preferred approach, the table description data is semantically encoded to obtain table description vectors. Each table description vector is then compared with the query vector for similarity to select target table description vectors and construct an initial candidate set. Specifically, this includes: Based on a preset encoding model, the table description data is semantically encoded sequentially to obtain a table description vector. The table description vector and the query vector are sequentially subjected to cosine similarity calculation, and the table description vector is sorted in descending order according to the cosine similarity. Select the table description vectors with the preset optimal number after sorting in descending order, and use them as the target table description vectors. Then, construct the initial candidate set from the target table description vectors.

[0010] As a preferred embodiment, the step of inputting the table description data corresponding to the query vector and the target table description vector into several preset scoring models for multi-dimensional scoring in sequence, and constructing a multi-dimensional feature vector based on the scoring results, specifically includes: The table description data corresponding to the query vector and the target table description vector are input into the large model semantic model to obtain semantic relevance features; The table description data corresponding to the query vector and the target table description vector are input into the lexical matching model to obtain lexical matching features; The table description data corresponding to the query vector and the target table description vector are input into the table structuring model to obtain the table structuring features; Input the table description data corresponding to the query vector and the target table description vector into the word overlap rate model to obtain the word overlap rate feature; The cosine similarity is used as the initial semantic feature; Based on the initial semantic features, semantic relevance features, lexical matching features, table structured features, and word overlap rate features, a multi-dimensional feature vector is constructed.

[0011] As a preferred embodiment, the multi-dimensional feature vector is input into a preset decision tree to obtain a table ranking score, and the target table description data corresponding to the query statement is determined based on the table ranking score, specifically including: The multi-dimensional feature vectors are input into a preset decision tree so that the preset decision tree performs a weighted summation of the multi-dimensional feature vectors to obtain a table ranking score for each multi-dimensional feature vector. The target table description data corresponding to the optimal table sort score is determined and used as the target table description data for the corresponding query statement.

[0012] As a preferred embodiment, the method for constructing the preset decision tree includes: Construct a classification and regression tree, and set the learning rate for the classification and regression tree; The classification regression tree is iteratively trained so that, during the iterative training process, the current classification regression tree is trained according to a preset training set, and the ranking error of the trained current classification regression tree is calculated using a validation set. When the ranking error is less than a preset error threshold, a new classification regression tree is constructed and its learning rate is set before training. The number of constructed classification regression trees is output when the ranking error is greater than the preset error threshold or the ranking error remains unchanged for a preset number of iterations. Based on the number of classification and regression trees constructed, and all classification and regression trees, a preset decision tree is constructed.

[0013] As a preferred embodiment, training the current classification and regression tree based on a preset training set specifically includes: Obtain natural language query data, as well as manually labeled correct tables and randomly sampled incorrect tables corresponding to the natural language query data. Based on the natural language query data, manually labeled correct tables and randomly sampled incorrect tables, construct triplet data as a preset training set. Set the loss function and train the current classification and regression tree using a preset training set.

[0014] Accordingly, the present invention also provides a database table retrieval system, comprising: The table description module is used to acquire data from the target database, extract the data from the target database, and generate table description data. The query vector module is used to obtain the query statement input by the user, perform semantic encoding on the query statement, and obtain a query vector. The initial candidate screening module is used to perform semantic encoding on the table description data to obtain table description vectors, and compare the similarity of each table description vector with the query vector to filter out target table description vectors and construct an initial candidate set. The feature vector module is used to input the table description data corresponding to the query vector and the target table description vector into several preset scoring models to perform multi-dimensional scoring in sequence, and to construct multi-dimensional feature vectors based on the scoring results; wherein, each multi-dimensional feature vector corresponds to a table description data; The retrieval output module is used to input the multi-dimensional feature vector into a preset decision tree to obtain a table sorting score, and based on the table sorting score, determine the target table description data corresponding to the query statement, and retrieve and output the original data of the target table description data corresponding to the target database.

[0015] Compared with the prior art, the embodiments of the present invention have the following beneficial effects: The technical solution of this invention generates table description data by acquiring target data, which can effectively overcome the limitations of traditional keyword retrieval, especially addressing the impact of pattern ambiguity and structural noise common in enterprise databases. It constructs query vectors based on user-input queries, compares their similarity with table description vectors, and filters out target table description vectors to build an initial candidate set. This reduces the computational burden of large models and improves the matching accuracy of subsequent deep analysis of table patterns, business logic, and query intent. Furthermore, by inputting query vectors and table description data into several preset scoring models for multi-dimensional scoring, and constructing multi-dimensional feature vectors based on the scoring results, a table ranking score is obtained using a preset decision tree. This avoids the semantic over-broadness problem that may result from single-vector retrieval. Simultaneously, the multi-layered retrieval architecture optimizes retrieval matching efficiency, avoids semantic retrieval result generalization, and improves retrieval accuracy and coverage. Attached Figure Description

[0016] Figure 1 : A flowchart illustrating the steps of a database table retrieval method provided in an embodiment of the present invention; Figure 2 : This is a schematic diagram of the structure of a database table retrieval system provided in an embodiment of the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] Example 1 Please refer to Figure 1 The present invention provides a database table retrieval method, comprising the following steps S101-S105: S101: Obtain data from the target database, extract the data from the target database, and generate table description data.

[0019] As a preferred embodiment, the step of acquiring data from the target database and extracting the data from the target database to generate tabular descriptive data specifically includes: Retrieve data from the target database; Extract data from the target database to obtain data tables corresponding to all data in the target database. Then, using a preset large language model, generate table summaries and column descriptions for all data tables in sequence. Use the table summary and column descriptions corresponding to each data table as table description data.

[0020] In this embodiment, the target database can be an enterprise-level database such as a power grid, and by calling the corresponding port / interface, such as a standardized interface like JDBC / ODBC, the data tables corresponding to all data in the target database can be collected and extracted. In particular, considering the characteristics of database sharding and heterogeneous data sources, an incremental synchronization mechanism can be used to extract data tables by only pulling the metadata of newly added / updated data tables.

[0021] In this embodiment, a large language model is used to extract data information from the target database. Specifically, the large language model is used to generate table summaries and column descriptions using the table schema. The specific expressions are as follows: in, The table description data output by the large language model For the table schema, Database table schema The content after feature extraction.

[0022] function Used for parsing table schemas Column information, function Used to extract the table The primary key and foreign key information, as well as the table name, etc. Table structure characteristics include: Core entity: {name_entity_recognition(table_name)}; Relationship topology: {generate_topology_describe(schema_tree)}; Key business column: {";".join([f"{c}({detect_business_semantic(c)})"for c inkey_columns])}.

[0023] Then, using a large language model, the extracted table structure features are output in JSON format: {{"summary":,"business_context":<business scenario>}}.

[0024] Understandably, table summaries and column descriptions generated through large language models can transform simplified, non-standardized table / column names in enterprise databases into natural language descriptions that conform to business semantics, effectively resolving the "pattern ambiguity" problem. This allows subsequent table retrieval tasks to match user queries based on semantics, rather than being limited to keywords. At the same time, the structured storage of description data can be directly connected to subsequent embedding retrieval processes, significantly improving the accuracy of embedding encoding and reducing the introduction of irrelevant tables.

[0025] As a preferred embodiment, after providing the table summary and column descriptions corresponding to each data table as table description data, the method further includes: Based on a preset generation control algorithm, the table description data is scored for quality, and the table description data with a quality score greater than a preset value is output to obtain the final table description data.

[0026] In this embodiment, a preset generation control algorithm can be used to calculate the quality of the table summary and column descriptions output by the large language model, thereby controlling the quality of the explanatory descriptions generated by the large language model. The quality score is actually the ratio between the table description data output by the large language model and the key columns involved in the database table schema. Preferably, the preset value is 80%, meaning that the column descriptions in the table description data generated by the large language model need to contain more than 80% of the key column names. For example, if a data table has 10 key columns, then at least 8 of them need to be mentioned in the column descriptions of the table description data before the data table can be output. Therefore, the table description data corresponding to the quality score exceeding the preset value is ultimately output, resulting in table description data containing more than 80% of the key columns.

[0027] As a preferred embodiment, the quality scoring of the table description data based on the preset generation control algorithm specifically includes: The column data of all tables in the target database are parsed sequentially to obtain column information; Extract all data tables from the target database to obtain table primary key information, table foreign key information, and table names; Based on the primary key information, foreign key information, table name, and column information of the table, construct the key columns corresponding to each data table; Based on the key columns and the table description data, the percentage score for each data table is calculated.

[0028] It should be noted that key columns can include primary keys, foreign keys, unique keys, index keys, composite keys, partition keys, etc. In this embodiment, key columns can be constructed using table primary key information, table foreign key information, table name, and column information. The column information is obtained by parsing the column data of all tables in the target database sequentially, while the table primary key information, table foreign key information, and table name can be extracted from all data tables.

[0029] In this embodiment, when calculating the percentage score, key columns can be matched with table description data. The coverage percentage of key columns in the table description data is calculated, and if this percentage is greater than 80%, the corresponding table description data is considered as compliant data and incorporated into subsequent semantic encoding. The interpretation generated by the large language model requires a generation control algorithm to control the quality of the table description data. The formula is as follows: in, The table description data output by the large language model For key columns involved in a database table schema, only when the key column names in the generated table description data account for more than 80% of the total key columns in the database, does it meet the requirements.

[0030] S102: Obtain the query statement input by the user, perform semantic encoding on the query statement, and obtain a query vector.

[0031] In this embodiment, the query statement is encoded in response to the query statement entered by the user. Preferably, the embedding model is used for encoding to obtain the encoding vector of the corresponding query statement, i.e., the query vector.

[0032] S103: Semantically encode the table description data to obtain table description vectors, and compare the similarity of each table description vector with the query vector to filter out target table description vectors and construct an initial candidate set.

[0033] As a preferred approach, the table description data is semantically encoded to obtain table description vectors. Each table description vector is then compared with the query vector for similarity to select target table description vectors and construct an initial candidate set. Specifically, this includes: Based on a preset encoding model, the table description data is semantically encoded sequentially to obtain a table description vector. The table description vector and the query vector are sequentially subjected to cosine similarity calculation, and the table description vector is sorted in descending order according to the cosine similarity. Select the table description vectors with the preset optimal number after sorting in descending order, and use them as the target table description vectors. Then, construct the initial candidate set from the target table description vectors.

[0034] In this embodiment, preferably, the preset encoding model is embedding model encoding. Through embedding model encoding, the table description data that meets the requirements (conditions) is encoded to obtain the corresponding table description vector. The obtained table description vector is then inserted into the vector database to obtain a table description vector library. .

[0035] In this embodiment, the query statement input by the user is semantically encoded using the embedding model encoding method to obtain a representation of the query vector. : in, For the query statement entered by the user, This is the output vector.

[0036] Calculate query vector With table description vector cosine similarity : By calculating the cosine similarity Sort the tables in descending order and obtain the top K (Top-K) table description vectors as the target table description vectors, thus constructing the initial candidate set. .

[0037] S104: Input the table description data corresponding to the query vector and the target table description vector into several preset scoring models to perform multi-dimensional scoring in sequence, and construct multi-dimensional feature vectors based on the scoring results; wherein, each multi-dimensional feature vector corresponds to a table description data.

[0038] As a preferred embodiment, the step of inputting the table description data corresponding to the query vector and the target table description vector into several preset scoring models for multi-dimensional scoring in sequence, and constructing a multi-dimensional feature vector based on the scoring results, specifically includes: The table description data corresponding to the query vector and the target table description vector are input into the large model semantic model to obtain semantic relevance features; The table description data corresponding to the query vector and the target table description vector are input into the lexical matching model to obtain lexical matching features; The table description data corresponding to the query vector and the target table description vector are input into the table structuring model to obtain the table structuring features; Input the table description data corresponding to the query vector and the target table description vector into the word overlap rate model to obtain the word overlap rate feature; The cosine similarity is used as the initial semantic feature; Based on the initial semantic features, semantic relevance features, lexical matching features, table structured features, and word overlap rate features, a multi-dimensional feature vector is constructed.

[0039] In this embodiment, for each target table description vector in the selected initial candidate set, and the table description data corresponding to the target table description vector, the relevant information of the data table corresponding to the user query vector and the table description data (including: table schema, generated table summary, column description, etc.) is used as input to calculate and construct a comprehensive multi-dimensional feature vector in parallel. Among them, multi-dimensional feature vectors This includes initial semantic features, semantic relevance features, lexical matching features, table structure features, and word overlap rate features. Among them, the initial semantic features are essentially constructed from the cosine similarity calculated in step S103 above, that is, the cosine similarity between the table description data corresponding to each query vector and the target table description vector.

[0040] In this embodiment, semantic relevance features are generated by inputting prompts containing user queries, table schema information, and table summaries into a large language model, extracting their deep semantic understanding and logical reasoning capabilities, including a semantic relevance score. ): The quantitative score directly output by the model on the degree of matching between the query and the table, which reflects the degree of fit at the business logic level.

[0041] For example, the prompt section may include: Analyzing the User Query, Checking the Table Schema, Explaining the Reference Table, Comprehensive Reasoning, and a Final Score. Analyzing the User Query involves carefully examining the user's question to understand their core intent, the entities involved, and the information they are seeking. Checking the Table Schema involves reviewing the provided table name, column names, data types, and constraints (such as primary keys and foreign keys). Explaining the Reference Table involves reading the AI-generated summary and business scenario description of the table. This provides crucial background information for understanding the table's business purpose and the actual meaning of its fields. Comprehensive Reasoning involves explaining, in the "Thinking Process" section, how the table's schema and business context relate to the user query through step-by-step analysis. It assesses whether the table's core entities and business scenario are consistent with the query's domain. For example, a query about "sales performance" should highly match a "sales order table." The Final Score is an integer score between 1 and 5 based on the reasoning process. The scoring criteria range from 1 to 5 points. 5 points (Excellent Match): The table is highly relevant to the query; the table's business scenario perfectly matches the query intent, and it appears to contain all the key fields needed to directly answer the query. 4 points (Good Match): The table is relevant to the query and answers the core part of the query, but may lack some secondary information or require some inference. 3 points (Possible Match): The table's subject is related to the query, but it's uncertain whether it contains the specific required information. It's a reasonable candidate table, but the match is not high. 2 points (Almost No Match): The table and query are only weakly related through some scattered keywords, but based on its table structure and business scenario, it clearly cannot answer the user's actual question. 1 point (Completely No Match): The table is completely irrelevant to the user's query.

[0042] In this embodiment, lexical matching features can compensate for the problem that pure semantic matching may ignore keywords such as specific entity names and codes. By calculating the lexical matching degree between the query and table metadata (a text set D_i such as table name, column name, and comments), matching on keywords and extended terms can be achieved. It is understandable that semantic retrieval based on embedding vectors often produces overly broad results, or even introduces irrelevant tables. SPLADE generates sparse vectors, which can not only perform keyword matching, but also implicitly expand queries using the contextual capabilities of BERT.

[0043] This embodiment uses the BERT-based sparse retrieval model (SPLADE) to calculate the lexical matching feature, i.e., the lexical matching score. : in, It is a user query vector The query sparse vector is obtained after encoding by the SPLADE model. It is table metadata The resulting sparse vector of table cells after encoding by the SPLADE model. This represents a user query. and the Metadata of each table The higher the score of the relevance between them at the lexical level, the closer their match is in terms of keywords and extended terms. , which means to iterate through every word $\omega$ in the vocabulary (e.g., BERT's vocabulary, which contains tens of thousands of words). , indicates calculating the product of weights, only when a word Also appearing in the query sparse vectors and tables The product is positive only when the weights are all in a sparse vector (i.e., none of the weights are 0). Finally, the weights of all words are summed to obtain the relevance score.

[0044] In this embodiment, the table's structured features reflect the inherent attributes and complexity of the table itself, and are unaffected by the query content, providing stable structured information for the ranking model: the number of columns and the density of primary and foreign keys; wherein, the number of columns ( ): Total number of columns in the candidate table. Primary and foreign key density ( The ratio of primary key (PK) and foreign key (FK) counts to the total number of columns in a table reflects the table's relevance and core nature. The formula for calculating primary and foreign key density is: in, Indicates the density of primary and foreign bonds. This means that the primary key and foreign key are combined to remove duplicates, resulting in a set. Indicates the first All columns in the table.

[0045] Preferably, the table structure features include the number of columns ( ) and main and foreign bond density ( ).

[0046] In this embodiment, the lexical overlap rate feature can be calculated using Jaccard similarity to determine the lexical overlap between the segmented content of a user query and the set of all column names in the table. Let... Let Q be the unique set of tokens obtained after word segmentation for a user query. For table The unique set of tokens obtained after word segmentation of all column names in the query is defined by the Jaccard similarity formula as the size of the intersection of two sets divided by the size of the union. Therefore, the token overlap rate feature, i.e., the overlap rate between the query word and column name, is... The calculation formula is as follows: This indicates the number of tokens shared by the user's query and the nouns in the table columns. This indicates the total number of tokens after deduplication of the user's query and the nouns in the table list.

[0047] In this embodiment, a multi-dimensional feature vector is constructed. : S105: Input the multi-dimensional feature vector into a preset decision tree to obtain a table sorting score, and determine the target table description data corresponding to the query statement based on the table sorting score, and retrieve and output the original data of the target table description data corresponding to the target database.

[0048] As a preferred embodiment, the multi-dimensional feature vector is input into a preset decision tree to obtain a table ranking score, and the target table description data corresponding to the query statement is determined based on the table ranking score, specifically including: The multi-dimensional feature vectors are input into a preset decision tree so that the preset decision tree performs a weighted summation of the multi-dimensional feature vectors to obtain a table ranking score for each multi-dimensional feature vector. The target table description data corresponding to the optimal table sort score is determined and used as the target table description data for the corresponding query statement.

[0049] In this embodiment, the ranking is mainly achieved through a pre-defined decision tree, employing an ensemble learning architecture, specifically composed of K classification and regression trees (CART). This pre-defined decision tree is an additive model, which, for a given feature vector of a candidate table... The final predicted table sorting score It is the weighted sum of all decision tree outputs: Indicates the first Decision trees for input vectors The predicted value (i.e. the weight of the leaf node). It is the first The learning rate of the trees, also known as the hyperparameter, can preferably be set to 0.1.

[0050] In this embodiment, after obtaining the table sorting score corresponding to each multi-dimensional feature vector, the target table description data corresponding to the optimal table sorting score can be selected as the target table description data corresponding to the query statement.

[0051] As a preferred embodiment, the method for constructing the preset decision tree includes: Construct a classification and regression tree, and set the learning rate for the classification and regression tree; The classification regression tree is iteratively trained so that, during the iterative training process, the current classification regression tree is trained according to a preset training set, and the ranking error of the trained current classification regression tree is calculated using a validation set. When the ranking error is less than a preset error threshold, a new classification regression tree is constructed and its learning rate is set before training. The number of constructed classification regression trees is output when the ranking error is greater than the preset error threshold or the ranking error remains unchanged for a preset number of iterations. Based on the number of classification and regression trees constructed, and all classification and regression trees, a preset decision tree is constructed.

[0052] In this embodiment, It is the first The learning rate of each tree, and The value can be determined using the early stopping method: First, divide the training data into a training set and a validation set. After training each tree, immediately run it on the validation set to calculate the sorting error. If the metrics on the validation set do not improve or even worsen in 30 consecutive rounds, stop training and obtain the value of k.

[0053] As a preferred embodiment, training the current classification and regression tree based on a preset training set specifically includes: Obtain natural language query data, as well as manually labeled correct tables and randomly sampled incorrect tables corresponding to the natural language query data. Based on the natural language query data, manually labeled correct tables and randomly sampled incorrect tables, construct triplet data as a preset training set. Set the loss function and train the current classification and regression tree using a preset training set.

[0054] In this embodiment, the pre-trained decision tree (GBDT model) is obtained on large-scale publicly available Text-to-SQL datasets (such as Spider-Spider2 and WikiSQL). First, a triplet dataset (Q, T^+, T^-) is constructed, where Q is the natural language query, T^+ is the manually labeled correct table (positive sample), and T^- is the randomly sampled incorrect table (negative sample) through semantic recall. For each sample, a feature vector is calculated using the aforementioned feature extraction method, resulting in positive sample features X_(T^+) and negative sample features X_(T^-). Then, a loss function is set; instead of directly predicting the absolute relevance score of the table, the model learns to make the score of positive samples higher than that of negative samples. as follows: in, and These are the model's scores for positive and negative samples, respectively. The loss function is minimized... L The model can learn different features (such as...) and The optimal weight combination for ranking importance is determined. The preset decision tree uses gradient descent for iteration. In the t-th iteration, the negative gradient of the loss function with respect to the current predicted value is calculated, used as the residual, and a new decision tree is trained to fit this residual, thereby gradually improving the accuracy of ranking.

[0055] In this embodiment, the semantic table summaries (such as business scenario descriptions and core entity recognition) generated by the large model effectively compensate for the limitations of traditional keyword retrieval, especially addressing common patterns of ambiguity (such as abbreviated column names) and structural noise (such as generic column names like "id" and "value") in enterprise databases. A two-stage approach—cosine similarity calculation and comprehensive multi-dimensional feature vector analysis—balances efficiency and accuracy. The semantic cosine similarity calculation utilizes a vector database to quickly filter Top-K candidate tables, reducing the computational burden on the large model; the comprehensive multi-dimensional feature vector analysis deeply analyzes the matching of table patterns, business logic, and query intent through the large model, avoiding the semantic over-broadness problem that may result from single-vector retrieval.

[0056] Furthermore, a generation control algorithm (such as ValidityScore) is introduced to ensure that the table summaries generated for large models cover more than 80% of the key column names, avoiding the retrieval of irrelevant or fictitious content. The generated business semantics are presented in intuitive natural language to help users understand the database structure.

[0057] Implementing the above embodiments has the following effects: The technical solution of this invention generates table description data by acquiring target data, which can effectively overcome the limitations of traditional keyword retrieval, especially addressing the impact of pattern ambiguity and structural noise common in enterprise databases. It constructs query vectors based on user-input queries, compares their similarity with table description vectors, and filters out target table description vectors to build an initial candidate set. This reduces the computational burden of large models and improves the matching accuracy of subsequent deep analysis of table patterns, business logic, and query intent. Furthermore, by inputting query vectors and table description data into several preset scoring models for multi-dimensional scoring, and constructing multi-dimensional feature vectors based on the scoring results, a table ranking score is obtained using a preset decision tree. This avoids the semantic over-broadness problem that may result from single-vector retrieval. Simultaneously, the multi-layered retrieval architecture optimizes retrieval matching efficiency, avoids semantic retrieval result generalization, and improves retrieval accuracy and coverage.

[0058] Example 2 Please see Figure 2 This invention provides a database table retrieval system, comprising: The table description module 201 is used to acquire data from the target database and extract the data from the target database to generate table description data; The query vector module 202 is used to obtain the query statement input by the user, perform semantic encoding on the query statement, and obtain a query vector; The initial candidate screening module 203 is used to perform semantic encoding on the table description data to obtain table description vectors, and compare the similarity of each table description vector with the query vector to filter out target table description vectors and construct an initial candidate set. The feature vector module 204 is used to input the table description data corresponding to the query vector and the target table description vector into several preset scoring models to perform multi-dimensional scoring in sequence, and to construct multi-dimensional feature vectors based on the scoring results; wherein, each multi-dimensional feature vector corresponds to a table description data. The retrieval output module 205 is used to input the multi-dimensional feature vector into a preset decision tree to obtain a table sorting score, and based on the table sorting score, determine the target table description data corresponding to the query statement, and retrieve and output the original data of the target table description data corresponding to the target database.

[0059] As a preferred embodiment, the step of acquiring data from the target database and extracting the data from the target database to generate tabular descriptive data specifically includes: Retrieve data from the target database; Extract data from the target database to obtain data tables corresponding to all data in the target database. Then, using a preset large language model, generate table summaries and column descriptions for all data tables in sequence. Use the table summary and column descriptions corresponding to each data table as table description data.

[0060] As a preferred embodiment, after providing the table summary and column descriptions corresponding to each data table as table description data, the method further includes: Based on a preset generation control algorithm, the table description data is scored for quality, and the table description data with a quality score greater than a preset value is output to obtain the final table description data.

[0061] As a preferred embodiment, the quality scoring of the table description data based on the preset generation control algorithm specifically includes: The column data of all tables in the target database are parsed sequentially to obtain column information; Extract all data tables from the target database to obtain table primary key information, table foreign key information, and table names; Based on the primary key information, foreign key information, table name, and column information of the table, construct the key columns corresponding to each data table; Based on the key columns and the table description data, the percentage score for each data table is calculated.

[0062] As a preferred approach, the table description data is semantically encoded to obtain table description vectors. Each table description vector is then compared with the query vector for similarity to select target table description vectors and construct an initial candidate set. Specifically, this includes: Based on a preset encoding model, the table description data is semantically encoded sequentially to obtain a table description vector. The table description vector and the query vector are sequentially subjected to cosine similarity calculation, and the table description vector is sorted in descending order according to the cosine similarity. Select the table description vectors with the preset optimal number after sorting in descending order, and use them as the target table description vectors. Then, construct the initial candidate set from the target table description vectors.

[0063] As a preferred embodiment, the step of inputting the table description data corresponding to the query vector and the target table description vector into several preset scoring models for multi-dimensional scoring in sequence, and constructing a multi-dimensional feature vector based on the scoring results, specifically includes: The table description data corresponding to the query vector and the target table description vector are input into the large model semantic model to obtain semantic relevance features; The table description data corresponding to the query vector and the target table description vector are input into the lexical matching model to obtain lexical matching features; The table description data corresponding to the query vector and the target table description vector are input into the table structuring model to obtain the table structuring features; Input the table description data corresponding to the query vector and the target table description vector into the word overlap rate model to obtain the word overlap rate feature; The cosine similarity is used as the initial semantic feature; Based on the initial semantic features, semantic relevance features, lexical matching features, table structured features, and word overlap rate features, a multi-dimensional feature vector is constructed.

[0064] As a preferred embodiment, the multi-dimensional feature vector is input into a preset decision tree to obtain a table ranking score, and the target table description data corresponding to the query statement is determined based on the table ranking score, specifically including: The multi-dimensional feature vectors are input into a preset decision tree so that the preset decision tree performs a weighted summation of the multi-dimensional feature vectors to obtain a table ranking score for each multi-dimensional feature vector. The target table description data corresponding to the optimal table sort score is determined and used as the target table description data for the corresponding query statement.

[0065] As a preferred embodiment, the method for constructing the preset decision tree includes: Construct a classification and regression tree, and set the learning rate for the classification and regression tree; The classification regression tree is iteratively trained so that, during the iterative training process, the current classification regression tree is trained according to a preset training set, and the ranking error of the trained current classification regression tree is calculated using a validation set. When the ranking error is less than a preset error threshold, a new classification regression tree is constructed and its learning rate is set before training. The number of constructed classification regression trees is output when the ranking error is greater than the preset error threshold or the ranking error remains unchanged for a preset number of iterations. Based on the number of classification and regression trees constructed, and all classification and regression trees, a preset decision tree is constructed.

[0066] As a preferred embodiment, training the current classification and regression tree based on a preset training set specifically includes: Obtain natural language query data, as well as manually labeled correct tables and randomly sampled incorrect tables corresponding to the natural language query data. Based on the natural language query data, manually labeled correct tables and randomly sampled incorrect tables, construct triplet data as a preset training set. Set the loss function and train the current classification and regression tree using a preset training set.

[0067] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the device described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0068] Implementing the above embodiments has the following effects: The technical solution of this invention generates table description data by acquiring target data, which can effectively overcome the limitations of traditional keyword retrieval, especially addressing the impact of pattern ambiguity and structural noise common in enterprise databases. It constructs query vectors based on user-input queries, compares their similarity with table description vectors, and filters out target table description vectors to build an initial candidate set. This reduces the computational burden of large models and improves the matching accuracy of subsequent deep analysis of table patterns, business logic, and query intent. Furthermore, by inputting query vectors and table description data into several preset scoring models for multi-dimensional scoring, and constructing multi-dimensional feature vectors based on the scoring results, a table ranking score is obtained using a preset decision tree. This avoids the semantic over-broadness problem that may result from single-vector retrieval. Simultaneously, the multi-layered retrieval architecture optimizes retrieval matching efficiency, avoids semantic retrieval result generalization, and improves retrieval accuracy and coverage.

[0069] Example 3 Accordingly, the present invention also provides a terminal device, comprising: a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the database table retrieval method as described in any of the above embodiments.

[0070] The terminal device in this embodiment includes a processor, a memory, and a computer program and computer instructions stored in the memory and executable on the processor. When the processor executes the computer program, it implements the various steps described in Embodiment 1 above, for example... Figure 1 The steps S101 to S105 are shown. Alternatively, when the processor executes the computer program, it implements the functions of each module / unit in the above-described device embodiment, such as the feature vector module 204.

[0071] For example, the computer program can be divided into one or more modules / units, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules / units can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in the terminal device. For example, the feature vector module 204 is used to input the table description data corresponding to the query vector and the target table description vector into several preset scoring models for multi-dimensional scoring, and to construct a multi-dimensional feature vector based on the scoring results.

[0072] The terminal device may be a desktop computer, laptop, handheld computer, or cloud server, etc. The terminal device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that the schematic diagram is merely an example of a terminal device and does not constitute a limitation on the terminal device. It may include more or fewer components than illustrated, or combine certain components, or different components. For example, the terminal device may also include input / output devices, network access devices, buses, etc.

[0073] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the terminal device, connecting all parts of the terminal device via various interfaces and lines.

[0074] The memory can be used to store the computer programs and / or modules. The processor implements various functions of the terminal device by running or executing the computer programs and / or modules stored in the memory and by calling data stored in the memory. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, applications required for at least one function, etc.; the data storage area may store data created based on the use of the mobile terminal, etc. In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, RAM, plug-in hard disk, smart media card (SMC), secure digital card (SD card), flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0075] Wherein, if the modules / units integrated in the terminal device are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by a processor, it can implement the steps of the various method embodiments described above. Wherein, the computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content contained in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0076] Example 4 Accordingly, the present invention also provides a computer-readable storage medium comprising a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform the database table retrieval method as described in any of the above embodiments.

[0077] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. In particular, it should be noted that any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention for those skilled in the art.

Claims

1. A database table retrieval method, characterized in that, include: Obtain data from the target database, extract the data from the target database, and generate a table describing the data; Obtain the query statement input by the user, perform semantic encoding on the query statement, and obtain a query vector; The table description data is semantically encoded to obtain table description vectors. The similarity of each table description vector is compared with the query vector to filter out target table description vectors and construct an initial candidate set. The query vector and the table description data corresponding to the target table description vector are input into several preset scoring models to perform multi-dimensional scoring in sequence, and multi-dimensional feature vectors are constructed based on the scoring results; wherein, each multi-dimensional feature vector corresponds to a table description data. The multi-dimensional feature vector is input into a preset decision tree to obtain a table sorting score. Based on the table sorting score, the target table description data corresponding to the query statement is determined, and the original data corresponding to the target table description data in the target database is retrieved and output.

2. The database table retrieval method as described in claim 1, characterized in that, The process of acquiring data from the target database and extracting the data from the target database to generate a table describing the data specifically includes: Retrieve data from the target database; Extract data from the target database to obtain data tables corresponding to all data in the target database. Then, using a preset large language model, generate table summaries and column descriptions for all data tables in sequence. Use the table summary and column descriptions corresponding to each data table as table description data.

3. The database table retrieval method as described in claim 2, characterized in that, After providing the table summary and column descriptions for each data table as table description data, the following is also included: Based on a preset generation control algorithm, the table description data is scored for quality, and the table description data with a quality score greater than a preset value is output to obtain the final table description data.

4. The database table retrieval method as described in claim 3, characterized in that, The preset generation control algorithm performs quality scoring on the table description data, specifically including: The column data of all tables in the target database are parsed sequentially to obtain column information; Extract all data tables from the target database to obtain table primary key information, table foreign key information, and table names; Based on the primary key information, foreign key information, table name, and column information of the table, construct the key columns corresponding to each data table; Based on the key columns and the table description data, the percentage score for each data table is calculated.

5. A database table retrieval method as described in claim 1, characterized in that, The table description data is semantically encoded to obtain table description vectors. Each table description vector is then compared with the query vector for similarity to select target table description vectors and construct an initial candidate set. Specifically, this includes: Based on a preset encoding model, the table description data is semantically encoded sequentially to obtain a table description vector. The table description vector and the query vector are sequentially subjected to cosine similarity calculation, and the table description vector is sorted in descending order according to the cosine similarity. Select the table description vectors with the preset optimal number after sorting in descending order, and use them as the target table description vectors. Then, construct the initial candidate set from the target table description vectors.

6. The database table retrieval method as described in claim 5, characterized in that, The step of inputting the table description data corresponding to the query vector and the target table description vector into several preset scoring models for multi-dimensional scoring, and constructing a multi-dimensional feature vector based on the scoring results, specifically includes: The table description data corresponding to the query vector and the target table description vector are input into the large model semantic model to obtain semantic relevance features; The table description data corresponding to the query vector and the target table description vector are input into the lexical matching model to obtain lexical matching features; The table description data corresponding to the query vector and the target table description vector are input into the table structuring model to obtain the table structuring features; Input the table description data corresponding to the query vector and the target table description vector into the word overlap rate model to obtain the word overlap rate feature; The cosine similarity is used as the initial semantic feature; Based on the initial semantic features, semantic relevance features, lexical matching features, table structured features, and word overlap rate features, a multi-dimensional feature vector is constructed.

7. A database table retrieval method as described in claim 6, characterized in that, The multi-dimensional feature vectors are input into a preset decision tree to obtain a table ranking score. Based on the table ranking score, the target table description data corresponding to the query statement is determined, specifically including: The multi-dimensional feature vectors are input into a preset decision tree so that the preset decision tree performs a weighted summation of the multi-dimensional feature vectors to obtain a table ranking score for each multi-dimensional feature vector. The target table description data corresponding to the optimal table sort score is determined and used as the target table description data for the corresponding query statement.

8. A database table retrieval method as described in any one of claims 1-7, characterized in that, The method for constructing the preset decision tree includes: Construct a classification and regression tree, and set the learning rate for the classification and regression tree; The classification regression tree is iteratively trained so that, during the iterative training process, the current classification regression tree is trained according to a preset training set, and the ranking error of the trained current classification regression tree is calculated using a validation set. When the ranking error is less than a preset error threshold, a new classification regression tree is constructed and its learning rate is set before training. The number of constructed classification regression trees is output when the ranking error is greater than the preset error threshold or the ranking error remains unchanged for a preset number of iterations. Based on the number of classification and regression trees constructed, and all classification and regression trees, a preset decision tree is constructed.

9. A database table retrieval method as described in claim 8, characterized in that, The step of training the current classification and regression tree based on a preset training set specifically includes: Obtain natural language query data, as well as manually labeled correct tables and randomly sampled incorrect tables corresponding to the natural language query data. Based on the natural language query data, manually labeled correct tables and randomly sampled incorrect tables, construct triplet data as a preset training set. Set the loss function and train the current classification and regression tree using a preset training set.

10. A database table retrieval system, characterized in that, include: The table description module is used to acquire data from the target database, extract the data from the target database, and generate table description data. The query vector module is used to obtain the query statement input by the user, perform semantic encoding on the query statement, and obtain a query vector. The initial candidate screening module is used to perform semantic encoding on the table description data to obtain table description vectors, and compare the similarity of each table description vector with the query vector to filter out target table description vectors and construct an initial candidate set. The feature vector module is used to input the table description data corresponding to the query vector and the target table description vector into several preset scoring models to perform multi-dimensional scoring in sequence, and to construct multi-dimensional feature vectors based on the scoring results; wherein, each multi-dimensional feature vector corresponds to a table description data; The retrieval output module is used to input the multi-dimensional feature vector into a preset decision tree to obtain a table sorting score, and based on the table sorting score, determine the target table description data corresponding to the query statement, and retrieve and output the original data of the target table description data corresponding to the target database.

Images