Table retrieval method and device for database cold start scene and electronic equipment

Abstract

The invention discloses a table retrieval method and device for a database cold boot scene and electronic equipment. The method comprises the following steps: S1, preprocessing structure information; s2, semantic enhancement and vector representation generation; s3, constructing a reverse index; s4, performing semantic recall; s5, carrying out keyword constraint and filtering; and S6, performing multi-score fusion sorting. The method is suitable for performing accurate, stable and interpretable retrieval on new or semantic sparse data tables in an enterprise database under the condition of lack of historical query logs, manual annotation or statistical information access. According to the method, data table filtering, semantic enhancement description data table generation and table-level semantic vector generation, reverse index construction, semantic similarity score calculation and semantic similarity score calculation are carried out on the whole. The processes of keyword matching score calculation, coverage rate score query, accurate matching score calculation, filtered candidate data table generation, fusion sorting score calculation and sorting are executed in sequence, and all the steps form a complete and closed-loop retrieval processing link.

Description

Technical Field

[0001] This invention relates to database table retrieval, and in particular to a table retrieval method, apparatus, and electronic device for database cold start scenarios. Background Technology

[0002] Existing enterprise database retrieval technologies mainly fall into two categories: keyword-based retrieval methods and semantic vector-based retrieval methods.

[0003] Keyword-based retrieval methods typically rely on inverted index structures to match and sort keywords in table names, field names, or field comments. This type of method is simple to implement, highly efficient, and provides relatively interpretable results, and has been applied in some database management and data asset management platforms. However, because table names in enterprise databases often contain abbreviations, non-standard naming conventions, or domain-specific terminology, keyword-based retrieval methods are prone to insufficient recall when the query semantics and table structure do not directly overlap at the literal level. This is especially true when newly created data tables have not yet accumulated usage records or manually labeled information, resulting in a significant drop in retrieval performance.

[0004] Semantic vector-based retrieval methods map query text to a unified vector space and utilize vector similarity to achieve semantic matching, thus mitigating the problem of word-level mismatch to some extent. However, these methods typically lack explicit control over keyword constraints, easily introducing semantic drift, where search results are semantically relevant but do not meet the user's true intent at the business constraint or keyword level. Furthermore, purely semantic retrieval methods are weak in interpretability, making it difficult to clearly explain the correspondence between search results and query conditions, which is detrimental to enterprise-level auditing, debugging, and manual verification.

[0005] In the actual operation of enterprise databases, a scenario known as "cold start" is common, where newly created data tables have not yet accumulated historical query logs, access records, or manually labeled information. Existing retrieval methods often struggle to simultaneously ensure semantic coverage, keyword constraint capabilities, and result stability in this scenario, making it difficult for newly created data tables to participate in query retrieval in a timely and accurate manner, thus affecting the availability and intelligence level of the database.

[0006] Therefore, how to perform efficient, accurate, and interpretable retrieval of newly created or semantically sparse data tables in enterprise databases without relying on historical query logs has become a pressing technical problem that current enterprise database retrieval technology needs to solve. Summary of the Invention

[0007] To address the aforementioned problems, this invention provides a table retrieval method, apparatus, and electronic device for database cold start scenarios.

[0008] This invention provides the following technical solution: a table retrieval method for a database cold start scenario, comprising the following steps: S1. Obtain the table structure information of several data tables, filter according to the table structure information, and obtain the filtered data tables. The table structure information includes at least the table name, field name, and field comments. S2. Generate field annotations for semantic enhancement descriptions based on the table structure information of the filtered data table, obtain the data table with semantic enhancement descriptions, and generate table-level semantic vectors based on the data table with semantic enhancement descriptions. S3. Construct an inverted index based on the data table with semantically enhanced description; S4. Obtain the query vector based on the input query, obtain the semantic similarity score based on the query vector and the table-level semantic vector, and filter the data tables for semantic enhancement description based on the semantic similarity score to obtain a set of candidate data tables. S5. Perform keyword matching calculation on the query in the candidate data table set according to the inverted index to obtain keyword matching score, query coverage score and exact match score. Filter the data tables that do not meet the mandatory keyword set coverage constraint or hit the negative keyword set to generate filtered candidate data tables. S6. Calculate the fusion ranking score for the filtered candidate data table based on the semantic similarity score, keyword matching score, query coverage score, and exact match score, and rank the filtered candidate data table according to the fusion ranking score.

[0009] Furthermore, in step S1, the filtering involves filtering structural items in the table structure information that do not have actual business semantics or have obvious template characteristics, while retaining table-level and field-level structural elements that have retrieval significance.

[0010] Furthermore, step S1 specifically includes: (1) Obtain the table structure information of several data tables in the database, wherein the table structure information includes at least the table name, field name, and field comments; (2) Perform keyword filtering and regular expression matching on the field names to remove fields whose names are string, number, value, col1, data1, temp, misc, unnamed or variations of the above field names, and obtain the filtered table structure information.

[0011] Furthermore, step S2 specifically includes: (1) Extract the table structure information of the filtered data table and its context information in the table creation statement; (2) Take the context information, field name, and field annotation as input, call the large language model to generate field annotations for semantically enhanced descriptions, replace the original field annotations with the field annotations for semantically enhanced descriptions, and obtain the data table for semantically enhanced descriptions; (3) The table structure information of the data table described by semantic enhancement is concatenated and a table-level semantic vector is generated by a vector encoding function.

[0012] Furthermore, step S3 specifically includes: (1) Construct a set of index terms from the table structure information of the data table described by semantic enhancement. Each index term is associated with its own data table identifier and field position identifier, which is used to locate the source of the match when a match is hit. The types of index terms include table name type, field name type, and field comment type. (2) Set field-level weights for different types of index terms, including table name weight, field name weight, and field comment weight; (3) Write the index terms, field weights and their positions in the corresponding data tables into the inverted index structure to build the inverted index.

[0013] Furthermore, step S4 specifically includes: (1) Vectorize the input query to obtain the query semantic vector; (2) Calculate the similarity between the query semantic vector and the table-level semantic vector of each data table to obtain the table-level semantic similarity score; (3) Based on the table-level semantic similarity score, a threshold filtering or Top-K strategy is used to obtain a set of candidate data tables.

[0014] Furthermore, step S5 specifically includes: (1) Perform query matching calculation on the candidate data table set in the inverted index to obtain the keyword matching score and obtain the set of hit keywords for each candidate data table, where the set of hit keywords represents the set of keywords hit in the table name, field name or comment of the candidate data table; (2) Perform keyword parsing on the query to obtain the set of required matching keywords, the set of optional extended keywords, and the set of negative keywords. The set of required matching keywords is used to express the constraints that cannot be missing in the user's intent, and the set of negative keywords is used to express the reverse semantics that need to be excluded. (3) Calculate the coverage of the hit set of each candidate data table to the set of keywords that must be matched, and obtain the query coverage score; Calculate an exact match score for each candidate data table based on whether it matches the set of negative keywords. Filter data tables that do not meet the requirement of matching the keyword set coverage constraint or hit the negative keyword set, and generate a candidate data table after keyword matching.

[0015] Furthermore, the table-level semantic similarity score is calculated using cosine similarity. Keyword matching scores are calculated using TF-IDF. The query coverage score is: (The number of keywords that must be matched in the keyword set and the hit keyword set) / (Denominator); The denominator is the larger of the number of keywords in the set of keywords that must be matched and 1. The exact match rating is: When a candidate data table matches the set of negative keywords, the exact match score for that candidate data table is 1; when a candidate data table does not match the set of negative keywords, the exact match score for that candidate data table is 0.

[0016] Furthermore, step S6 specifically includes: The candidate data table after keyword matching is calculated with a fusion ranking score based on semantic similarity score, keyword matching score, query coverage score, and exact match score. The fusion ranking score = α × semantic similarity score + β × keyword matching score + γ × exact match score + δ × query coverage score; Wherein, α, β, γ, and δ are non-negative weight parameters used to characterize the contribution of each score to the final ranking, and satisfy α+β+γ+δ=1; The candidate data table after keyword matching is sorted according to the fusion sorting score.

[0017] Furthermore, when the fusion ranking scores of candidate data after multiple keyword matching are the same, they are ranked in parallel in the order of exact match score first, query coverage score second, and keyword matching score third, to ensure the stability of the results.

[0018] A table retrieval device for a database cold start scenario includes: The first acquisition module is used to acquire the table structure information of several data tables, filter the data according to the table structure information, and obtain the filtered data tables. The table structure information includes at least the table name, field name, and field comments. The first generation module is used to generate field annotations for semantically enhanced descriptions based on the table structure information of the filtered data table, obtain the data table with semantically enhanced descriptions, and generate table-level semantic vectors based on the data table with semantically enhanced descriptions. The building module is used to construct inverted indexes based on data tables with semantically enhanced descriptions; The first calculation module is used to obtain a query vector based on the input query, obtain a semantic similarity score based on the query vector and the table-level semantic vector, and filter the data table of semantically enhanced description based on the semantic similarity score to obtain a set of candidate data tables. The second generation module is used to perform keyword matching calculation on the query in the candidate data table set according to the inverted index, obtain keyword matching score, query coverage score and exact match score, filter data tables that do not meet the mandatory keyword set coverage constraint or hit the negative keyword set, and generate filtered candidate data tables. The second calculation module is used to calculate the fusion ranking score of the filtered candidate data table based on the semantic similarity score, keyword matching score, query coverage score and exact matching score, and to rank the filtered candidate data table according to the fusion ranking score.

[0019] An electronic device, comprising: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors perform the methods described above.

[0020] A computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of the method described above.

[0021] The beneficial effects of this invention are as follows: This invention is applicable to accurate, stable, and interpretable retrieval of newly created or semantically sparse data tables in enterprise databases when historical query logs, manual annotation, or access statistics are lacking. The method involves the following steps: filtering data tables, generating semantically enhanced descriptions of the data tables and table-level semantic vectors, constructing inverted indexes, calculating semantic similarity scores, calculating keyword matching scores, query coverage scores, and exact match scores to generate filtered candidate data tables, and calculating and ranking the fusion and sorting scores. These steps form a complete, closed-loop retrieval processing chain. Attached Figure Description

[0022] Figure 1 This is a flowchart of the table retrieval method for a database cold start scenario according to the present invention; Figure 2 This is a block diagram of the table retrieval device for a database cold start scenario according to the present invention. Detailed Implementation

[0023] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structure, features and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided below.

[0024] The method of this invention includes: performing pattern cleaning and field extraction on the database table structure to construct a weighted inverted index; generating semantically enhanced descriptions of the tables and fields based on a large language model, and generating corresponding semantic vector representations; in the query stage, firstly performing coarse-grained recall based on semantic vectors to obtain a candidate table set, and then combining the inverted index for keyword constraint and filtering; finally, integrating semantic similarity, keyword matching degree, query coverage, and exact match score to comprehensively rank the candidate tables and output the retrieval results.

[0025] This invention enables effective retrieval of newly created database tables without relying on historical query logs. It balances semantic generalization and keyword constraint capabilities, improving the recall, accuracy, and interpretability of enterprise-level database table retrieval in cold start scenarios. It is applicable to application scenarios such as natural language query, intelligent query and data asset discovery.

[0026] The embodiments of the present invention will be further described below with reference to several examples.

[0027] Example 1

[0028] In this embodiment, a billing database exists within the enterprise, containing a table named `billing_records`. This table records customer billing information. However, due to historical reasons, some field names contain abbreviations, field comments are incomplete, and the table is a recently created table with no accumulated query logs. When users input natural language queries such as "table related to customer billing" or "amount information excluding customer IDs," traditional keyword-based or purely semantic methods struggle to reliably find the correct table. This invention addresses these issues through the following steps, the specific implementation of which will be described below.

[0029] like Figure 1 A table retrieval method for a database cold start scenario includes the following steps: S1. Obtain the table structure information of several data tables, filter according to the table structure information, and obtain the filtered data tables. The table structure information includes at least the table name, field name, and field comments. In step S1, the filtering involves filtering structural items in the table structure information that do not have actual business semantics or have obvious template characteristics, while retaining table-level and field-level structural elements that have retrieval significance.

[0030] Step S1 is as follows: (1) Obtain the table structure information of several data tables in the database, wherein the table structure information includes at least the table name, field name, and field comments; (2) Perform keyword filtering and regular expression matching on the field names to remove fields whose names are string, number, value, col1, data1, temp, misc, unnamed or variations of the above field names, and obtain the filtered table structure information.

[0031] In this embodiment, the table structure information of all data tables is first obtained from the enterprise database. This information can be obtained in various ways, including parsing the database table creation statements, calling the database's metadata interface, or reading from a structure export file. For each data table, at least the following table structure information is extracted: database name, table name, field name, field type, field comments, and the field's position in the table.

[0032] For example, for the data table billing_records, the parsed original structure information can be represented as: { "database": "billing_system", "table": "billing_records", "columns": ["id", "cust_id", "amt", "period_start", "period_end"], "comments": ["", "", "amount", "start time", "end time"] }; The above structural information serves as the raw input for subsequent field cleaning and semantic processing.

[0033] (2) Invalid field identification and filtering As enterprise databases evolve over time, they often generate a large number of automatically generated fields, template fields, or intermediate fields. These fields typically do not carry explicit business semantics, and if directly used in retrieval, they will significantly interfere with keyword matching and semantic reasoning. Therefore, after obtaining the original structural information, an invalid field filtering operation is performed on the field-level structure.

[0034] In this embodiment, the filtering strategy includes, but is not limited to: Perform keyword rule matching and regular expression detection on field names, and remove fields with names of string, number, value, col1, data1, temp, misc, unnamed, or variations of the above field names.

[0035] (3) Output of structural cleaning results After the above processing, the filtered data table is obtained, generating a structure-cleaned result where each field contains the original structure information. For example, the cleaned field structure is represented as follows: { "field_name": "amt", "type": "DECIMAL", "comment": "amount", "valid": true, "confidence": 0.95 }; The filtered data table serves as the input for step S2, providing a high-quality, low-noise structural foundation for subsequent generation of semantic descriptions and construction of semantic vectors using a large language model.

[0036] In some embodiments, the method further includes performing integrity checks on field comments, marking the field as a low-confidence field when the comment is empty or contains only placeholder content; specifically: The percentage of fields deemed invalid or low-confidence in each data table is calculated. When this percentage exceeds a preset threshold, the entire corresponding data table is marked as a low-confidence table, and a penalty factor is applied during subsequent sorting. In the `billing_records` section, the fields `id` and `cust_id`, being identifier fields, are marked as low-weight fields, while the fields `amt`, `period_start`, and `period_end` are retained as valid structure fields. Correspondingly, the structure cleaning results also include validity and confidence labels to guide the processing strategies of subsequent modules.

[0037] S2. Generate field annotations for semantic enhancement descriptions based on the table structure information of the filtered data table, obtain the data table with semantic enhancement descriptions, and generate table-level semantic vectors based on the data table with semantic enhancement descriptions. Step S2 is as follows: (1) Extract the table structure information of the filtered data table and its context information in the table creation statement; (2) Take the context information, field name, and field annotation as input, call the large language model to generate field annotations for semantically enhanced descriptions, replace the original field annotations with the field annotations for semantically enhanced descriptions, and obtain the data table for semantically enhanced descriptions; (3) The table structure information of the data table described by semantic enhancement is concatenated and a table-level semantic vector is generated by a vector encoding function.

[0038] After the structural information preprocessing is completed in step S1, a set of table-level and field-level structural information with relatively clear semantics but still mainly based on structured naming is obtained. However, in enterprise databases, field naming often suffers from abbreviations, coding, or lack of comments, and relying solely on field names and comments is still insufficient to accurately reflect their true business meaning. Therefore, based on the structural cleaning results, this invention introduces a semantic enhancement modeling step S2 to supplement and strengthen the semantic expressive power of structural elements.

[0039] (1) Semantic modeling input construction In this embodiment, the table structure information of the filtered data table output in step S1 is used as input to construct semantically enhanced input for each data table and its fields. Specifically, for each field, at least the following information is integrated as semantic modeling context: Field name, field comments (if any), field's structural position in the table, table name and table-level context information of the data table to which the field belongs, field type and constraints.

[0040] For example, for the field amt in the billing_records table, after structural cleaning, the following semantic modeling input is constructed: Field name: amt; Field type: DECIMAL; Field Note: Amount; Belonging table: billing_records; Table Purpose: Billing Record Table; This input includes not only the field's own information, but also the context of the table in which it resides, in order to avoid semantic ambiguity between fields in different business tables.

[0041] (2) Semantic Enhancement Generation After constructing the semantic modeling input, a large language model with natural language understanding and code structure understanding capabilities is invoked to perform semantic reasoning on the fields and data tables, generating corresponding semantically enhanced field annotations. These semantically enhanced field annotations are used to describe the true meaning of fields or tables at the business level in natural language, thereby compensating for issues such as non-standard field naming or missing annotations.

[0042] In the example above, the large language model can generate the following semantically enhanced description for "Field annotation: Amount": "Indicates the amount of fees incurred by the customer during the billing cycle." The semantically enhanced field annotations replace the original field annotations and participate in subsequent processing along with other original table structure information. The semantically enhanced data table is obtained by replacing the original field annotations with the semantically enhanced field annotations.

[0043] (3) Semantic vector representation generation After generating the data table with semantically enhanced descriptions, the semantically enhanced descriptions are further combined and encoded with the corresponding structural identifiers to generate a vectorized representation for subsequent semantic retrieval. In this embodiment, the generation process of field-level and table-level semantic vectors includes: The process involves concatenating field names, field comments, and semantic enhancement descriptions, inputting the concatenated text into a vector encoding model, and outputting field-level semantic vectors to represent the position of the field in the semantic space. Similarly, the table name is concatenated with the table-level semantic enhancement description to generate a table-level semantic vector.

[0044] For example, the vectorized input text of the field amt can be represented as: The AMT amount represents the amount of charges incurred by the customer during the billing cycle. The corresponding field-level semantic vector V(amt) is generated and used for subsequent semantic similarity calculation.

[0045] (4) Output of semantic enhancement results After completing the semantic enhancement modeling, a set of structured output results will be generated for each data table and its fields. The results will include at least: table name, field name, field annotations with semantic enhancement descriptions, and table-level semantic vectors.

[0046] The output will serve as the common input for both step S3 (inverted index construction) and step S4 (semantic recall), enabling simultaneous keyword matching and semantic similarity calculations in subsequent steps.

[0047] S3. Construct an inverted index based on the data table with semantically enhanced description; Step S3 is as follows: (1) Construct a set of index terms from the table structure information of the data table described by semantic enhancement. Each index term is associated with its own data table identifier and field position identifier, which is used to locate the source of the match when a match is hit. The types of index terms include table name type, field name type, and field comment type. (2) Set field-level weights for different types of index terms, including table name weight, field name weight, and field comment weight; (3) Write the index terms, field weights and their positions in the corresponding data tables into the inverted index structure to build the inverted index.

[0048] After completing structural information preprocessing and semantic enhancement modeling, a set of data tables with semantic enhancement descriptions has been obtained, including table names, field names, and field annotations with semantic enhancement descriptions. To support efficient, controllable, and interpretable keyword matching and constraint filtering during the query phase, this invention constructs an inverted index that supports multi-field weighted matching.

[0049] (1) Construction of index terms In this embodiment, the structural information of each data table is first broken down into several index terms. The index terms are derived from at least the following structural dimensions: table name, field name, and field comments with semantic enhancement descriptions.

[0050] Each index term is bound to its corresponding data table identifier and field position identifier during construction, so that the source of the hit can be accurately located when the query is hit, forming a traceable relationship of "keyword-field-data table".

[0051] For example, for the field amt in the billing_records table, the following set of index terms can be constructed: Table name type: billing_records, billing records, customer billing table; Field name type: amt, amount, fee amount; Field annotation type: The amount of fees incurred by the customer during the billing cycle.

[0052] All of the above terms are mapped to the corresponding data table billing_records, and their source field is marked as amt.

[0053] (2) Field-level weight settings Considering the varying importance of different structural dimensions in expressing the business semantics of data tables, this invention introduces a field-level weighting mechanism when constructing inverted indexes to characterize the contribution of different index terms to "data table relevance determination".

[0054] In this embodiment, the setting of field-level weights follows these principles: The index term corresponding to the table name of the data table has the highest weight; The next best option is the index term corresponding to the field name; The index terms corresponding to the field annotations described by semantic enhancement have relatively low weights.

[0055] By setting the weights as described above, during the query phase, when a user's keywords match the table name or core field name, the data table can be given a higher matching score, while the cases where only annotations or semantic descriptions are matched are used as auxiliary scores in the ranking, thereby improving the stability and accuracy of the search results.

[0056] (3) Inverted index generation After completing the construction and weight settings of the index terms, the index terms, field-level weights, and position information are written into the inverted index structure. The inverted index uses keywords as entry points and records the set of data tables that the keyword may hit, as well as the position and weight information of their corresponding fields.

[0057] In this embodiment, the logical structure of the inverted index is represented as follows: { "Amount": [ { "table": "billing_records", "column": "amt", "weight": 0.8 } ], "client": [ { "table": "billing_records", "column": "cust_id", "weight": 0.6 } ] }; This inverted index not only supports fast keyword matching, but also outputs the hit fields and weight information during the matching stage, providing basic data support for subsequent keyword constraint filtering and fusion sorting.

[0058] (4) Index result output and connection After the inverted index is built, a set of structured, weighted, and interpretable index data will be generated as input for subsequent modules: Provides field and table-level structure context support for semantic recall in step S4; This provides a basis for calculating keyword hit rate and coverage for keyword constraint and filtering in step S5; This provides the basic data for keyword matching scores in step S6, which involves merging and ranking.

[0059] S4. Obtain the query vector based on the input query, obtain the semantic similarity score based on the query vector and the table-level semantic vector, and filter the data tables for semantic enhancement description based on the semantic similarity score to obtain a set of candidate data tables. Step S4 is as follows: (1) Vectorize the input query to obtain the query semantic vector; (2) Calculate the similarity between the query semantic vector and the table-level semantic vector of each data table to obtain the table-level semantic similarity score; (3) Based on the table-level semantic similarity score, a threshold screening method or a Top-K strategy screening method is used to screen the data tables of semantic enhancement descriptions to obtain a set of candidate data tables.

[0060] Specifically: (1) The input query is vectorized to obtain the query semantic vector V(Q), where Q represents the query and V() represents the vectorized representation used for semantic similarity calculation. In this embodiment, the table-level semantic similarity score is calculated using cosine similarity.

[0061] (2) Calculate the similarity between V(Q) and the table-level semantic vector V(Ti) of each data table to obtain the table-level semantic similarity score s. sem (Q,Ti) is used to measure the relevance of each data table in the semantic space between the query and the data table of the semantic enhancement description, and supports semantic generalization matching in the cold start scenario; where Ti represents the data table of the semantic enhancement description with number i. (3) Based on the table-level semantic similarity score, the data tables of semantic enhancement description are filtered by threshold filtering or Top-K strategy to obtain the candidate data table set Ti'.

[0062] After completing the inverted index construction, a set of structured, weighted, and interpretable index data has been obtained, along with semantic vector representations at both the table and field levels. To avoid insufficient recall due to relying solely on keyword matching in cold start scenarios, this invention first performs a semantic recall step during the query processing phase, which is used to filter out a set of candidate tables that are highly semantically relevant to the query from the full dataset.

[0063] (1) Query semantic encoding When a user initiates a query request, the keywords or natural language query entered by the user are first semantically encoded, mapping the query text into a query semantic vector. This vector is used to characterize the overall intent of the query in the semantic space.

[0064] In this embodiment, the user may enter the following query: "Find tables related to customer fees, but excluding customer IDs." After preprocessing the query, it is fed into the semantic encoding model to generate a query semantic vector V(Q), where Q represents the user query text and V(Q) represents the corresponding vectorized semantic representation.

[0065] (2) Calculation of semantic similarity at the table level After obtaining the query semantic vector V(Q), its similarity is calculated with the table-level semantic vector V(Ti) corresponding to each data table in the database. The table-level semantic vector V(Ti) is generated in step S2 and is used to comprehensively characterize the semantic meaning of the data table at the business level.

[0066] In this embodiment, the similarity between the query vector V(Q) and the table-level semantic vector V(Ti) of each data table is calculated to obtain the table-level semantic similarity score s. sem (Q, Ti). This score reflects the degree of match between the query and the overall semantics of the data table.

[0067] For example, the table billing_records has a table-level semantic description of "used to record billing and fee information of customers in different time periods". Therefore, under the query related to "customer fees", its table-level semantic similarity score will be significantly higher than that of unrelated data tables.

[0068] (3) Generation of semantic recall candidate table After calculating the table-level semantic similarity, a candidate data table set is generated based on the comprehensive table-level semantic similarity score. The candidate data table set can be generated using any of the following methods or a combination thereof: Filtering based on semantic similarity threshold; Select the top-K data tables based on semantic similarity; The threshold is dynamically adjusted while ensuring the minimum number of recalls.

[0069] In this embodiment, the top-ranked data tables in terms of table-level semantic similarity scores are selected as a set of candidate data tables. Among them, the billing_records table is stably included in the set of candidate data tables because its table-level semantic similarity score is highly matched with the query intent.

[0070] This candidate data set will serve as input for keyword constraint and filtering in step S5, and will be used to further refine the semantic recall results by combining keyword rules.

[0071] S5. Perform keyword matching calculation on the query in the candidate data table set according to the inverted index to obtain keyword matching score, query coverage score and exact match score. Filter the data tables that do not meet the mandatory keyword set coverage constraint or hit the negative keyword set to generate filtered candidate data tables. Step S5 is as follows: (1) Perform query matching calculation on the candidate data table set in the inverted index to obtain the keyword matching score and obtain the set of hit keywords for each candidate data table, where the set of hit keywords represents the set of keywords hit in the table name, field name or comment of the candidate data table; (2) Perform keyword parsing on the query to obtain the set of required matching keywords, the set of optional extended keywords, and the set of negative keywords. The set of required matching keywords is used to express the constraints that cannot be missing in the user's intent, and the set of negative keywords is used to express the reverse semantics that need to be excluded. (3) Calculate the coverage of the hit set of each candidate data table to the set of keywords that must be matched, and obtain the query coverage score; Calculate an exact match score for each candidate data table based on whether it matches the set of negative keywords. Filter data tables that do not meet the requirement of matching the keyword set coverage constraint or hit the negative keyword set, and generate a candidate data table after keyword matching.

[0072] Keyword matching scores are calculated using TF-IDF. Keyword matching score lex (Q,Ti): Scoring the keyword matching strength of each data table in the candidate data table set based on the inverted index, used to measure the accuracy of word-meaning hit; The query coverage score is: (The number of keywords that must be matched in the keyword set and the hit keyword set) / (Denominator); The denominator is the larger of the number of keywords in the set of keywords that must be matched and 1. Query coverage score cov (Q, Ti): This measures the degree to which the required keywords are matched across all tables in the candidate data table set, serving as a constraint satisfaction metric. The query coverage score is calculated as follows: s cov (Q,Ti)=(|K must ∩hits(Ti')|) / (max(1,|K must |)); |K must ∩hits(Ti')| represents the number of keywords that must be matched in the keyword set and the hit keyword set; where K must This indicates that a set of keywords must be matched. `hits(Ti')` represents the set of keywords that must be matched from the candidate data table set, where `Ti'` is the candidate data table set. The denominator is `max(1,|K)`. must |) is used to avoid division by zero errors when the required keyword set is empty. |K must | represents the number of keywords in the set of keywords that must be matched, max(1,|K) must |) represents taking values ​​of 1 and |K. must The value of | Zhongda.

[0073] The exact match rating is: When a candidate data table matches the set of negative keywords, the exact match score for that candidate data table is 1; when a candidate data table does not match the set of negative keywords, the exact match score for that candidate data table is 0.

[0074] Specifically, the exact match score is: When the candidate data table matches the set of negative keywords K not When the candidate data table does not match the set of negative keywords K, the exact match score is 1; when the candidate data table does not match the set of negative keywords K, the score is 1. not At that time, the exact match score for the candidate data table was 0.

[0075] Exact match ratings exact (Q,T): 1 is set when the candidate table hits a high-precision anchor point (including but not limited to precise phrases, unique identifiers, key business codes, etc.), and 0 is set otherwise. This is used to improve the sorting stability in strongly constrained scenarios.

[0076] After semantic recall, a set of candidate data tables that are highly relevant to the user's query at the semantic level has been obtained. However, semantic similarity itself has a certain degree of generalization, and without constraints, it may introduce results that are not entirely consistent with the user's explicit intent. To address this, this invention introduces keyword constraints and filtering steps on top of semantic recall, and further filters and refines the candidate table set by combining inverted indexes and query semantic structure.

[0077] (1) Keyword analysis and classification In this embodiment, the user query is first parsed using keywords, breaking down the query statement into several sets of keywords with different semantic constraints. Specifically, this includes: Must match keyword set K must : Used to express the core constraints that are indispensable in a user query; Optional extended keyword set K exp Used to assist in matching and improve search coverage; Negative keyword set K not : Used to express the reverse semantic conditions that need to be excluded.

[0078] For example, regarding user queries: "Find tables related to customer fees, but excluding customer IDs." The following set of keywords can be obtained through parsing: K must ={"Customer","Expense"}; K not ={"ID","CustomerID"}; The above keyword set will serve as the basis for subsequent keyword matching and filtering.

[0079] (2) Keyword matching based on inverted index After obtaining the keyword set, based on the inverted index constructed in step S3, keyword matching is performed on the candidate data table set for semantic recall in step S4. For each candidate data table in the candidate data table set, the query keywords are checked in its structural dimensions such as table name, field names, semantic enhancement descriptions, and field comments.

[0080] In this embodiment, for the candidate table billing_records: The keyword "cost" can be used to target the field comments and semantic enhancement descriptions of the field "amt"; The keyword "customer" can be used to match the field name and semantic description of the field cust_id.

[0081] The above hit results are recorded as the keyword hit set hits(Ti) of the candidate table, and the position and type of the hit fields are retained for subsequent coverage calculation and result interpretation.

[0082] (3) Coverage constraints and filtering After keyword matching is completed, a query coverage score for each candidate table to match the required keywords is further calculated to measure whether the candidate table meets the user's explicit constraints. The query coverage score measures the completeness to which the required keywords are matched in the candidate tables.

[0083] In this embodiment, the candidate tables are assessed for coverage based on the query coverage score calculation formula. When a candidate table's query coverage score for required keywords is lower than a preset threshold, the candidate table will be directly eliminated or subject to significant penalties in subsequent ranking.

[0084] For example, if a candidate table only matches "customer" but not "fees", its query coverage score is insufficient and does not meet the user's core constraints, so it will be filtered out from the candidate data table set.

[0085] (4) Negation semantic filtering To avoid introducing results that conflict with users' explicit exclusion conditions during the semantic recall phase, a negative semantic filtering mechanism is further introduced on top of the coverage constraint. This mechanism works when the keyword hit set of the candidate data table contains a set of negative keywords K. not When a term is selected from the candidate data table, the candidate data table will be either removed or given a negative weight.

[0086] In the example above, the user explicitly stated the condition "does not contain customer ID". If a candidate data table contains "customer ID" in its field name or semantic description, the candidate data table will be filtered or significantly downweighted according to the negation semantic rule, thereby preventing that result from entering the final output.

[0087] (5) Output and connection of filtering results After keyword matching, coverage constraints, and negative semantic filtering, a set of filtered candidate data tables will be obtained that are semantically relevant and satisfy the user's explicit intent at the keyword constraint level. These filtered candidate data tables will serve as input for step S6 multi-rating fusion ranking, used to further integrate multiple relevance scores for fine-grained ranking.

[0088] S6. Calculate the fusion ranking score for the filtered candidate data table based on the semantic similarity score, keyword matching score, query coverage score, and exact match score, and rank the filtered candidate data table according to the fusion ranking score.

[0089] Step S6 is as follows: The candidate data table after keyword matching is calculated with a fusion ranking score based on semantic similarity score, keyword matching score, query coverage score, and exact match score. The fusion ranking score = α × semantic similarity score + β × keyword matching score + γ × exact match score + δ × query coverage score; Wherein, α, β, γ, and δ are non-negative weight parameters used to characterize the contribution of each score to the final ranking, and satisfy α+β+γ+δ=1; The candidate data table after keyword matching is sorted according to the fusion sorting score.

[0090] Step S6 is as follows: The candidate data table after keyword matching is calculated with a fusion ranking score based on semantic similarity score, keyword matching score, query coverage score, and exact match score. S core (Q,Ti)=α⋅s sem (Q,Ti)+β⋅s lex (Q,Ti)+γ⋅s exact (Q,Ti)+δ⋅s cov (Q,Ti); Wherein, α, β, γ, and δ are non-negative weight parameters used to characterize the contribution of each score to the final ranking, and satisfy α+β+γ+δ=1; The candidate data table after keyword matching is sorted according to the fusion sorting score.

[0091] When multiple candidate data matching keywords have the same fusion ranking score, they are ranked in parallel in the order of exact match score first, query coverage score second, and keyword match score third, to ensure the stability of the results.

[0092] When the merged ranking scores of candidate data after multiple keyword matches are the same, the exact match score will be used. exact (Q,Ti) priority, s cov (Q,Ti) Query coverage score is second, followed by keyword matching score. lex (Q,Ti) are then sorted in parallel to ensure the stability of the results.

[0093] After keyword constraints and filtering, a set of filtered candidate data tables that satisfy the user's intent in terms of semantic relevance and explicit constraints has been obtained. To finely differentiate among multiple filtered candidate data tables and output the optimal result while ensuring stability and interpretability, this invention performs a multi-rating fusion sorting and result output step.

[0094] (1) Calculation of merged ranking score After obtaining the semantic similarity score, keyword matching score, query coverage score, and exact match score, these scores are weighted and fused to calculate the fusion ranking score for each filtered candidate data table. The weights of different scores in the fusion reflect their contribution to the overall relevance judgment and can be configured or optimized according to actual business needs.

[0095] In this embodiment, the filtered candidate data tables are sorted according to the fusion ranking score, so that the data tables with the best overall performance in terms of semantic relevance, keyword matching degree and constraint satisfaction are ranked first.

[0096] For example, in a scenario where a user queries "find tables related to customer fees, but excluding customer IDs", the billing_records table, although containing the cust_id field, still meets the core constraints after keyword constraints and negative semantic filtering, and has a significant semantic and keyword matching advantage in the amt field. Therefore, it gets a higher score in the fusion ranking and appears at the top of the ranking results.

[0097] (3) Parallel sorting and stability control When the combined ranking scores of multiple filtered candidate data tables are close or the same, a stability control rule is further introduced to perform parallel ranking of the filtered candidate data tables. Specifically, the tables can be ranked in parallel in the following order: first, exact match score; second, query coverage score; and third, keyword match score. This secondary ranking avoids unstable fluctuations in the ranking results under different query or parameter conditions.

[0098] (4) Result output and interpretation information generation After completing the fusion and sorting, the final sorting result is output to the user or downstream application module. In addition to returning the sorted list of candidate data tables, explanatory information corresponding to the search results is also generated simultaneously to enhance the understandability and verifiability of the results.

[0099] In this embodiment, the explanation information includes at least: The field names and their corresponding structure types in the candidate data table that were matched; Query the correspondence between keywords and matched fields or table names; The relative contributions of semantic similarity score, keyword matching score, query coverage score, and exact match score to the fusion ranking score.

[0100] For example, for the billing_records table, the following explanatory information can be provided: "The field amt (amount) was hit, contributing significantly to semantic similarity; the keyword 'fee' was hit in the field annotation; the negative keyword 'customer ID' was not hit," thus enabling users to intuitively understand why the table was selected.

[0101] (5) Final result output After completing the above steps, the sorting results and corresponding explanatory information will be output as the final search results, and the entire hybrid table search process will then be completed.

[0102] This embodiment focuses on the cold start scenario of enterprise databases and provides a hybrid table retrieval method based on structural information preprocessing, semantic enhancement modeling, inverted index construction, semantic recall, keyword constraint filtering, and multi-score fusion ranking. Each processing module uses the output of the previous module as the input of the next module, forming a complete and closed-loop retrieval process. This achieves effective retrieval of newly created or semantically sparse data tables without relying on historical query logs or manually labeled information.

[0103] By filtering invalid fields and low-confidence structures during the structural preprocessing stage, introducing a large language model to supplement the business semantic expression of fields and tables during the semantic enhancement stage, and combining semantic similarity scoring and keyword constraint mechanisms during the retrieval stage, this embodiment can balance semantic recall capability and precise keyword control, effectively avoiding the semantic drift problem introduced by pure semantic retrieval. Furthermore, through a multi-scoring fusion ranking and explanatory information output mechanism, it can not only output stable and accurate retrieval results, but also provide clear hit criteria and ranking reasons, improving the interpretability and verifiability of the results.

[0104] The method described in this embodiment can be widely applied to enterprise-level database management, intelligent data querying, data asset governance and related scenarios, and is especially suitable for database structures.

[0105] Example 2

[0106] like Figure 2 A table retrieval device for a database cold start scenario, comprising: The first acquisition module is used to acquire the table structure information of several data tables, filter the data according to the table structure information, and obtain the filtered data tables. The table structure information includes at least the table name, field name, and field comments. The first generation module is used to generate field annotations for semantically enhanced descriptions based on the table structure information of the filtered data table, obtain the data table with semantically enhanced descriptions, and generate table-level semantic vectors based on the data table with semantically enhanced descriptions. The building module is used to construct inverted indexes based on data tables with semantically enhanced descriptions; The first calculation module is used to obtain a query vector based on the input query, obtain a semantic similarity score based on the query vector and the table-level semantic vector, and filter the data table of semantically enhanced description based on the semantic similarity score to obtain a set of candidate data tables. The second generation module is used to perform keyword matching calculation on the query in the candidate data table set according to the inverted index, obtain keyword matching score, query coverage score and exact match score, filter data tables that do not meet the mandatory keyword set coverage constraint or hit the negative keyword set, and generate filtered candidate data tables. The second calculation module is used to calculate the fusion ranking score of the filtered candidate data table based on the semantic similarity score, keyword matching score, query coverage score and exact matching score, and to rank the filtered candidate data table according to the fusion ranking score.

[0107] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0108] Accordingly, this application also provides an electronic device, including: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors perform the methods described above.

[0109] Accordingly, this application also provides a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of any of the above methods.

[0110] In the embodiments provided in this application, it should be understood that the disclosed methods and embodiments can also be implemented in other ways. The methods and embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of methods and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using dedicated hardware-based implementations that perform the specified functions or actions, or can be implemented using a combination of dedicated hardware and computer instructions.

[0111] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0112] On the other hand, a computer-readable storage medium stores computer instructions thereon, which, when executed by a processor, implement the steps of the above-described method. When the computer program is executed by the processor, it implements the method as described in any of the first aspects above. If the function is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0113] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A table retrieval method for a database cold start scenario, characterized in that, Includes the following steps: S1. Obtain the table structure information of several data tables, filter according to the table structure information, and obtain the filtered data tables. The table structure information includes at least the table name, field name, and field comments. S2. Generate field annotations for semantic enhancement descriptions based on the table structure information of the filtered data table, obtain the data table with semantic enhancement descriptions, and generate table-level semantic vectors based on the data table with semantic enhancement descriptions. S3. Construct an inverted index based on the data table with semantically enhanced description; S4. Obtain the query vector based on the input query, obtain the semantic similarity score based on the query vector and the table-level semantic vector, and filter the data tables for semantic enhancement description based on the semantic similarity score to obtain a set of candidate data tables. S5. Perform keyword matching calculation on the query in the candidate data table set according to the inverted index to obtain keyword matching score, query coverage score and exact match score. Filter the data tables that do not meet the mandatory keyword set coverage constraint or hit the negative keyword set to generate filtered candidate data tables. S6. Calculate the fusion ranking score for the filtered candidate data table based on the semantic similarity score, keyword matching score, query coverage score, and exact match score, and rank the filtered candidate data table according to the fusion ranking score.

2. The method according to claim 1, characterized in that, In step S1, the filtering involves filtering structural items in the table structure information that do not have actual business semantics or have obvious template characteristics, while retaining table-level and field-level structural elements that have retrieval significance.

3. The method according to claim 1, characterized in that, Step S1 is as follows: (1) Obtain the table structure information of several data tables in the database, wherein the table structure information includes at least the table name, field name, and field comments; (2) Perform keyword filtering and regular expression matching on the field names to remove fields whose names are string, number, value, col1, data1, temp, misc, unnamed or variations of the above field names, and obtain the filtered table structure information.

4. The method according to claim 1, characterized in that, Step S2 is as follows: (1) Extract the table structure information of the filtered data table and its context information in the table creation statement; (2) Take the context information, field name, and field annotation as input, call the large language model to generate field annotations for semantically enhanced descriptions, replace the original field annotations with the field annotations for semantically enhanced descriptions, and obtain the data table for semantically enhanced descriptions; (3) The table structure information of the data table described by semantic enhancement is concatenated and a table-level semantic vector is generated by a vector encoding function.

5. The method according to claim 1, characterized in that, Step S3 is as follows: (1) Construct a set of index terms from the table structure information of the data table described by semantic enhancement. Each index term is associated with its own data table identifier and field position identifier, which is used to locate the source of the match when a match is hit. The types of index terms include table name type, field name type, and field comment type. (2) Set field-level weights for different types of index terms, including table name weight, field name weight, and field comment weight; (3) Write the index terms, field weights and their positions in the corresponding data tables into the inverted index structure to build the inverted index.

6. The method according to claim 1, characterized in that, Step S4 is as follows: (1) Vectorize the input query to obtain the query semantic vector; (2) Calculate the similarity between the query semantic vector and the table-level semantic vector of each data table to obtain the table-level semantic similarity score; (3) Based on the table-level semantic similarity score, a threshold filtering or Top-K strategy is used to obtain a set of candidate data tables.

7. The method according to claim 1, characterized in that, Step S5 is as follows: (1) Perform query matching calculation on the candidate data table set in the inverted index to obtain the keyword matching score and obtain the set of hit keywords for each candidate data table, where the set of hit keywords represents the set of keywords hit in the table name, field name or comment of the candidate data table; (2) Perform keyword parsing on the query to obtain the set of required matching keywords, the set of optional extended keywords, and the set of negative keywords. The set of required matching keywords is used to express the constraints that cannot be missing in the user's intent, and the set of negative keywords is used to express the reverse semantics that need to be excluded. (3) Calculate the coverage of the hit set of each candidate data table to the set of keywords that must be matched, and obtain the query coverage score; Calculate an exact match score for each candidate data table based on whether it matches the set of negative keywords. Filter data tables that do not meet the requirement of matching the keyword set coverage constraint or hit the negative keyword set, and generate a candidate data table after keyword matching.

8. The method according to claim 7, characterized in that, The table-level semantic similarity score is calculated using cosine similarity. Keyword matching scores are calculated using TF-IDF. The query coverage score is: (The number of keywords that must be matched in the keyword set and the hit keyword set) / (Denominator); The denominator is the larger of the number of keywords in the set of keywords that must be matched and 1. The exact match rating is: When a candidate data table matches the set of negative keywords, the exact match score for that candidate data table is 1; when a candidate data table does not match the set of negative keywords, the exact match score for that candidate data table is 0.

9. The method according to claim 7, characterized in that, Step S6 is as follows: The candidate data table after keyword matching is calculated with a fusion ranking score based on semantic similarity score, keyword matching score, query coverage score, and exact match score. The fusion ranking score = α × semantic similarity score + β × keyword matching score + γ × exact match score + δ × query coverage score; Wherein, α, β, γ, and δ are non-negative weight parameters used to characterize the contribution of each score to the final ranking, and satisfy α+β+γ+δ=1; The candidate data table after keyword matching is sorted according to the fusion sorting score.

10. The method according to claim 8, characterized in that, When multiple candidate data matching keywords have the same fusion ranking score, they are ranked in parallel in the order of exact match score first, query coverage score second, and keyword match score third, to ensure the stability of the results.

11. A table retrieval device for a database cold start scenario, characterized in that, include: The first acquisition module is used to acquire the table structure information of several data tables, filter the data according to the table structure information, and obtain the filtered data tables. The table structure information includes at least the table name, field name, and field comments. The first generation module is used to generate field annotations for semantically enhanced descriptions based on the table structure information of the filtered data table, obtain the data table with semantically enhanced descriptions, and generate table-level semantic vectors based on the data table with semantically enhanced descriptions. The building module is used to construct inverted indexes based on data tables with semantically enhanced descriptions; The first calculation module is used to obtain a query vector based on the input query, obtain a semantic similarity score based on the query vector and the table-level semantic vector, and filter the data table of semantically enhanced description based on the semantic similarity score to obtain a set of candidate data tables. The second generation module is used to perform keyword matching calculation on the query in the candidate data table set according to the inverted index, obtain keyword matching score, query coverage score and exact match score, filter data tables that do not meet the mandatory keyword set coverage constraint or hit the negative keyword set, and generate filtered candidate data tables. The second calculation module is used to calculate the fusion ranking score of the filtered candidate data table based on the semantic similarity score, keyword matching score, query coverage score and exact matching score, and to rank the filtered candidate data table according to the fusion ranking score.

12. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-10.

13. A computer-readable storage medium storing computer instructions thereon, characterized in that, When executed by the processor, this instruction implements the steps of the method as described in any one of claims 1-10.

Images