Method and apparatus for index construction of a data table, querying a data table

Abstract

Embodiments of the present specification provide a method and device for index construction of a data table and querying the data table. The method comprises: obtaining metadata information and supplementary information of a target data table to be processed, the supplementary information comprising data association information between the target data table and other data tables and / or historical query information, the historical query information comprising historical query records performed by a user on the target data table; performing semantic analysis on the metadata information and the supplementary information based on a large model to generate enhanced description information for representing business semantics of the target data table; performing vectorization processing on the enhanced description information to obtain vector representation, and constructing a vector index library corresponding to a plurality of target data tables based on the vector representation. The method can reduce the dependence of the data table query process on the quality of the original metadata.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of large model technology, specifically to a method and apparatus for constructing an index for a data table and querying a data table. Background Technology

[0002] As enterprises continue their digital transformation, the number of data tables in data warehouses is growing rapidly. Large-scale big data computing platforms typically contain tens of thousands of data tables, which constitute a crucial carrier of enterprise data assets. To facilitate the management and use of these data assets, metadata management is usually required for the data tables. This involves recording the structured definition information of the data tables, such as field names, data types, and field comments. This information collectively constitutes the schema information of the data tables.

[0003] In real-world data development and analysis scenarios, developers typically need to quickly locate the appropriate data table from a large number of data tables based on business requirements. This process is usually referred to as data table discovery, data table query, or table lookup.

[0004] Existing data table discovery solutions primarily rely on data catalog tools or keyword-based retrieval systems. These solutions typically create a metadata index for the data tables, match the indexed content with the user's input of query keywords, and return data tables containing relevant keywords. With the development of natural language processing technology, some solutions further incorporate vector retrieval techniques. This involves using embedding models to convert the descriptive text of the data tables into vector representations, while simultaneously converting the user's natural language query into a query vector, and then retrieving relevant data tables by calculating vector similarity.

[0005] However, in practical applications, the above solutions often fail to achieve accurate data table discovery in complex scenarios. The main reason is that existing retrieval mechanisms rely heavily on the quality of the original metadata of the data tables. During the actual maintenance of large data warehouses, due to a lack of standardized annotation habits among table creators, or frequent iterations of business logic without timely updates to annotation information, many data tables suffer from missing, simplified, or outdated descriptive information (such as field comments) in their original metadata. When the original metadata information fails to accurately reflect the actual business meaning of the data table, keyword indexes or vector representations built based on this information are prone to bias, leading to search results that do not match the user's true intent.

[0006] For example, given two data tables with similar functions, if one table has more complete descriptive information while the other has missing or incomplete information, existing solutions often only retrieve the table with complete descriptive information, ignoring the other table that may better meet user needs in actual business operations. Furthermore, retrieval methods based on surface-level text descriptions are generally unable to effectively depict complex business semantics and the relationships between data tables, thus failing to respond to natural language queries with strong abstraction or implicit business logic.

[0007] Therefore, how to reduce the dependence of the data table query process on the quality of the original metadata and improve the accuracy of data table discovery in complex scenarios has become a technical problem that urgently needs to be solved in the current technology field. Summary of the Invention

[0008] This specification provides an embodiment of a scheme for constructing indexes for data tables and querying data tables, which can improve the accuracy of data table queries.

[0009] In a first aspect, embodiments of this specification provide a method for constructing an index for a data table, comprising: acquiring metadata information and supplementary information of a target data table to be processed, wherein the supplementary information includes data association information between the target data table and other data tables and / or historical query information, wherein the historical query information includes historical query records executed by users on the target data table; performing semantic analysis on the metadata information and the supplementary information based on a large model to generate enhanced description information for characterizing the business semantics of the target data table; vectorizing the enhanced description information to obtain a vector representation, and constructing a vector index library corresponding to multiple target data tables based on the vector representation.

[0010] In one implementation, the process of acquiring the historical query information includes: receiving a data query request input by a user, the data query request being used to query target data in a target data table; converting the data query request into an executable query statement based on a large model, and during the generation of the query statement, prompting the large model to perform semantic parsing of the data query request and insert annotation fields containing business semantics into the query statement through a preset prompt word strategy; executing the query statement, and generating historical query information for the data table based on the execution result and the query statement.

[0011] In one implementation, the annotation field includes at least one of the following: a description of the business to which the query belongs, used to identify the business scenario to which the query belongs; a description of the user's requirements, used to explain the purpose of the user initiating the query; and a description of the purpose of the data table, used to explain the purpose of the target data table being queried in the current query scenario.

[0012] In one implementation, the data association information is used to characterize the upstream source relationship and downstream dependency relationship between the target data table and other data tables.

[0013] Secondly, embodiments of this specification provide a method for querying a data table. The method includes: receiving a user query from a target user and vectorizing the user query to obtain a query vector; recalling multiple candidate vector representations matching the query vector in a vector index library, wherein the vector index library is constructed using the method described in any one of the first aspects, and each candidate vector representation corresponds to enhanced description information of a data table; for each enhanced description information of a candidate vector representation, calculating a score for the data table corresponding to the candidate vector representation based on the correlation between the user query and the enhanced description information; and based on the score, selecting a target vector representation from the multiple candidate vector representations and determining the data table corresponding to the target vector representation as the query result.

[0014] In one implementation, for each candidate vector representation of enhanced description information, calculating a score for the data table corresponding to the candidate vector representation based on the relevance between the user query and the enhanced description information includes: obtaining the target user's historical demand pattern, where the historical demand pattern represents the target user's demand in historical queries; and for each candidate vector representation of enhanced description information, calculating a score for the data table corresponding to the candidate vector representation based on the relevance between the historical demand pattern and the enhanced description information, and the relevance between the user query and the enhanced description information.

[0015] In one implementation, for each candidate vector representation's enhanced description information, calculating the score of the data table corresponding to the candidate vector representation based on the relevance between the user query and the enhanced description information includes: for each candidate vector representation, obtaining the query frequency of the data table corresponding to the candidate vector representation, where the query frequency represents the frequency at which the data table is queried; and calculating the score of the data table corresponding to the candidate vector representation based on the relevance between the user query and the enhanced description information and the citation frequency.

[0016] In one implementation, for each candidate vector representation's enhanced description information, calculating the score of the data table corresponding to the candidate vector representation based on the relevance between the user query and the enhanced description information includes: for each candidate vector representation, obtaining upstream and downstream reference information of the data table corresponding to the candidate vector representation based on data association information, wherein the upstream and downstream reference information is used to indicate the number of upstream data tables and / or downstream data tables that the data table depends on; and calculating the score of the data table corresponding to the candidate vector representation based on the relevance between the user query and the enhanced description information and the upstream and downstream reference information.

[0017] Thirdly, embodiments of this specification provide an index building apparatus for a data table, comprising: an acquisition module, configured to acquire metadata information and supplementary information of a target data table to be processed, wherein the supplementary information includes data association information between the target data table and other data tables and / or historical query information, wherein the historical query information includes historical query records executed by users on the target data table; an analysis module, configured to perform semantic analysis on the metadata information and the supplementary information based on a large model to generate enhanced description information for characterizing the business semantics of the target data table; and a construction module, configured to vectorize the enhanced description information to obtain a vector representation, and construct a vector index library corresponding to multiple target data tables based on the vector representation.

[0018] Fourthly, embodiments of this specification provide an apparatus for querying a data table. The apparatus includes: a receiving module, configured to receive a user query from a target user and vectorize the user query to obtain a query vector; a recall module, configured to recall multiple candidate vector representations matching the query vector from a vector index library, the vector index library being constructed using the method described in any one of the first aspects, each candidate vector representation corresponding to enhanced description information of a data table; a scoring module, configured to calculate a score for the data table corresponding to each candidate vector representation based on the correlation between the user query and the enhanced description information; and a filtering module, configured to filter a target vector representation from the multiple candidate vector representations based on the score, and determine the data table corresponding to the target vector representation as the query result.

[0019] Fifthly, embodiments of this specification provide a computing device including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements the method described in either the first or second aspect.

[0020] In the solution provided in this specification, by leveraging a large model to deeply integrate the original metadata, data association, and historical query information of the data table, enhanced descriptive information containing more complete business semantics of the data table is generated. This process effectively compensates for semantic gaps caused by missing manual annotations, brief descriptions, or outdated content, significantly reducing the retrieval system's dependence on the quality of the original metadata. Furthermore, the vector representation derived from this enhanced descriptive information has higher semantic richness and accuracy. The constructed vector index library enables the system to not only achieve basic semantic matching when responding to user queries, but also to accurately understand deep business logic and abstract query intent, thereby significantly improving the recall and precision of data table discovery and ensuring that semantically relevant and business-compliant data tables can be quickly and accurately located even in complex business scenarios. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments in this specification, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart of an index construction method for a data table in one of the embodiments of this specification;

[0023] Figure 2 This is a flowchart of a method for querying a data table according to an embodiment of this specification;

[0024] Figure 3 This is a schematic diagram of the process of querying the data table in the embodiments of this specification;

[0025] Figure 4 This is a schematic diagram of the structure of the index building device for the data table in the embodiments of this specification;

[0026] Figure 5 This is a schematic diagram of the device for querying a data table in the embodiments of this specification. Detailed Implementation

[0027] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0028] To facilitate understanding of the solutions described in this manual, some terms used in this manual are explained below.

[0029] In this specification, the Large Language Model (LLM) may also be referred to simply as the Large Model. A Large Language Model is a natural language processing model based on deep learning techniques, typically with billions to hundreds of billions or even more parameters, possessing powerful language understanding and generation capabilities. Large Language Models can employ the Transformer architecture or its variants (such as GPT, BERT, etc.), which utilizes an attention mechanism to globally model sequential data, efficiently handling long-distance dependencies and thus performing exceptionally well in natural language tasks. Large Language Models learn the statistical features and semantic relationships of language through pre-training on large-scale corpora, enabling them to generalize. The core capabilities of Large Language Models include, but are not limited to: understanding contextual semantics, generating coherent and grammatically correct text, performing logical reasoning, and handling multi-task scenarios. Its usage typically includes two modes: direct inference and fine-tuning. In direct inference mode, the user guides the Large Language Model to generate specific outputs by designing prompts. Cue words can be task descriptions or instructions in text form, used to stimulate the semantic understanding and generation capabilities of large language models. In fine-tuning mode, large language models are further trained on small-scale datasets in specific domains to optimize their performance on specific tasks. The powerful generalization ability and flexibility of large language models make them an important tool in the field of artificial intelligence, providing efficient and accurate solutions for automated text generation and understanding.

[0030] In some embodiments, large language models can also understand and generate data from other modalities (such as visual and audio data). In this case, large language models can also be called multimodal large language models (MLLMs). MLLMs provide a richer and more natural interactive experience by integrating multiple types of input and output, such as text, images, and sound. The core advantage of MLLMs lies in their ability to process and understand information from different modalities and fuse this information to complete complex tasks. For example, MLLMs can analyze an image and generate descriptive text, or generate a corresponding image based on a text description. This cross-modal understanding and generation capability makes MLLMs widely applicable across multiple fields.

[0031] It should be noted that the key technologies of large language models can be found in the detailed description in the paper "A Survey of Large Language Models" (paper number: arXiv:2303.18223v16, published on March 11, 2025), and will not be repeated here.

[0032] Vectorization (Embedding) refers to using machine learning or deep learning models to map unstructured data such as text, code, and table descriptions into vectors in a high-dimensional space. This makes semantically similar but differently expressed content appear closer together in the vector space. Based on this capability, data table names, field information, business descriptions, and other content can be vectorized and encoded using models such as BERT, and stored in a vector retrieval system. During queries, user input is also vectorized, and the most semantically relevant tables or fields are quickly retrieved by calculating vector similarity, thus achieving semantic search that differs from traditional keyword matching.

[0033] Reranking refers to the process of further evaluating and refining the candidate results after the system has completed the initial recall and obtained a batch of candidate results. This process utilizes more complex semantic models, relevance calculation methods, or business ranking strategies. The goal is to improve the relevance and accuracy of the final returned results while maintaining retrieval efficiency.

[0034] Upstream and downstream information refers to the source and destination information between data tables in the data processing flow, based on their processing, transmission, and dependency relationships. "Upstream" refers to the data source table of the current table, that is, the table that provides input data to it; "downstream" refers to tables that rely on the data from the current table as input for subsequent processing or analysis. Upstream and downstream relationships together constitute the data lineage, used to describe the flow path of data from its source to its consumer.

[0035] As mentioned earlier, current data development and analysis scenarios commonly suffer from a large number of data tables and disorganized management. Specifically, systems often accumulate tens of thousands of data tables, but lack effective classification, organization, and tagging systems, leading to overall chaotic management. Simultaneously, metadata quality varies greatly; many data tables lack complete and accurate descriptive information, or the existing descriptions are too brief or outdated, failing to accurately reflect the actual content and business implications of the data tables. Consequently, data developers often spend considerable time searching for suitable data tables when facing new requirements, typically relying on manual inquiries, experience-based judgment, or repeated trial and error. This not only results in low retrieval efficiency but also easily leads to misjudgments and redundant work. Therefore, there is an urgent need for an automated, intelligent, efficient, and accurate data table discovery method to help users quickly locate the required data tables and improve the efficiency of data development and usage.

[0036] Based on this, this specification proposes a method for constructing indexes for data tables and a method for querying data tables. The aim is to leverage the semantic understanding and reasoning capabilities of large models to supplement the original metadata information by integrating information from different dimensions, generating enhanced descriptions rich in business semantics, effectively compensating for the lack and lag of descriptive information in the original metadata, and improving the efficiency and accuracy of the core data development activity of "finding tables".

[0037] The following section will first explain the process of building indexes for data tables.

[0038] Figure 1 This is a flowchart of an index construction method for a data table in an embodiment of this specification. This method can be applied to any device, platform, or device cluster with computing and processing capabilities. The method includes steps 101-103 as shown below.

[0039] In step 101, the metadata and supplementary information of the target data table to be processed are obtained.

[0040] In this embodiment, the target data table can be any data table in the database management system where the index is to be built.

[0041] Metadata information, also known as schema information, refers to structured definition information describing the logical structure and field attributes of a data table. Metadata information can be used to characterize the organization of a data table, its field composition, and field constraints. For example, it may include: table name, field names, field data types, field lengths, field precision, default values, whether null values ​​are allowed, primary key constraints, foreign key constraints, uniqueness constraints, index information, and field comments. In some implementations, it may also include encoding rules, enumerated value ranges, time formats, the subject area to which a field belongs, version information, and relationships between fields.

[0042] Metadata information can be obtained in various ways. For example, it can be read from the data dictionary, system tables, information schema, or metadata management interface provided by the database system; it can also be extracted by parsing table creation statements, data modeling documents, interface definition files, configuration files, or table structure description information maintained in the data governance platform; it can also be obtained through manual input or semi-automatic annotation. This embodiment does not limit the metadata information obtained from different sources.

[0043] For example, when metadata information comes from a database management system, you can directly query the system metadata table corresponding to the target data table to obtain information such as field names, field types, field lengths, constraint information, and index information; when metadata information comes from external documents or configuration files, you can extract the corresponding structured description content by parsing preset format files (such as JSON files, XML files, YAML files, CSV files, or DDL scripts).

[0044] Supplementary information includes data relationship information between the target data table and other data tables, and / or historical query information. This embodiment does not strictly limit the specific fields included in the data relationship information and historical query information; any information that can characterize the relationships between tables or user behavior is applicable.

[0045] Data relationship information, also known as data lineage information, represents the relationships between a target data table and other data tables. This includes field-level relationships, primary and foreign key correspondences, inter-table mappings, and business logic constraints. Data relationship information clarifies the target data table's position within the overall data structure, as well as its data references, dependencies, and interactions with other tables. It provides the target data table's dynamic context within the data flow (such as upstream and downstream relationships), which is often key to understanding the table's business meaning and importance.

[0046] Historical query information includes records of queries executed by users on the target data table, primarily referring to the collection of specific query records performed by users on the target data table. For example, historical query records can contain complete query statements or instructions, recording the filtering conditions used by the user during retrieval, the involved tables and their join methods, the fields involved in statistical calculations, and the data sorting rules. Furthermore, this information can also include the time point of the query, the execution frequency, and the identity or role background of the user initiating the query. Query logs generated by users during historical usage often contain rich business context and tacit knowledge. This information, brought in by the user when executing the query, reflects the actual usage of the data table in real-world business scenarios. By integrating this content, historical query information objectively reflects the access paths of the target data table in actual business operations, the specific data subsets of interest, and the usage habits of different user groups, constituting a direct description of the actual operating status of the data table.

[0047] In one implementation, the historical query information does not only contain natural language annotations, but is a structured composite record that includes at least the following three dimensions of information:

[0048] The semantic annotation layer consists of natural language descriptions (i.e., the aforementioned annotation fields) generated by the large model, which contain business scenarios, user needs, and data usage, and are used to represent explicit business intentions.

[0049] The logic code layer contains the original executable query statements (such as SQL). This part retains the field-level selection logic (SELECT), table join methods (JOIN), and filtering conditions (WHERE). It serves as the objective basis for extracting implicit statistical metrics (such as the specific calculation formula for 'GMV') and verifying lineage relationships.

[0050] The execution context layer contains the execution status and result metadata of the query statement, including but not limited to: whether the execution was successful, the number of rows returned, the execution time, and the trigger time. This information is used to quantify the usage frequency of the data table and filter invalid or erroneous query records, ensuring that all information entering the index has been verified in practice.

[0051] For example, when recording historical query information, the system will store the above three elements together. When the large model performs semantic analysis, it will not only read the semantic annotation layer, but also cross-validate whether the calculation logic in the logic code layer is consistent with the annotation (for example, if the annotation says 'status paid orders', the model will check whether the SQL contains the filter condition status='paid'), and combine it with the frequency data in the execution context layer to dynamically adjust the evaluation weight of the business importance of the data table.

[0052] In one implementation, data association information is used to characterize the upstream source relationships and downstream dependencies between the target data table and other data tables. The upstream source relationship characterizes the data tables that provide input data for generating the target data table, i.e., the data tables that provide raw or pre-processed data to the target data table; the downstream dependency relationship characterizes the data tables that use the target data table as an input source and undergo subsequent processing, i.e., the data tables that depend on the target data table for further processing, aggregation analysis, or business applications.

[0053] In one implementation, historical query information can be collected during a user's database query. The process of acquiring this information may include: receiving a user's data query request to retrieve target data from a target data table; transforming the query request into an executable query statement based on a large model; during query statement generation, using a pre-defined prompting strategy to prompt the large model to perform semantic parsing of the query request and insert annotation fields containing business semantics into the query statement; executing the query statement; and generating historical query information for the data table based on the execution result and the query statement. The annotation fields may include at least one of the following: a description of the relevant business; a description of the user's requirements; or a description of the data table's purpose.

[0054] Specifically, when data developers or analysts input data query requirements through a natural language interface, the system backend can call a large language model to transform the requirements into executable Structured Query Language (SQL) statements. For example, a data query requirement could be a request to query or analyze specific target data, such as "query the sales fluctuations of a certain project over the past thirty days" or "analyze the risk data trends of a certain type of customer."

[0055] During this transformation process, the system pre-sets specific prompting strategies to guide the large language model not only to generate syntactically correct SQL code, but also to deeply analyze the user's query intent and automatically insert comment fields containing business semantics into the generated SQL statement, such as inserting them at the beginning of the SQL statement. These comment fields can specifically include: a description of the relevant business context, used to identify the business scenario to which this query belongs, such as "marketing domain - Double 11 event"; a description of the user's needs, used to explain the purpose of the user's query, such as "investigating abnormal fluctuations in product inventory"; and a description of the purpose of the data table, used to explain the specific use of the target data table in this query scenario, such as "providing transaction records as a core fact table".

[0056] Subsequently, the system executes the query statement with rich annotation fields, obtaining the execution result, such as extracting actual indicator data or trend data from the database. The execution result, along with the complete query statement (including automatically generated annotation fields), is recorded to form a structured historical query record. These records, after aggregation, constitute the historical query information for the target data table. In other embodiments, only the annotation fields may be used as the historical query record for the target data table.

[0057] This mechanism makes explicit the tacit knowledge users generate in their daily work, solving the problem of insufficient business semantics in traditional metadata management. In the traditional model, after a data analyst completes a data retrieval task, the underlying business thinking (why this table was queried, what specific problem this table solves) is often lost when the session ends, leaving only a SELECT statement. However, this embodiment captures the user's understanding of the data through annotations automatically generated by the large model. For example, when multiple different analysts execute queries on the same table at different times, and the large model identifies that the table is used in "risk control" or "real-time dashboard display" scenarios, these scattered records, after aggregation, can strongly demonstrate the importance of the table in a specific business domain.

[0058] It's important to note that users can specify the name of the target data table in their data query requirements. Alternatively, if the user doesn't explicitly specify a table name, the data table located by the system's internal table-finding tool will be designated as the target table. Historical query records will remember that "the user ultimately used this table for a need like analyzing sales fluctuations," thus establishing a strong association between the user's intent and the data table. Over time, the more historical query records accumulated, the more accurate and relevant the enhanced descriptive information generated becomes to real-world business scenarios. This significantly reduces the reliance on manual metadata maintenance during the retrieval process and resolves inaccurate table finding issues caused by missing or outdated annotations.

[0059] In other embodiments, the acquisition of historical query information is not limited to real-time interaction processes. The system can also interface with database audit logs or query engine logs to extract historically executed native SQL statements. For these existing SQL statements, the system can call a large language model to perform "semantic reverse engineering": the model not only parses the syntax structure of the SQL (identifying SELECT fields, JOIN join keys, WHERE filter conditions, and aggregate function logic), but also combines the naming conventions of table names and field names to reverse-deduce the business intent behind the query. For example, when it detects that a certain SQL statement executes SUM(pay_amount) on the order table with the filter condition status='paid', the model automatically generates a business semantic annotation similar to "used to calculate the total transaction amount of paid orders". In addition, the system constructs a dynamic implicit lineage graph by statistically analyzing the frequency and join methods of different data tables appearing together in the same SQL statement. If it is found that although Table A and Table B do not have foreign key constraints defined in DDL, but are frequently joined through specific fields in hundreds of different historical SQL queries, the system determines that there is a strong business relationship between the two and clearly marks them in the generated historical query information as "often used in conjunction with Table B for XX scenario analysis". This mechanism allows the system to have rich business semantic indexes and a preliminary lineage network from the initial stage of deployment, and can provide table lookup services without waiting for new users to generate interaction data.

[0060] In step 102, semantic analysis is performed on the metadata and supplementary information based on the large model to generate enhanced descriptive information that characterizes the business semantics of the target data table.

[0061] This step leverages the powerful reasoning capabilities of the large language model to deeply integrate metadata and supplementary information. Specifically, the system assembles the metadata and supplementary information into the input context of the large language model and guides the model to perform semantic parsing, information extraction, and induction operations through preset prompt word templates.

[0062] In one implementation, when supplementary information includes historical query information, the large language model performs deduplication, conflict detection, and semantic summarization operations on multiple historical query records that may exist for the same target data table. Specifically, this may include: comparing the business description fields in multiple historical query records; if differences or conflicts are detected in the usage descriptions of different records for the same data table, the model evaluates the confidence level of each description based on preset weighting rules (e.g., based on query frequency in historical records, the permission level of the submitting user, or time freshness), prioritizing the retention of high-confidence descriptions, and summarizing scattered business scenario tags into standardized business domain terms. For example, business terms naturally included by users during queries (such as "DAU") are recorded, and the large model automatically analyzes and extracts standardized business terminology based on this.

[0063] In another implementation, when supplementary information includes data association information, the large language model not only explicitly labels the role of the data table by combining quantitative indicators in the data association information (such as the number of downstream dependencies, the complexity of upstream sources, etc.), but also deeply analyzes the dynamic context of the data table in the overall data flow, i.e., the upstream and downstream lineage. This dynamic context is a key dimension for understanding the business meaning and importance of the data table: upstream data tables reveal the "data source" and "original context" of the current target data table, explaining how the data is cleaned and transformed from the business source, thereby helping the model judge the credibility and timeliness of the data; downstream data tables reflect the "data destination" and "consumption scenario" of the current target data table, showing which key reports, algorithm models, or business processes the table supports, thus directly reflecting its actual value and influence in the enterprise's data assets.

[0064] Specifically, large language models can construct a complete data link view by analyzing these upstream and downstream relationships. For example, if a target data table is detected to be directly connected to the core transaction log system upstream, the model will generate a description in the enhanced description information stating that "this table originates from real-time transaction logs and retains the most granular user behavior records," clarifying its attribute as a source fact table. If the table is detected to be relied upon by dozens of downstream high-frequency query tasks or core risk control models, the model will infer that it is located at the hub of the data link and generate the conclusion that "this table is the core middleware layer of the entire link, widely serving real-time dashboard monitoring and anti-fraud decision-making." By introducing this dynamic context based on relationships, the generated enhanced description information is no longer limited to static field definitions but can vividly illustrate the data table's role in connecting upstream and downstream processes in business flow, thereby improving the depth and accuracy of the understanding of the business semantics of the data table.

[0065] The final enhanced descriptive information is text data in natural language form. This enhanced descriptive information includes at least one of the following dimensions: a business scenario summary, used to describe the business domain to which the data table belongs and the specific application scenario of the service; deep semantic content, used to explain the business meaning of key fields, data statistical standards, and data source links; link role identifiers, used to determine the position of the data table in the data flow based on data relationships (such as core fact tables, dimension tables, etc.); and dynamic value indicators, used to generate popularity evaluations or importance classifications based on historical query frequency.

[0066] Through the above processing, the generated enhanced description information not only includes the original field definitions, but also incorporates the business context mined from historical behaviors and relationships, thereby making the implicit knowledge of the data table explicit and improving the richness and accuracy of the semantic representation of the data table.

[0067] In one example, for a target data table A, its enhanced description information could include: This table is the core fact table of the e-commerce transaction domain, mainly recording the actual payment details after a user places an order. The data is widely used in key business scenarios such as 'real-time monitoring of the Double 11 promotion', 'after-sales refund risk analysis', and 'daily financial reconciliation'. The core field `pay_amount` represents the user's actual payment amount (confirmed based on historical query context as the payment amount before deducting red envelopes), and is a key indicator for calculating GMV (Gross Merchandise Volume) trends and financial receipts; the `refund_flag` field is used to quickly filter abnormal refund orders in after-sales risk control scenarios. This table's data is cleaned from the `ods_order_raw` raw logs and is currently strongly depended on by 458 downstream ETL tasks and 120 application tables. It has been actively queried over 2000 times in the past 30 days, making it a Level 1 core asset. Its data latency directly affects the accuracy of the promotional dashboard and financial statements.

[0068] In another example, for a target data table B, the generated enhanced description information may include: This table belongs to the marketing middle platform and is a standard dimension table storing basic information about marketing activities across the entire platform. It primarily serves the product selection and discount configuration scenarios for S-level promotional activities such as 'XX Project' and 'Summer Promotion'. The table records detailed information such as activity ID, name, start and end times, discount type (e.g., full reduction, discount), and associated merchant recruitment scheme ID. Based on historical query records, this table is frequently used by data analysts for 'verifying the list of products registered for activities' and 'calculating the redemption rate of coupons across various channels'. The data source integrates activity configuration information from multiple ticketing suppliers such as XXX and XX. Although this table is updated daily, as the associated master table for all activity reports, it has extremely high downstream referencing breadth (covering 90% of marketing reports), making it a reliable source for identifying activity affiliation and statistically analyzing activity performance.

[0069] The enhanced description information explicitly identifies the business domain to which the table belongs and the specific service scenario, enabling the system to accurately respond to users' fuzzy queries about specific projects. By introducing this information, the system can establish a mapping relationship between data tables and business intents, thereby supporting users to perform precise searches based on fuzzy business needs (such as specific project names or non-technical descriptions), solving the problem of low recall caused by traditional searches relying solely on table names or field names. The description text of the deep semantic content summary not only lists basic fields (such as product ID, name, and price) but also further explains the business meaning of the data, such as indicating that the table data is "integrated from multiple ticketing suppliers" and "associated with investment promotion plans." This description goes beyond the traditional schema definition, revealing the data's source chain and business context. The dynamic value indicators in the enhanced description information quantify the access popularity, query frequency, or degree of dependence of the data table on downstream tasks within a preset time window (such as indicating the number of queries in the past 30 days). This indicator can also serve as a key weighting factor for subsequent search ranking algorithms, allowing core data tables with high frequency of use or high dependence to receive higher priority display in search results, thereby optimizing the user's table search efficiency.

[0070] To further ensure the accuracy of the enhanced description information, the system can introduce a confidence assessment mechanism after generating the description. If the business semantics output by the large model conflict with the existing metadata tags, or if the confidence level given by the model itself is lower than a preset threshold, the record will be marked as 'pending review' and pushed to the data governance platform for administrator confirmation; alternatively, the system will dynamically correct the description information based on user clicks and acceptance of search results in subsequent user query feedback loops.

[0071] In step 103, the enhanced description information is vectorized to obtain a vector representation, and a vector index library corresponding to multiple target data tables is constructed based on the vector representation.

[0072] This step utilizes a pre-trained text embedding model to convert the natural language text rich in business semantics generated in step 102 into a vector representation. Specifically, the system can select an embedding model suitable for database domain terminology as the vectorization engine. This model has been trained on a massive corpus and can capture deep semantic relationships in the text, such as recognizing the high semantic similarity between "sales revenue" and "transaction records," even though their literal expressions are completely different.

[0073] In some implementations, the general embedding model can be fine-tuned using specialized corpora containing database schemas, SQL statements, and data dictionaries to further improve its feature extraction accuracy for data table description information. By inputting the enhanced description information of each target data table into the model, and performing word segmentation, context encoding, and pooling operations, a fixed-dimensional array of real numbers is output as the vector representation of the data table. The geometric position of this vector in multidimensional space is uniquely determined by the semantic content of the enhanced description information, ensuring that data tables with more similar semantics have closer vectors in space (e.g., cosine distance), thus achieving a digital and continuous representation of the business meaning of the data tables.

[0074] After obtaining the vector representations of multiple target data tables, the system stores these vectors and their corresponding metadata identifiers (such as table ID, table name, physical storage path, etc.) together in a vector database or a dedicated index structure to form a vector index library. To support efficient retrieval of large-scale data tables, this embodiment can use the Approximate Nearest Neighbor (ANN) algorithm to organize these vectors.

[0075] The specific index structure can be flexibly selected according to the data scale. For example, a cluster-based inverted file index can be used to divide the vector space into multiple clusters to accelerate the search, or a graph-based hierarchical navigation small-world algorithm can be used to build a multi-level connected graph structure to achieve extremely fast query speed while ensuring high recall, which is particularly suitable for high-dimensional vector scenarios.

[0076] During the construction process, the system establishes a mapping relationship between each vector and the metadata of the original data table. This ensures that when a user query request is received later, the system only needs to convert the user's query intent into a vector to quickly locate one or more target data tables in the index that are semantically closest to it and return the corresponding data table information.

[0077] Considering that the business meaning, usage frequency, and relationships of a data table may dynamically change over time, leading to changes in its enhanced description information, this embodiment also provides a dynamic update and maintenance mechanism for the index. When changes are detected in the metadata of the target data table, the accumulation of new high-frequency historical query records, or significant adjustments to its upstream and downstream dependencies, the system automatically triggers a recalculation process, re-executing the aforementioned steps to generate the latest enhanced description information and the corresponding new vector representation. Subsequently, the system locates the old vector entry corresponding to the data table in the vector index library, replaces it with the new vector, or marks the old vector as invalid and inserts the new vector. This dynamic maintenance mechanism ensures that the vector index library always reflects the latest business status of the data table, avoids retrieval errors caused by index lag, and guarantees the timeliness and accuracy of table lookup results.

[0078] Through the aforementioned vectorization and indexing processes, traditional keyword matching retrieval is upgraded to semantic understanding-based vector retrieval, effectively solving the problem of word mismatch. Users no longer need to remember precise table or field names; even when using colloquial or vague business descriptions for queries, the system can accurately match semantically relevant target data tables through the semantic proximity of the vector space. Furthermore, because vector representation incorporates dynamic information such as historical query records and data association information, the retrieval results are not only semantically relevant but also tend to prioritize displaying core data tables with higher business value and more frequent use, significantly reducing the time cost for data developers to filter invalid tables. Simultaneously, the vectorization process uniformly encodes scattered metadata, lineage relationships, and historical query records into the same vector space, enabling the system to discover implicit cross-domain relationships and automatically recommend data tables with different names but highly reusable business logic, thereby comprehensively improving the intelligence level of data retrieval and user experience.

[0079] After the index is built, the following section will explain the process of querying data tables using the vector index library built above. Compared to retrieval methods based on keywords or simple semantic matching, which are difficult to understand complex business logic and cannot accurately respond to abstract query requirements (such as "finding the core table for calculating GMV"), this embodiment introduces the deep semantic understanding capabilities of a large language model and a multi-source information fusion mechanism to achieve a query that moves from "text similarity" to "intent matching".

[0080] Figure 2 This is a flowchart of a method for querying a data table according to an embodiment of this specification. This method can be applied to any device, platform, or device cluster with computing and processing capabilities. The method includes steps 201-203 as shown below.

[0081] In step 201, the user query from the target user is received, and the user query is vectorized to obtain a query vector.

[0082] In this step, the user query input by the target user can be a description in natural language. The user query contains a description of the user's needs for the required data table, such as "query the core fact table used to calculate the total transaction amount of goods during Double Eleven" or "find the table containing customer risk scores that is frequently called by the risk control model". The system first preprocesses the received user query, including cleaning, word segmentation, and standardization operations. Then, using the same or compatible pre-trained embedding model as the index building stage, the processed user query is mapped to a high-dimensional vector space to generate a query vector representing the semantic intent of the user query. This query vector can capture the deep business meaning in the user query, accurately expressing the user's needs for the business functions, application scenarios, and importance of the data table even if the specific table name or field name does not appear in the query.

[0083] In step 202, multiple candidate vector representations that match the query vector are retrieved from the vector index.

[0084] The vector index library is obtained through the indexing method of the data table in any of the above embodiments. In this index library, each candidate vector representation corresponds to an enhanced description of a data table. This embodiment does not limit the specific recall method used, such as approximate nearest neighbor search, exact nearest neighbor search, hybrid retrieval (combining keywords and vectors), vector retrieval based on metadata filtering, or hierarchical retrieval based on clustering. For example, the system can use an approximate nearest neighbor search algorithm to calculate the similarity (such as cosine similarity) between the query vector and each candidate vector representation in the index library, and quickly recall the Top-K candidate vector representations with the highest similarity. This stage ensures that the retrieval results are highly relevant to the user query at the semantic level, reducing the problem of missed detections caused by non-standard table names or differences in field naming.

[0085] In step 203, for each candidate vector representation of enhanced description information, the score of the data table corresponding to the candidate vector representation is calculated based on the relevance between the user query and the enhanced description information.

[0086] Specifically, since the candidate vectors recalled in step 202 are based on coarse-ranked results obtained through recall methods such as approximate nearest neighbor search, there may be some semantically similar vectors that do not fully match the business intent. Therefore, this step introduces a more refined relevance calculation mechanism to further improve the accuracy of the ranking.

[0087] This embodiment does not limit the method of calculating relevance. For example, the system can extract the complete enhanced description information corresponding to each candidate vector representation and input this information together with the original user query text into a pre-trained language model or cross-encoder. The model deeply analyzes the degree of matching between the business intent in the user query (such as specific statistical standards, business scenario constraints, etc.) and the business semantics (such as data source, field meaning, applicable scenarios, etc.) contained in the enhanced description information.

[0088] Unlike the rough estimation based on geometric distance in vector space in step 202, the calculation process in this step can capture deeper semantic logical connections. For example, when a user query contains words that imply importance judgments, such as "core" or "main source," the model will focus on evaluating whether the enhanced description information contains business semantic descriptions representing the key position of the data table; when a user query involves specific business metrics (such as "GMV"), the model will verify whether the explanation of the calculation logic of that metric in the enhanced description information is consistent with the query intent. Figure 1 Finally, a quantitative score is output.

[0089] In this process, the model not only focuses on the semantic overlap on the surface of the text, but also makes a comprehensive judgment based on multi-dimensional business characteristics: for example, for "detail table" type requirements, the model will identify whether the table name contains standard identifiers such as "detail"; for "core table" type requirements, the importance of the table is quantified based on the frequency of its reference in the business chain.

[0090] Finally, based on the matching results of the deep semantic logic and business rules mentioned above, the model outputs a quantitative relevance score, thereby accurately selecting data tables that match the user's true intent.

[0091] Existing retrieval solutions typically treat data tables as isolated entities, paying little attention to their dynamic context within the data processing flow. In data lineage, the upstream sources and downstream dependencies of a data table often implicitly reveal its importance and business purpose. For example, a data table frequently referenced by multiple core downstream tasks is generally more important and accurate than one referenced only by a few temporary tasks. Furthermore, users' query patterns throughout their historical usage often reveal their query needs. Current technologies generally lack effective mechanisms for integrating this multi-source heterogeneous information, failing to incorporate dynamic signals such as data lineage, reference frequency, and user historical behavior into the retrieval ranking process.

[0092] To further improve the accuracy and interpretability of search results, this embodiment introduces a multi-dimensional re-ranking mechanism. Traditional retrieval often relies solely on geometric distance in the vector space, while this embodiment constructs a comprehensive scoring model by explicitly fusing multiple signals such as data lineage and user historical preferences. Specifically, the scoring calculation process may include one or more of the following implementation methods:

[0093] In one implementation, the system acquires the target user's historical demand patterns. These patterns are feature vectors or rule sets mined from the user's past query records, accessed data table types, and business domain preferences, representing the user's specific demand tendencies in historical queries. For each candidate vector representation, the system calculates the first relevance between the user's query and the enhanced description information, and the second relevance between the historical demand pattern and the enhanced description information. The final score is obtained by a weighted fusion of these two. The weighting coefficients can be dynamically adjusted according to the query type; for fuzzy queries, semantic relevance is emphasized, while for queries explicitly seeking the 'core table,' the weight of upstream and downstream reference information is increased. For example, if a user has long focused on "marketing domain" data, when performing a fuzzy query for "activity performance table," the system will give higher scores to candidate tables belonging to the marketing domain and historically frequently accessed by the user, thus achieving personalized table recommendations. The system analyzes the user's individual historical query frequency, personalizes the weighting of frequently used tables, and improves their sorting priority.

[0094] In another implementation, the system obtains the query frequency of the corresponding data table for each candidate vector representation. The query frequency reflects the popularity of that data table among global users or a specific user group within a preset time window. The system calculates a comprehensive score based on the relevance of the user query to the enhanced description information and the query frequency. This mechanism can be used to distinguish between tables with similar functions but different usage scopes (e.g., official tables vs. test tables). Under this mechanism, when multiple candidate tables have similar semantic similarity to the user query, those widely used, more validated, high-popularity core tables or official tables will receive higher ranking weights. This ensures that search results prioritize displaying high-value, high-credibility data assets, avoiding the dilemma of users being trapped in the selection process of low-frequency tables, test tables, or obsolete tables.

[0095] In another implementation, for each candidate vector representation, the system obtains the upstream and downstream reference information of its corresponding data table based on data association information. The upstream and downstream reference information quantifies the position and influence of the data table in the data flow, including the number of upstream data tables it depends on (reflecting the complexity of the data source) and the number of downstream data tables it depends on (reflecting the importance of the data output). The system calculates a score by combining the relevance of the user query and the enhanced description information with the upstream and downstream reference information. The upstream and downstream reference information can be used as a dynamic weighting coefficient or as a direct bonus factor incorporated into the final result.

[0096] Specifically, when used as a weight, the system amplifies the basic semantic relevance score based on the number of downstream data tables it depends on, giving high-popularity tables higher confidence in the ranking. When used as a direct bonus, the system pre-defines feature-based scoring rules based on graph topology. In particular, when user queries contain words implying importance such as "core," "foundation," and "source," data tables with large downstream dependency networks (i.e., high in-degree) will be given significant bonus weights or have fixed scores directly added. Conversely, when user queries contain words such as "details," "recent," "debugging," and "specific cases," the system increases the weights of "data freshness" and "field richness" while reducing the global popularity weight to avoid over-recommending outdated, wide tables and ignoring recently created dedicated detail tables. This mechanism enables the system to intelligently identify and prioritize key tables located at the data link hub, responding to abstract business needs such as "finding the core table."

[0097] In another implementation, the system constructs a comprehensive scoring model that deeply integrates the relevance of user queries to enhanced description information, relevance to historical demand patterns, data table query frequency, and upstream and downstream reference information to calculate the final candidate vector representation score. Specifically, the system first quantifies the scores of the above four dimensions: first, a basic score based on the semantic relevance of user queries to enhanced description information output by a pre-trained model; second, a personalized preference score based on the degree of matching between the target user's historical demand patterns and enhanced description information; third, a query frequency score reflecting the global or group popularity of the data table; and fourth, an upstream and downstream reference information score characterizing the pivotal position of the data table in the data flow (including upstream dependency complexity and downstream dependency importance). Subsequently, the system aggregates the scores of these four dimensions into a final comprehensive score through a weighted summation or nonlinear fusion algorithm. For example, when a user initiates a query, the system will not only prioritize displaying the semantically most matching table, but also assign extremely high comprehensive weights to data tables that simultaneously meet the criteria of "high historical access by the user," "high global query popularity," and "being at the core node of the data link (high downstream in-degree)."

[0098] Furthermore, in one implementation, such as Figure 3As shown, the aforementioned multi-dimensional signals (such as historical demand patterns and upstream / downstream reference information) can be uniformly input into a large language model re-ranking module. This module no longer performs simple linear weighting; instead, it uses the original user query, complete enhanced descriptions of candidate tables, and various quantitative indicators as contextual prompts. Leveraging the reasoning capabilities of the large language model, it integrates multiple sources of information to comprehensively assess the relevance and importance of the tables, assigning scores to each, and then re-ranking them based on the combined scores. The large language model can understand the logical relationships between complex concepts such as "core tables," "detail tables," and "business definitions," simulating the decision-making process of a senior data architect and outputting a final relevance score that better aligns with business logic.

[0099] It should be noted that the solution in this manual is optimized for two types of abstract query requirements: "business terminology mapping" and "dynamic value assessment".

[0100] In terms of business terminology mapping, taking the search for "GMV" as an example, the deep reasoning capabilities of large models can be leveraged to deeply scan the logical code layer of historical queries during the index building phase. When it is detected that a certain data table is used in multiple historical SQL queries for specific calculation patterns such as `SUM(pay_amount - refund_amount)`, even if the field names of the table do not directly contain the word "gmv", the large model will automatically inject the semantic tag "supports the calculation of total merchandise transaction volume (GMV)" into the generated enhanced description. This enables vector retrieval to break through the traditional literal matching limitations and achieve accurate recall based on user intent.

[0101] In terms of dynamic value assessment, taking the search for "core tables" as an example, a "lineage heat coefficient" can be introduced during the re-ranking stage. This coefficient is dynamically calculated by combining the number of downstream dependencies in the data association information and the execution frequency of historical query information. When a user's query contains words that imply importance, such as "core," "main," and "basic," the system will automatically increase the score weight of data tables that have a large downstream dependency network (i.e., high in-degree) and high recent query frequency. Conversely, for data tables that are accessed infrequently or marked as "temporary" or "test," even if their semantic similarity is high, their final ranking will be reduced accordingly. This ensures that the recommendation results are not only semantically relevant but also in line with actual business value and application scenarios.

[0102] In step 204, based on the score, the target vector representation is selected from multiple candidate vector representations, and the data table corresponding to the target vector representation is determined as the query result.

[0103] Specifically, the system can sort the scores of all candidate vector representations calculated in step 203 in descending order to construct a candidate list sorted by relevance from high to low. Then, the system selects the target vector representation from this list according to a preset filtering strategy. The filtering strategy can be implemented in various ways:

[0104] In one implementation, a fixed-quantity truncation method can be used. This involves pre-setting an upper limit on the number of returned results (Top-K, e.g., K=5 or K=10), and directly selecting the top K candidate vectors with the highest scores from the sorted list as the target vector representation. This approach is suitable for scenarios with limited user interface space or where a faster presentation of optimal solutions is required, ensuring that users see the data table with the highest matching degree first.

[0105] In another implementation, a dynamic threshold filtering method can be used. A minimum relevance score threshold is set, and only candidate vectors with scores higher than this threshold are retained as target vectors. If the highest score in the sorted list is still below this threshold, the current recall results are deemed insufficient to meet the user's intent, and the system can trigger a "no results" feedback or automatically expand the search scope for a secondary recall. This mechanism effectively filters out noisy data that, despite re-sorting, still represents a low-quality match, ensuring the confidence level of the query results.

[0106] In another implementation, a hybrid hierarchical screening method can be used. The system divides the scores into different levels such as "strongly correlated" and "weakly correlated," prioritizing the selection of all candidate vectors within the "strongly correlated" interval. If the number of vectors in this interval is insufficient, it is extended to the "weakly correlated" interval to supplement the required number. Furthermore, if there are cases of identical scores, the system can introduce a secondary sorting key (such as the update time of the data table, the frequency of global queries, or the depth of upstream and downstream references) for secondary sorting to break ties and ensure the determinism of the screening results.

[0107] Finally, the selected target vectors are mapped back to their original corresponding enhanced descriptions or metadata, and then packaged into standard query results and returned to the user. Through this process, the system completes end-to-end optimization from coarse-grained retrieval and fine-grained scoring to final decision-making, ensuring that what is delivered to the user is not only semantically similar data tables, but also assets with the highest degree of business intent matching and that meet the user's personalized needs.

[0108] The solution presented in this specification employs a two-stage architecture of "index building – querying," integrating the deep semantic understanding capabilities of large language models with the structured information of data lineage graphs. This effectively overcomes the limitations of traditional embedding models, which rely solely on literal matching or simple vector distance. The system can deeply parse abstract concepts such as "core tables," "detail tables," and specific "business definitions," achieving a leap from surface-level semantic similarity to deep business intent matching. For example, when a user searches for "GMV," even if the field names of candidate data tables do not directly contain the term, the system can accurately recall the data table actually used to calculate GMV by leveraging deep reasoning about the calculation logic and business scenario in the enhanced description. This significantly improves the accuracy of table retrieval even with incomplete metadata.

[0109] At the ranking decision level, this solution is based on a "data value map" fused with multi-dimensional signals. During the re-ranking stage, it explicitly integrates multiple high-value signals, such as upstream and downstream relationships, global query frequency, and user historical behavior preferences. This ensures that the ranking results not only conform to semantic logic but also possess business value and scenario relevance. Specifically, the system can dynamically adjust its strategy based on the query context: when faced with needs such as "finding core tables," it automatically increases the weight of data tables with high downstream dependency networks (high in-degree); when faced with personalized scenarios, it prioritizes recommending assets that match the user's historical habits. This mechanism ensures that the final results presented are high-value data that has been validated by the business, effectively distinguishing data tables with similar functions but different importance (such as official tables and test tables).

[0110] Furthermore, this solution fundamentally solves the problems of low-quality and rudimentary metadata in data warehouses by automatically generating high-quality enhanced descriptive information. The system utilizes a large language model to comprehensively analyze the schema structure, dependency chains, and references of data tables, transforming unstructured business logic into standardized semantic information, laying a solid foundation for accurate retrieval. This technical approach lowers the barrier to data use, enabling novice developers without deep business backgrounds to quickly locate required assets through natural language, reducing reliance on experienced personnel and significantly improving overall data development efficiency.

[0111] Figure 4 This is a schematic diagram of the index building device for the data table in the embodiments of this specification. This device can be applied to any device, platform, or device cluster with computing and processing capabilities. The device includes:

[0112] The acquisition module 41 is used to acquire metadata information and supplementary information of the target data table to be processed. The supplementary information includes data association information between the target data table and other data tables and / or historical query information. The historical query information includes historical query records executed by the user on the target data table.

[0113] Analysis module 42 is used to perform semantic analysis on metadata and supplementary information based on the large model, and generate enhanced descriptive information to represent the business semantics of the target data table.

[0114] Module 43 is used to vectorize the enhanced description information to obtain a vector representation, and to build a vector index library corresponding to multiple target data tables based on the vector representation.

[0115] In some embodiments, the process of obtaining historical query information includes: receiving a data query request input by a user, the data query request being used to query target data in a target data table; transforming the data query request into an executable query statement based on a large model, and during the generation of the query statement, prompting the large model to perform semantic parsing of the data query request and inserting annotation fields containing business semantics into the query statement through a preset prompt word strategy; executing the query statement, and generating historical query information for the data table based on the execution result and the query statement.

[0116] In some embodiments, the annotation fields include at least one of the following: a description of the business to which the query belongs, used to identify the business scenario to which the query belongs; a description of the user's requirements, used to explain the purpose of the user initiating the query; and a description of the purpose of the data table, used to explain the purpose of the target data table being queried in the current query scenario.

[0117] In some embodiments, data association information is used to characterize the upstream source relationship and downstream dependency relationship between the target data table and other data tables.

[0118] Figure 5 This is a schematic diagram of the device structure for querying a data table in the embodiments of this specification. This device can be applied to any device, platform, or device cluster with computing and processing capabilities. The device includes:

[0119] The receiving module 51 is used to receive user queries from the target user and vectorize the user queries to obtain query vectors.

[0120] The recall module 52 is used to recall multiple candidate vector representations that match the query vector in the vector index library. The vector index library is constructed using the index construction method of the data table in any of the above embodiments, and each candidate vector representation corresponds to the enhanced description information of a data table.

[0121] The scoring module 53 is used to calculate the score of the data table corresponding to each candidate vector representation based on the relevance between the user query and the enhanced description information.

[0122] The filtering module 54 is used to filter the target vector representation from multiple candidate vector representations based on the score, and determine the data table corresponding to the target vector representation as the query result.

[0123] In some embodiments, the scoring module 53 is specifically used to obtain the historical demand patterns of the target user, which represent the target user's demand in historical queries; for each candidate vector representation of enhanced description information, the score of the data table corresponding to the candidate vector representation is calculated based on the correlation between the historical demand patterns and the enhanced description information, as well as the correlation between the user query and the enhanced description information.

[0124] In some embodiments, the scoring module 53 is specifically used to obtain the query frequency of the data table corresponding to each candidate vector representation, whereby the query frequency is used to indicate the frequency at which the data table is queried; and to calculate the score of the data table corresponding to the candidate vector representation based on the relevance between the user query and the enhanced description information and the citation frequency.

[0125] In some embodiments, the scoring module 53 is specifically used to, for each candidate vector representation, obtain the upstream and downstream reference information of the data table corresponding to the candidate vector representation based on data association information, wherein the upstream and downstream reference information is used to indicate the number of upstream data tables and / or downstream data tables that the data table depends on; and calculate the score of the data table corresponding to the candidate vector representation based on the relevance between the user query and the enhanced description information and the upstream and downstream reference information.

[0126] This specification also provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed in a computer, it causes the computer to perform the method described in any of the above embodiments.

[0127] This specification also provides a computing device, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements the method described in any of the above embodiments.

[0128] This specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the method described in any of the above embodiments.

[0129] In some cases, the actions or steps described in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0130] Those skilled in the art will understand that one or more embodiments of this specification can be provided as a method, system, or computer program product. Therefore, one or more embodiments of this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, one or more embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0131] One or more embodiments of this specification can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a particular task or implement a particular abstract data type. One or more embodiments of this specification can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0132] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, system embodiments are basically similar to method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. In the description of this specification, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this specification. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described can be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0133] The above description is merely an embodiment of one or more embodiments of this specification and is not intended to limit the scope of these embodiments. Various modifications and variations can be made to these embodiments by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims.

Claims

1. A method for constructing an index for a data table, the method comprising: Obtain metadata and supplementary information of the target data table to be processed. The supplementary information includes data association information between the target data table and other data tables and / or historical query information. The historical query information includes historical query records executed by the user on the target data table. Based on the large model, semantic analysis is performed on the metadata information and the supplementary information to generate enhanced description information that characterizes the business semantics of the target data table; The enhanced description information is vectorized to obtain a vector representation, and a vector index library corresponding to multiple target data tables is constructed based on the vector representation.

2. The method according to claim 1, wherein, The process of obtaining the historical query information includes: Receive data query requests input by users, the data query requests being used to query target data in a target data table; Based on the large model, the data query requirements are transformed into executable query statements. During the generation of the query statements, a preset prompt word strategy prompts the large model to perform semantic parsing of the data query requirements and insert annotation fields containing business semantics into the query statements. Execute the query statement, and based on the execution result and the query statement, generate historical query information for the data table.

3. The method according to claim 2, wherein, The annotation field must include at least one of the following: The relevant business description identifies the business scenario to which this query belongs. User requirement description, which explains the purpose of the user's query; The purpose of the data table is explained, which describes the purpose of the target data table in this query scenario.

4. The method according to claim 1, wherein, The data association information is used to characterize the upstream source relationship and downstream dependency relationship between the target data table and other data tables.

5. A method for querying a data table, the method comprising: Receive user queries from target users and vectorize the user queries to obtain query vectors; Retrieve multiple candidate vector representations that match the query vector in a vector index library, wherein the vector index library is constructed by the method described in any one of claims 1-4, and each candidate vector representation corresponds to enhanced description information of a data table; For each candidate vector representation of enhanced description information, a score for the data table corresponding to the candidate vector representation is calculated based on the relevance between the user query and the enhanced description information. Based on the score, a target vector representation is selected from the multiple candidate vector representations, and the data table corresponding to the target vector representation is determined as the query result.

6. The method according to claim 5, wherein, For each candidate vector representation of enhanced description information, based on the relevance between the user query and the enhanced description information, a score for the data table corresponding to the candidate vector representation is calculated, including: Obtain the historical demand patterns of the target user, which represent the demands of the target user in historical queries; For each candidate vector representation of enhanced description information, a score for the data table corresponding to the candidate vector representation is calculated based on the correlation between the historical demand pattern and the enhanced description information, and the correlation between the user query and the enhanced description information.

7. The method according to claim 5, wherein, For each candidate vector representation of enhanced description information, based on the relevance between the user query and the enhanced description information, a score for the data table corresponding to the candidate vector representation is calculated, including: For each candidate vector representation, the query frequency of the data table corresponding to the candidate vector representation is obtained, and the query frequency is used to represent the frequency of the data table being queried; Based on the relevance between the user query and the enhanced description information, as well as the reference frequency, the score of the data table corresponding to the candidate vector is calculated.

8. The method according to claim 5, wherein, For each candidate vector representation of enhanced description information, based on the relevance between the user query and the enhanced description information, a score for the data table corresponding to the candidate vector representation is calculated, including: For each candidate vector representation, the upstream and downstream reference information of the data table corresponding to the candidate vector representation is obtained based on the data association information. The upstream and downstream reference information is used to indicate the number of upstream data tables that the data table depends on and / or the number of downstream data tables that the data table depends on. Based on the correlation between the user query and the enhanced description information, as well as the upstream and downstream reference information, the score of the data table corresponding to the candidate vector is calculated.

9. An index building apparatus for a data table, the apparatus comprising: The acquisition module is used to acquire metadata information and supplementary information of the target data table to be processed. The supplementary information includes data association information between the target data table and other data tables and / or historical query information. The historical query information includes historical query records executed by the user on the target data table. The analysis module is used to perform semantic analysis on the metadata information and the supplementary information based on the large model, and generate enhanced description information to characterize the business semantics of the target data table; The construction module is used to vectorize the enhanced description information to obtain a vector representation, and to construct a vector index library corresponding to multiple target data tables based on the vector representation.

10. An apparatus for querying a data table, the apparatus comprising: The receiving module is used to receive user queries from target users and vectorize the user queries to obtain query vectors. The recall module is used to recall multiple candidate vector representations that match the query vector in a vector index library, wherein the vector index library is constructed by the method described in any one of claims 1-4, and each candidate vector representation corresponds to the enhanced description information of a data table. The scoring module is used to calculate the score of the data table corresponding to each candidate vector representation based on the relevance between the user query and the enhanced description information for each candidate vector representation; The filtering module is used to filter out the target vector representation from the multiple candidate vector representations based on the score, and determine the data table corresponding to the target vector representation as the query result.

11. A computing device comprising a memory and a processor, wherein the memory stores executable code, and the processor, when executing the executable code, implements the method of any one of claims 1-8.

Images