Method and system for automatically discovering association relationship of data lake tables

Abstract

The invention relates to the technical field of databases and artificial intelligence, and discloses a data lake table association relationship automatic discovery method and system, and the method comprises the steps: generating a table description and a column description of a new table through automatic reasoning based on a knowledge base and a language model; calculating a semantic similarity and a value overlapping distance between the query column and a target column of any table in the data lake; screening out a candidate column associated with the query column based on the semantic similarity and the value overlapping distance, and considering that an association relationship exists between the candidate column and the query column; and constructing a metadata graph based on the table description, the column description, the affiliation relationship between the table and the column and the discovered association relationship between the columns, and performing multi-hop reasoning by calculating a pass closure to derive the potential association relationship between the columns. According to the method, through knowledge-enhanced metadata construction, fusion of semantic features and value overlapping features, and a mapping metadata organization and graph reasoning technology, automatic inference of the connection relation between tables in a data lake is achieved.

Description

Technical Field

[0001] This invention relates to the fields of database and artificial intelligence technology, specifically to a method and system for automatically discovering relationships in a data lake table. Background Technology

[0002] In modern data analytics and intelligent question-answering systems, data lakes are increasingly becoming the primary storage method for structured and semi-structured data within enterprise organizations. Compared to traditional database management systems (DBMS), data lakes emphasize the direct storage of raw data, lacking strict schema constraints and robust metadata management mechanisms. Consequently, information such as semantic relationships, table descriptions, and associations between tables in a data lake environment is often incomplete.

[0003] In this context, while agents can obtain basic structural information about data tables (such as table headers, value types, and value examples), they struggle to directly understand the relationships between different tables. This lack of information severely impacts automated data analysis tasks such as Natural Language to SQL (NL2SQL). The absence of metadata detailing relationships prevents models from determining which fields can be joined, leading to incomplete SQL generation, query failures, or inconsistent results. This has become a critical bottleneck for NL2SQL technology in data lake scenarios.

[0004] Given the high dependence of NL2SQL tasks on table join (JOIN) inference, the following discussion focuses on the importance and challenges of automatically discovering table join (JOIN) type relationships in a data lake environment.

[0005] Discovering table join relationships in a data lake faces the following typical problems: (1) Lack of foreign key and constraint information: Data tables from different sources usually do not have explicit primary and foreign key constraints, making it difficult to infer the connection method directly through structured constraints.

[0006] (2) Inconsistent field naming and semantics: Different data sources have vastly different naming styles for table headers, making it difficult to determine the matching relationship based solely on field names.

[0007] (3) Value range overlap and complex statistical features: There may be partial value overlap, fuzzy overlap, and different types of mapping between fields, which increases the difficulty of automatic association.

[0008] (4) High computational cost of large-scale data: Data lakes are huge and may contain hundreds of thousands of tables and millions of fields. Exhaustive comparison is too costly and requires efficient filtering, indexing and inference mechanisms.

[0009] Existing table join discovery techniques can be mainly divided into two categories: set-based methods and semantic-based methods.

[0010] (1) Set-based methods: If two columns can be used as join keys, their value sets should have a high degree of overlap. Set-based methods infer possible joinable column pairs by calculating set similarity or set containment. For example, JOSIE measures the overlap of value ranges by evaluating the set similarity between two columns, introduces a cost model for candidate filtering, and utilizes inverted indexes to improve the efficiency of large-scale searches. LSH Ensemble, on the other hand, builds an index based on MinHash-LSH, determines the potential relationship between columns by estimating the set containment score, and returns columns as joinable pairs when the score exceeds a set threshold.

[0011] (2) Semantic-based methods: If two columns can serve as a join key, their field names, column value text, and contextual information should have high semantic similarity. Semantic-based methods discover joinable column pairs by embedding and encoding the columns and performing approximate vector search in the embedding space. The main differences lie in the encoding method and index structure. For example, Pexeso uses fastText to represent the text of cells in a column as high-dimensional vectors and combines inverted indexes with a hierarchical grid structure to accelerate retrieval. DeepJoin fine-tunes pre-trained language models (such as DistilBERT and MPNet), inputs a pair of columns into the model to obtain two sets of semantic vectors, calculates cosine similarity, and uses HNSW for efficient similar column search.

[0012] However, both of these methods face limitations in real-world data lake environments. On one hand, real-world data often contains dirty data, noisy fields, or inconsistent formats, making set-based methods unable to accurately identify fuzzy overlaps, thus missing joinable column pairs. On the other hand, when multiple columns belong to similar domains but have different query intents and no real overlap values ​​exist, semantic-based methods are prone to misjudgments due to semantic vector similarity, returning a large number of false join pairs. Furthermore, single-column-level judgments struggle to cover table-level structural information, neglecting the contextual relationships between columns within the table, such as primary key structure, composite key clues, or semantic synergies between fields.

[0013] Therefore, the identification of joinable tables in real-world scenarios still faces the following core challenges: (1) how to accurately capture the semantic similarity of columns; (2) how to robustly assess the degree of overlap of column values; and (3) how to effectively utilize table-level context information to improve the overall accuracy of join relationship inference. Summary of the Invention

[0014] To address the aforementioned technical issues, this invention provides a method and system for automatically discovering relationships between tables in a data lake. By constructing knowledge-enhanced metadata, fusing semantic features and value overlap features, organizing graph-based metadata, and using graph reasoning techniques, the method enables automatic inference of join relationships between tables in a data lake.

[0015] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: In a first aspect, the present invention provides a method for automatically discovering relationships in a data lake table, comprising: Build a searchable knowledge base based on knowledge documents; When a new table is registered in the data lake, the table description and column description of the new table are automatically generated based on the knowledge base and language model. Calculate the semantic similarity and value overlap distance between the query column and the target column of any table in the data lake; wherein the semantic similarity is obtained based on the cosine similarity of the semantic vectors of the query column and the target column; and the value overlap distance is calculated based on the minimum asymmetric inclusion rate of the cell value set of the query column and the cell value set of the target column. Based on semantic similarity and value overlap distance, candidate columns associated with the query column are selected, and it is assumed that there is an association between the candidate columns and the query column; A metadata graph is constructed based on table descriptions, column descriptions, the relationships between tables and columns, and the discovered associations between columns. Multi-hop reasoning is performed by calculating transitive closures to deduce the potential associations between columns.

[0016] In one embodiment, the step of constructing a searchable knowledge base based on knowledge documents specifically includes: Receive and verify external knowledge documents; The external knowledge document is segmented to obtain knowledge fragments; The language model is invoked to generate a knowledge summary for the knowledge fragment; The knowledge fragments and summaries are vectorized and stored in the knowledge base.

[0017] In one embodiment, when a new table is registered in the data lake, automatically generating a table description and column description of the new table based on a knowledge base and language model specifically includes: Extract the header information of the new table, and randomly sample M cell values ​​from each column of the new table as column value examples; Using the header information as the query criteria, relevant knowledge points are retrieved from the knowledge base; The header information, column value examples, and related knowledge points are input into the language model to generate column descriptions and table descriptions that include semantic domains and typical examples.

[0018] In one embodiment, the semantic similarity is obtained based on the cosine similarity of the semantic vectors of the query column and the target column, specifically including: An embedding model is used to embed the column labels and column descriptions of the query column, resulting in a column label vector for the query column. and description vector Weighted concatenation is performed, and the weights are directly reflected in the distance metric by concatenating and scaling the vectors, resulting in the semantic vector of the query column. : ; An embedding model is used to embed the column labels and column descriptions of the target column, resulting in a column label vector for the target column. and description vector Weighted concatenation is performed to obtain the semantic vector of the target column. ; ; Query column and target column semantic similarity , Let be the cosine similarity.

[0019] In one embodiment, the value overlap distance is calculated based on the minimum asymmetric coverage ratio between the set of cell values ​​in the query column and the set of cell values ​​in the target column, specifically including: Let the set of cell values ​​in the query column be . The set of cell values ​​in the target column is The following formula is used to calculate the query column. and target column Value overlap distance : .

[0020] In one embodiment, the step of filtering candidate columns associated with the query column based on semantic similarity and value overlap distance specifically includes: Based on the semantic similarity, the target columns are pre-screened, and the target columns with semantic similarity greater than or equal to the similarity threshold are added to the candidate list; Based on the value overlap distance, the target columns are pre-filtered, and the target columns whose value overlap distance is less than or equal to the distance threshold are added to the candidate list; Calculate the joint score for the target column in the candidate list. : ; For query columns and target column semantic similarity, For query columns and target column The value of overlap distance, These are the weighting coefficients; The target columns in the candidate list are sorted based on the joint score, and the target columns with the highest joint score of a set number are selected as candidate columns. It is assumed that there is a direct correlation between the candidate columns and the query columns.

[0021] In one embodiment, the construction of a metadata graph based on table descriptions, column descriptions, the relationships between tables and columns, and discovered column-to-column associations, followed by multi-hop reasoning through transitive closure calculations to deduce potential associations between columns, specifically includes: Using tables and columns as nodes, table descriptions as node attributes of table nodes, and column descriptions as node attributes of column nodes, the relationship between tables and columns, as well as the association between columns, is used as edges to construct a metadata graph. Perform multi-hop potential relationship reasoning based on transitive closure computation on the metadata graph to discover potential associations.

[0022] Secondly, this invention provides an automatic discovery system for data lake table relationships, comprising: The knowledge management module builds a searchable knowledge base based on knowledge documents; The lake table registration and metadata enhancement module automatically generates table and column descriptions for new tables based on knowledge base and language model when a new table is registered in the data lake. The semantic and value range representation module calculates the semantic similarity and value overlap distance between the query column and the target column of any table in the data lake; wherein, the semantic similarity is obtained based on the cosine similarity of the semantic vectors of the query column and the target column; the value overlap distance is calculated based on the minimum asymmetric inclusion rate of the cell value set of the query column and the cell value set of the target column. The association discovery module filters out candidate columns associated with the query column based on semantic similarity and value overlap distance, and considers that there is an association between the candidate columns and the query column; The graph construction and graph reasoning module constructs a metadata graph based on table descriptions, column descriptions, the relationships between tables and columns, and the discovered associations between columns. It performs multi-hop reasoning by calculating transitive closures to deduce the potential associations between columns.

[0023] In one embodiment, the step of constructing a searchable knowledge base based on knowledge documents specifically includes: Receive and verify external knowledge documents; The external knowledge document is segmented to obtain knowledge fragments; The language model is invoked to generate a knowledge summary for the knowledge fragment; The knowledge fragments and summaries are vectorized and stored in the knowledge base.

[0024] In one embodiment, when a new table is registered in the data lake, automatically generating a table description and column description of the new table based on a knowledge base and language model specifically includes: Extract the header information of the new table, and randomly sample M cell values ​​from each column of the new table as column value examples; Using the header information as the query criteria, relevant knowledge points are retrieved from the knowledge base; The header information, column value examples, and related knowledge points are input into the language model to generate column descriptions and table descriptions that include semantic domains and typical examples.

[0025] The system and method in this invention correspond to each other; the specific technical solutions applicable to the method are also applicable to the system.

[0026] Compared with the prior art, the beneficial technical effects of the present invention are: DABstep is a multi-step inference NL2SQL dataset containing various heterogeneous data sources (unstructured external knowledge and structured / semi-structured data tables) and NL2SQL tasks based on real-world business scenarios. This invention was validated on the DABstep dataset, achieving a precision and recall rate of over 87% for association discovery, demonstrating the effectiveness of the proposed method.

[0027] LakeBench is a data benchmark that comprehensively evaluates association discovery methods, containing large-scale data lakes and a ground truth. This invention was validated on the LakeBench OpenData dataset, which contains 13,000 data lakes and 500 query tables. Experiments compared the proposed algorithm with the ground truth in the selection of Top-10, 20, 30, 40, and 50 candidate columns. For the Top-10 candidate columns, the precision reached 66% and the recall 57%; for the Top-50 candidate columns, the precision was 60% and the recall reached 87%. Experimental results show that the proposed association discovery algorithm significantly outperforms other schemes in recall, and its F1 score is stable and significantly higher than other schemes, demonstrating the generalizability of this method.

[0028] High recall ensures that agents receive complete relational information, preventing the omission of important data. After using enhanced metadata, downstream agents showed a significant improvement in accuracy on the NL2SQL task on the DABstep dataset.

[0029] Experimental results show that the method of column description reasoning in this invention, which integrates knowledge enhancement, outperforms methods that focus on independent column semantics or are based on set domains. Attached Figure Description

[0030] Figure 1 This is the overall flowchart of the present invention; Figure 2 This is a flowchart illustrating the knowledge extraction process of this invention. Figure 3 This is a schematic diagram of the enhanced metadata process of the present invention; Figure 4 This is a schematic diagram of the directed metadata graph and graph reasoning of the present invention; Figure 5 This is a schematic diagram of the system application scenario of the present invention. Detailed Implementation

[0031] A preferred embodiment of the present invention will now be described in detail with reference to the accompanying drawings.

[0032] This invention includes the following core processes: Knowledge entry and knowledge enhancement involve receiving external knowledge input from users, generating knowledge summaries, and constructing a searchable knowledge vector library.

[0033] Lake table registration and column description inference: Register new tables in the data lake, automatically infer enhanced metadata information such as table description, column description, and column value type, and associate them with the knowledge base.

[0034] Association filtering: Construct semantic similarity and value overlap distance of column-level semantic vectors, and comprehensively judge the association relationship between columns based on semantic similarity and value overlap distance.

[0035] Graph reasoning enhancement based on metadata graphs: Based on tables and columns, enhance metadata to organize into a graph structure and perform potential connection relationship reasoning.

[0036] For the overall process of this invention, please refer to [link / reference]. Figure 1 The method of the present invention will be described in detail below in several parts.

[0037] 1. Knowledge entry and knowledge enhancement: First, it receives domain knowledge, data descriptions, or task requirements from users, and verifies the file name and format according to string matching methods. It supports uploading documents in Markdown, TXT, Doc / Docx, and JSON formats.

[0038] Then, the document is segmented based on length and delimiters (such as "#"), and a segmentation overlap value is set. Specifically, for very long documents, they are segmented based on chapter identifiers, with different parts overlapping to ensure that the model can understand the context of the fragments when retrieving knowledge later.

[0039] Subsequently, the language model is invoked to generate knowledge summaries from document slices and extract auxiliary information tables. Finally, the knowledge is structured and standardized, and the original knowledge and summaries are vectorized and stored in a value vector database.

[0040] This process creates a knowledge base with retrieval capabilities, providing external semantic supplementation for subsequent metadata enhancement.

[0041] 2. Lake table registration and table description reasoning: When registering a data lake table, the column descriptions and column value types are inferred based on the knowledge base and column values.

[0042] First, this invention automatically extracts the table header information, retrieves relevant knowledge from the knowledge base based on the embedding of the table header information, and uses a vector database algorithm to recall the top k most similar knowledge fragments.

[0043] At the same time, M cell values ​​are randomly sampled from each column as column value examples.

[0044] The retrieved knowledge, column value examples, and table header information are then input into the language model to infer and generate table descriptions, column descriptions, and value types.

[0045] To address the challenge of accurately identifying connectable columns using set-based methods when value domains exhibit fuzzy overlap, this invention requires that the generated column description text explicitly include the semantic domain to which the column values ​​belong and typical examples during the model inference column description process. This semantic description, after vectorization, effectively captures the implicit relationships between value domains, thereby compensating for the limitations of traditional set overlap indices in handling fuzzy or incomplete matching scenarios in terms of semantic similarity.

[0046] The generated enhanced metadata is a semantic supplement to the original data lake, ensuring semantic alignment between different data sources and improving the accuracy of subsequent relationship discovery.

[0047] 3. Semantic and value range filtering of candidate related columns: Define the query column as Data Lake Tables The set is The present invention calculates the query column. For any target column in the data lake , semantic similarity Sum overlap distance And based on similarity threshold, distance threshold and joint score Filter candidate columns.

[0048] (1) Semantic similarity: This invention employs an embedding model to embed column labels and column descriptions, resulting in a semantic vector for each column. From label vector and description vector The weighted splicing is obtained.

[0049] ; Let the semantic vector of the query column be . The semantic vector of the target column is The semantic similarity between the query column and the target column is defined by the cosine distance: .

[0050] (2) Value overlap similarity: Let the set of cell values ​​in the query column be... The set of cell values ​​in the target column is The overlap distance between the query column and the target column is measured using the following formula: ; This distance metric retains the ability of traditional Jaccard similarity to characterize explicit overlapping instances, while effectively handling scenarios where the intersection of column values ​​is small but the union size is too large, or where there is a significant inclusion relationship, which causes the Jaccard metric to fail, thus more accurately identifying the Top-k overlapping candidates of the target column.

[0051] 4. Candidate Column Filtering: (1) Semantic pre-screening: This invention employs an HNSW (Hierarchical Navigable SmallWorld) structure when constructing the semantic vector index. It uses a hierarchical graph structure with a maximum number of layers L. The top layer has fewer nodes, while the bottom layer contains all nodes. Each layer consists of several nodes (vectors) and connecting edges. Each node is assigned to a different layer based on a random layer function. .

[0052] in Uniformly random numbers, It is a hyperparameter that controls the sparsity of hierarchical distribution.

[0053] The HNSW query process can be divided into two stages: coarse search and fine search. The coarse search stage starts from the top level and performs a self-defined search to quickly locate the node with the closest domain query vector in the current level. The fine search stage performs higher-precision domain expansion and candidate set maintenance at the bottom level until the boundary conditions are met, providing Top-k approximate neighbor results.

[0054] In this invention, a similarity threshold is set when retrieving semantic vectors. Target columns with similarity higher than a threshold are filtered out and added to the candidate list. That is, for the query column. Any target column All of them have: .

[0055] (2) Distance pre-screening: In this invention, a distance threshold is set, and filters are performed when the distance is less than the threshold. The target column is selected and added to the candidate list. That is, for the query column... Any target column All of them have: .

[0056] (3) Joint scoring: The joint score between the two columns is: ; This invention uses joint scores as the sorting criterion, selects multiple candidate columns with the highest joint scores from the candidate list, and assumes that there is a correlation between these candidate columns and the query column.

[0057] 5. Connection reasoning based on metadata graph: This invention constructs the enhanced metadata into a graph, using tables and columns as graph nodes and table-column relationships and associations as graph edges, thus constructing a directed graph (Digraph) based on a network structure, thereby providing a unified metadata knowledge structure.

[0058] set up It is a The Boolean adjacency matrix of a directed graph with n vertices is then A transitive closure is a A Boolean matrix whose elements , if and only if (1) Or (2) from There exists a directed path from the starting point to the destination. Also known as vertex arrive Reachable.

[0059] Based on this, after obtaining the table column associations, the present invention implements a multi-hop latent relation reasoning method based on transitive closure computation, that is, finding the dependent columns according to the directed associations. Departure, all reachable trains and believe There is a potential connection.

[0060] Graph reasoning can compensate for the lack of direct features between columns and discover deep or indirect potential relationships.

[0061] Compared with existing technologies, this invention combines semantic features and value overlap features to comprehensively evaluate table relationships, overcoming the limitations of single-dimensional approaches. The relationship discovery system proposed in this invention, as a core module of a data lake management system, can interface with the data perception module of an intelligent agent. By incorporating retrieval enhancement techniques, it provides structured relational knowledge for upper-level question answering, analysis, and task orchestration.

[0062] This invention proposes an automated association discovery method that integrates value overlap features and semantic similarity features. Through collaborative design at the feature level, inference mechanism, and metadata structure, it overcomes the limitations of existing set-based and semantic-based methods, and improves the ability to identify table join relationships in a data lake environment.

[0063] This invention constructs a knowledge-enhanced metadata representation system, enabling intelligent completion and enhancement of table-level and column-level information, significantly reducing manual maintenance costs. By enhancing and generating table descriptions, column descriptions, and their contextual information, this invention can accurately capture field semantics even when the original data lacks structured metadata, providing richer prior knowledge for subsequent association inferences.

[0064] This invention proposes a novel method for calculating value overlap similarity to overcome the shortcomings of traditional Jaccard similarity in real-world data environments. This method effectively handles scenarios where Jaccard similarity fails, such as small column value intersections but large unions or significant inclusion relationships. Therefore, it more reliably reflects the true degree of overlap between fields and significantly improves recall capabilities based on value range features.

[0065] This invention constructs an enhanced metadata graph structure, organizing tables, columns, and relationships into a unified graph model. Based on this enhanced metadata graph, graph computation and graph reasoning mechanisms can be used to mine potential related columns, supporting multi-hop reasoning for complex table relationships and solving the problem of difficulty in modeling relationships from a single-column perspective.

[0066] In terms of system implementation, the framework of this invention is compatible with various open-source vector databases, supports collaborative retrieval of dense vectors and graph indexes, and enables flexible deployment and horizontal scaling. This design ensures the portability, maintainability, and computational performance of the method, making it suitable for online inference scenarios in large-scale data lakes.

[0067] By employing multimodal feature fusion, knowledge-enhanced reasoning, and graph-based metadata design, this invention achieves high recall and a stable F1 score in table join discovery tasks. It effectively bridges the semantic gap between original data, meets the semantic understanding and automatic reasoning requirements of tasks such as NL2SQL and Data Agent for table relationships in data lakes, and has broad engineering application prospects.

[0068] Example: This embodiment provides a method for discovering relationships in a data lake environment, which includes five stages: knowledge storage, lake table registration, enhanced metadata construction, candidate column screening, and graph reasoning.

[0069] Knowledge entry stage: 1. In this embodiment, as Figure 2 As shown, the system first builds a knowledge base to support subsequent text enhancement and table description reasoning.

[0070] The knowledge base includes: (1) User-provided domain knowledge (e.g., business rules, field descriptions, knowledge graphs, etc.); (2) Summary knowledge generated through the model.

[0071] After receiving the natural language description input by the user, the system summarizes and generalizes the text through a large language model to generate structured knowledge units, and encodes the original knowledge and the summarized knowledge into vector representations respectively.

[0072] In one embodiment, Elasticsearch is chosen as the vector database for the knowledge base.

[0073] The storage structure for text-based knowledge is as follows: ; in For knowledge text, Embedding 1024-dimensional knowledge text. This refers to the source information of the knowledge (including file paths and domain tags).

[0074] The storage structure for graph-based knowledge is as follows: ; ; Regarding node information: Number the nodes in the graph. For node name, Embed a 1024-dimensional vector for the node name. This indicates the attributes contained in the node (including node description, embedding of the description, etc.). This refers to the file source for this node.

[0075] Regarding edge information: and The corresponding node's numerical identifier. For relational types, For the description of the relationship, This refers to the source of the file on this side.

[0076] 2. Lake Surface Registration and Metadata Inference Stage: When a user registers a data lake table with the system, the system performs the following steps: (1) The system extracts basic metadata from the table file, including: table name, column name, column value sampling, identifiable data type, table description, and column description.

[0077] (2) Knowledge filtering and context construction: Use the extracted basic metadata as a query vector In the knowledge base Perform semantic retrieval to obtain the set of knowledge items most relevant to that column. ; The knowledge items obtained The sampled column values ​​are concatenated and written into the inference context to enhance the model input.

[0078] (3) Table description and column description reasoning The system calls the language model to generate table descriptions and column descriptions, as well as column value types, based on "column name + sampled value + related knowledge items".

[0079] To handle overlapping ambiguous values, this embodiment requires that the column description explicitly include: the domain that the column value may correspond to (e.g., ID, place name, date, amount, etc.) and typical value range examples. This ensures that the semantic vector can capture potential entity consistency.

[0080] 3. Semantic and Range Representation Stage: This stage includes the calculation of two dimensions: semantic similarity and value overlap distance.

[0081] (1) Semantic similarity: In one specific embodiment, the Jina-embedding-v3 model is used to embed column labels and column descriptions into 1024-dimensional vectors, and then... , column label vector and column description vector Weighted concatenation into a 2048-dimensional fusion vector : ; Let the semantic vector of the query column be . The semantic vector of the target column is The filtering is performed by calculating the cosine similarity between the query column and the target column: .

[0082] (2) Value overlap similarity: In one specific implementation, the set of query column cell values ​​is extracted. and the set of cell values ​​in the target column Calculate the query column and target column Distance: .

[0083] 4. Candidate Related Column Screening Stage: (1) Semantic similarity screening: The fusion vector constructed in the semantic representation stage Build an HNSW index. For the query column. The system performs a KNN search based on the HNSW index of the vector database: .

[0084] return Cosine similarity greater than or equal to the similarity threshold Candidate column set: .

[0085] (2) Value overlap distance filtering: For query columns and target column Distance threshold is Target columns whose value overlap distance is less than or equal to a distance threshold: .

[0086] (3) Joint scoring: In one specific embodiment, the pre-screened candidate columns are ranked based on their joint scores. Weighting is then applied. Calculate the joint score of the candidate column and the query column: ; The candidate columns are truncated into Top-k related candidate columns, and it is assumed that there is a relationship between these candidate columns and the query columns.

[0087] 5. Enhance the metadata graph construction and potential connection reasoning stages: (1) The enhanced metadata is constructed into a graph, and the storage structure is as follows: ; ; Node information: Number the nodes in the graph. For node types (table or column). This is the node name (table name or column name). Embed a 1024-dimensional vector for the node name. This indicates the attributes contained in the node (including descriptions of tables or columns, and embeddings of the descriptions). This is the path to the file to which this node belongs.

[0088] Regarding edge information: and The corresponding node's numerical identifier. For relation types (column types or association types). This refers to the source of the file on this side.

[0089] (2) Construct a transitive closure of the above graphs to establish relationships, i.e., if the list If a relationship exists, then the Boolean adjacency matrix will be... Set the value to true and derive its transitive closure. Add the newly added true value to the original graph as a potential association; see [link to graph]. Figure 4 .

[0090] 6. Save results: like Figure 5 As shown, the enhanced metadata graph is ultimately stored in a vector database. This result can be directly retrieved and called by the NL2SQL model or Data Agent, realizing the enhancement of metadata and automatic association awareness in the data lake.

[0091] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0092] It should be understood that although the steps in the flowcharts of the accompanying drawings are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the flowcharts of the accompanying drawings may include multiple steps or stages, which are not necessarily completed at the same time, but may be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but may be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0093] Based on the description of the above method embodiments, the present invention also provides a system. The system may be a system that uses software (applications), modules, components, servers, clients, etc., using the methods described in the embodiments of this specification, combined with necessary implementation hardware. Based on the same innovative concept, the systems in one or more embodiments provided in this disclosure are as described in the following embodiments. Since the implementation schemes and methods for solving the problem are similar, the specific system implementations in the embodiments of this specification can refer to the implementations of the foregoing methods, and repeated details will not be repeated. As used below, the terms "module" or "module group" refer to a combination of software and / or hardware capable of implementing a predetermined function. Although the systems described in the following embodiments are preferably implemented in software, hardware implementations, or a combination of software and hardware, are also possible and contemplated.

[0094] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0095] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention, and no reference numerals in the claims should be construed as limiting the scope of the claims.

[0096] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A method for automatically discovering relationships in a data lake table, characterized in that, include: Build a searchable knowledge base based on knowledge documents; When a new table is registered in the data lake, the table description and column description of the new table are automatically generated based on the knowledge base and language model. Calculate the semantic similarity and value overlap distance between the query column and the target column of any table in the data lake; wherein the semantic similarity is obtained based on the cosine similarity of the semantic vectors of the query column and the target column; and the value overlap distance is calculated based on the minimum asymmetric inclusion rate of the cell value set of the query column and the cell value set of the target column. Based on semantic similarity and value overlap distance, candidate columns associated with the query column are selected, and it is assumed that there is an association between the candidate columns and the query column; A metadata graph is constructed based on table descriptions, column descriptions, the relationships between tables and columns, and the discovered associations between columns. Multi-hop reasoning is performed by calculating transitive closures to deduce the potential associations between columns.

2. The method for automatically discovering data lake table relationships according to claim 1, characterized in that, The construction of a searchable knowledge base based on knowledge documents specifically includes: Receive and verify external knowledge documents; The external knowledge document is segmented to obtain knowledge fragments; The language model is invoked to generate a knowledge summary for the knowledge fragment; The knowledge fragments and summaries are vectorized and stored in the knowledge base.

3. The method for automatically discovering data lake table relationships according to claim 1, characterized in that, When a new table is registered in the data lake, the table description and column description of the new table are automatically generated based on the knowledge base and language model, specifically including: Extract the header information of the new table, and randomly sample M cell values ​​from each column of the new table as column value examples; Using the header information as the query criteria, relevant knowledge points are retrieved from the knowledge base; The header information, column value examples, and related knowledge points are input into the language model to generate column descriptions and table descriptions that include semantic domains and typical examples.

4. The method for automatically discovering data lake table relationships according to claim 1, characterized in that, The semantic similarity is obtained based on the cosine similarity of the semantic vectors of the query column and the target column, specifically including: An embedding model is used to embed the column labels and column descriptions of the query column, resulting in a column label vector for the query column. and description vector Perform weighted splicing This yields the semantic vector of the query column. : ； An embedding model is used to embed the column labels and column descriptions of the target column, resulting in a column label vector for the target column. and description vector Weighted concatenation is performed to obtain the semantic vector of the target column. ; ； Query column and target column semantic similarity , For cosine similarity, All are weights.

5. The method for automatically discovering data lake table relationships according to claim 1, characterized in that, The value overlap distance is calculated based on the minimum asymmetric coverage ratio between the set of cell values ​​in the query column and the set of cell values ​​in the target column, specifically including: Let the set of cell values ​​in the query column be . The set of cell values ​​in the target column is The following formula is used to calculate the query column. and target column Value overlap distance : 。 6. The method for automatically discovering data lake table relationships according to claim 1, characterized in that, The process of filtering candidate columns associated with the query column based on semantic similarity and value overlap distance specifically includes: Based on the semantic similarity, the target columns are pre-screened, and the target columns with semantic similarity greater than or equal to the similarity threshold are added to the candidate list; Based on the value overlap distance, the target columns are pre-filtered, and the target columns whose value overlap distance is less than or equal to the distance threshold are added to the candidate list; Calculate the joint score for the target column in the candidate list. : ； For query columns and target column semantic similarity, For query columns and target column The value of overlap distance, These are the weighting coefficients; The target columns in the candidate list are sorted based on the joint score, and the target columns with the highest joint score of a set number are selected as candidate columns. It is assumed that there is a direct correlation between the candidate columns and the query columns.

7. The method for automatically discovering data lake table relationships according to claim 1, characterized in that, The metadata graph is constructed based on table descriptions, column descriptions, the relationships between tables and columns, and the discovered associations between columns. Multi-hop reasoning is performed by calculating transitive closures to deduce the potential associations between columns. Specifically, this includes: Using tables and columns as nodes, table descriptions as node attributes of table nodes, and column descriptions as node attributes of column nodes, the relationship between tables and columns, as well as the association between columns, is used as edges to construct a metadata graph. Perform multi-hop potential relationship reasoning based on transitive closure computation on the metadata graph to discover potential associations.

8. An automatic discovery system for data lake table relationships, characterized in that, include: The knowledge management module builds a searchable knowledge base based on knowledge documents; The lake table registration and metadata enhancement module automatically generates table and column descriptions for new tables based on knowledge base and language model when a new table is registered in the data lake. The semantic and value range representation module calculates the semantic similarity and value overlap distance between the query column and the target column of any table in the data lake; wherein, the semantic similarity is obtained based on the cosine similarity of the semantic vectors of the query column and the target column; the value overlap distance is calculated based on the minimum asymmetric inclusion rate of the cell value set of the query column and the cell value set of the target column. The association discovery module filters out candidate columns associated with the query column based on semantic similarity and value overlap distance, and considers that there is an association between the candidate columns and the query column; The graph construction and graph reasoning module constructs a metadata graph based on table descriptions, column descriptions, the relationships between tables and columns, and the discovered associations between columns. It performs multi-hop reasoning by calculating transitive closures to deduce the potential associations between columns.

9. The automatic discovery system for data lake table relationships according to claim 8, characterized in that, The construction of a searchable knowledge base based on knowledge documents specifically includes: Receive and verify external knowledge documents; The external knowledge document is segmented to obtain knowledge fragments; The language model is invoked to generate a knowledge summary for the knowledge fragment; The knowledge fragments and summaries are vectorized and stored in the knowledge base.

10. The automatic discovery system for data lake table relationships according to claim 8, characterized in that, When a new table is registered in the data lake, the table description and column description of the new table are automatically generated based on the knowledge base and language model, specifically including: Extract the header information of the new table, and randomly sample M cell values ​​from each column of the new table as column value examples; Using the header information as the query criteria, relevant knowledge points are retrieved from the knowledge base; The header information, column value examples, and related knowledge points are input into the language model to generate column descriptions and table descriptions that include semantic domains and typical examples.

Images