Data discovery method for table data in data lakes

Abstract

The present invention provides a data discovery method for table data in a data lake, which has the following characteristics: step S1, constructing an index for a table; step S2, retrieving and calculating the correlation data of the associated table according to the index; step S3, constructing multiple related subgraphs according to all the correlation data; step S4, converting the related subgraphs into a first virtual document; step S5, calculating a first similarity; step S6, constructing a primary screening graph; step S7, merging all primary screening graphs to obtain a merged graph; step S8, converting the merged graph into a second virtual document; step S9, calculating a second similarity; step S10, constructing a secondary screening graph; step S11, sorting the relevance of all related subgraphs corresponding to n secondary screening graphs by a cost function, and using the sorted related subgraphs as retrieval results. In short, this method can obtain more accurate retrieval results based on retrieval keywords.

Description

Technical Field

[0001] The present invention relates to the field of data discovery, and in particular to a data discovery method for table data in a data lake. Background Art

[0002] Data lake technology and data discovery technology are two key technologies for data lake table data discovery methods.

[0003] The concept of a data lake was first proposed in 2010 by James Dixon, CTO of Pentaho, as a solution capable of processing raw data and supporting diverse user needs. Data lakes store data in its original format, reducing the number of pre-processing steps required for data warehouses. They have the following characteristics:

[0004] (1) Data lakes store raw data. As a form of data repository, data lakes acquire data from various sources, store it in its original format, and allow users to query and search it. However, the ingestion of raw data may result in the loss of data constraints and mapping information, which were not clearly defined or required in the early days of the data lake. Therefore, metadata management is crucial for data reasoning, query processing, and data quality management. Without any metadata, a data lake is difficult to use because the structure and semantics of the data are unknown, which will quickly turn the data lake into a "data swamp."

[0005] (2) A data lake is more than just a storage system. The primary function of a data lake is to serve as a data repository, which can be a centralized repository or a set of distributed repositories. However, a data lake is more than just a storage system. For example, a repository on top of the Hadoop file system needs to provide capabilities for managing and governing data so that applications can be built on top of it.

[0006] Data lakes support both data processing and data querying. A key characteristic of data lakes is ad-hoc or on-demand search, which means that data schema definition, integration, and indexing should only be performed when necessary for data access. This may result in higher data access costs, but it also increases and enhances the flexibility of data querying and application.

[0007] Existing literature defines a dataset as a collection of related observational data organized and formatted for a specific purpose. Based on this description, each data item in a data lake can be considered a dataset. Due to the massive volume of data in data lakes and the lack of a data catalog, efficiently and accurately extracting the desired datasets from data lakes has become a critical issue.

[0008] Currently, the general method of data discovery is to model it on the existing keyword-based information retrieval search system. Usually, users send requests to the search interface, and the system dynamically generates results based on the directed request and returns a sorted list of existing data sets.

[0009] However, existing data lake data discovery systems only support searches based on dataset metadata. For example, Google Dataset Search (GOODS) indexes a wide range of datasets published online and can extract and organize metadata for datasets generated and used internally. The collected metadata ranges from specific dataset information, such as owner, timestamp, and schema, to relationships between multiple datasets, such as their similarities and sources. In addition to extracting metadata from data sources, the Constance system also enriches data sources by semantically annotating the data and metadata.

[0010] In summary, searching based on dataset metadata significantly limits users’ ability to express their information needs. Furthermore, published metadata is often incomplete and, in some cases, may not be consistent with the actual data. Relying solely on metadata also limits the discoverability of datasets. Summary of the Invention

[0011] The present invention is made to solve the above problems, and its purpose is to provide a data discovery method for table data in a data lake.

[0012] The present invention provides a data discovery method for table data in a data lake, which is used to obtain retrieval results from all tables in the data lake according to query keywords, and has the following characteristics: Step S1, for each table, extracting the name, value, format, word embedding and domain distribution from the table as the index of the table; Step S2, retrieving and calculating the degree of association between two associated tables according to the index as the correlation data of the two tables; Step S3, constructing multiple related subgraphs according to all the correlation data; Step S4, converting the related subgraphs into corresponding virtual documents as first virtual documents through a topological semantic aggregator TSA; Step S5, calculating the similarity between each first virtual document and the query keyword through a BM25 function as the first correlation data. similarity; step S6, sort all the first similarities from large to small, and select the relevant subgraphs corresponding to the first m first similarities as the primary screening graph; step S7, merge all the primary screening graphs through the merging function to obtain multiple merged graphs; step S8, convert the merged graph into the corresponding virtual document as the second virtual document through the topological semantic aggregator TSA; step S9, calculate the similarity between each second virtual document and the query keyword through the BM25 function as the second similarity; step S10, sort all the second similarities from large to small, and select the merged graph corresponding to the first n second similarities as the re-screening graph; step S11, sort the relevance of all the relevant subgraphs corresponding to the n re-screening graphs through the cost function, and use the sorted relevant subgraphs as the retrieval result.

[0013] The data discovery method for table data in a data lake provided by the present invention may also have the following features: in step S1, the name is the unique identifier of the table, which is used to distinguish different tables, the value is the data in the table, the format is the data type in the table, including text, date and number, the word embedding is the numerical representation of the text in the table, and the domain distribution is the distribution of data values in the columns in the table.

[0014] The data discovery method for table data in a data lake provided by the present invention may also have the following feature: wherein, in step S2, the calculation expression for the degree of association is:

[0015] Where D(T,S) is the degree of association between the target table T and the related table S, w t is the correlation of the t-th index, is a five-dimensional vector The tth value in D N (T, S) is the name distance between the target table T and the related table S, D V (T, S) is the value distance between the target table T and the related table S, D F (T, S) is the format distance between the target table T and the related table S, DE (T, S) is the word embedding distance between the target table T and the related table S, D D (T, S) is the domain distribution distance between the target table T and the related table S, D t In (T, S), t∈{N,V,F,E,D}, m is the number of related indexes in the target table T and the related table S, is the distance between the target table T and the related table S on the t-th related index on the i-th attribute, R t It is the set of all distance values between the target table T and all other related tables in the data lake at the t-th related index.

[0016] The data discovery method for table data in a data lake provided by the present invention may also have the following features: in step S3, the relevant subgraph includes multiple vertices and edges, the vertices represent corresponding tables, and the edges are used to connect two vertices, indicating the correlation data between the tables corresponding to the two vertices. The greater the correlation data, the greater the weight of the edge.

[0017] The data discovery method for table data in a data lake provided by the present invention may also have the following features: in step S7, the merging function converts the preliminary screening graph into a set of multiple triples. For the preliminary screening graph G1 and the preliminary screening graph G2, when the value |G1∩G2| is greater than the threshold percentage h, the preliminary screening graph G1 and the preliminary screening graph G2 are merged into a merged graph according to the set.

[0018] The data discovery method for table data in a data lake provided by the present invention may also have the following feature: in step S11, a cost function is constructed based on compactness, word frequency, and weights of similar edges.

[0019] Functions and effects of the invention

[0020] According to the data discovery method for table data in a data lake involved in the present invention, on the one hand, the degree of association is calculated through the constructed index, and a related subgraph is constructed based on the degree of association. The related subgraph contains the correlation of multiple attribute information between tables, which can fully reflect the correlation between tables; on the other hand, the related subgraph is screened multiple times according to the query keyword in combination with the BM25 function and the topological semantic aggregator TSA to obtain retrieval results, thereby improving the retrieval speed while ensuring the retrieval accuracy. Therefore, the data discovery method for table data in a data lake of the present invention can obtain more accurate retrieval results based on retrieval keyword retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 1 is a flow chart of a data discovery method for table data in a data lake according to an embodiment of the present invention;

[0022] Figure 2 is a schematic diagram of the principle of constructing a related subgraph in an embodiment of the present invention;

[0023] Figure 3 Schematic diagram of the principle of generating search results based on related subgraphs in an embodiment of the present invention;

[0024] Figure 4 Schematic diagram of the accuracy results of various methods under the Synthetic dataset in an embodiment of the present invention;

[0025] Figure 5 1 is a schematic diagram of the accuracy results of various methods under the Smaller Real dataset in an embodiment of the present invention;

[0026] Figure 6 1 is a schematic diagram of the recall rate results of each method under the Smaller Real dataset in an embodiment of the present invention;

[0027] Figure 7 1 is a comparative diagram of query processing time of various query systems in an embodiment of the present invention;

[0028] Figure 8 3 is a schematic diagram comparing the query effectiveness of the system of the present invention and the GOODS system in an embodiment of the present invention. DETAILED DESCRIPTION

[0029] To facilitate understanding of the technical means, creative features, objectives, and effects of the present invention, the following embodiments, combined with accompanying drawings, specifically illustrate the data discovery method for table data in a data lake.

[0030] In this embodiment, the data lake is constructed from the Synthetic dataset and the Smaller Real dataset. The Synthetic dataset contains 32 base tables of Canadian open government data, which are synthesized using random projection and selection on the base tables. It is approximately 1.1GB in size and contains approximately 5,000 tables. The Smaller Real dataset contains tables of real data from the UK open government, including information in areas such as business, health, transportation, and public services. It is approximately 500MB in size and contains approximately 700 tables.

[0031] Figure 1 2 is a flow chart of a data discovery method for table data in a data lake according to an embodiment of the present invention.

[0032] like Figure 1 As shown, the data discovery method for table data in the data lake implemented in this embodiment is used to obtain retrieval results from all tables in the data lake based on query keywords, including the following steps:

[0033] Figure 2It is a schematic diagram of the principle of constructing a related subgraph in an embodiment of the present invention.

[0034] like Figure 2 As shown, index construction is performed on each table to obtain the corresponding index, and then the tables with correlation are retrieved based on all tables and indexes and the correlation degree is calculated. The relevant subgraph is constructed based on the correlation degree calculation result. The specific process is shown in the following steps S1 to S3:

[0035] In step S1, for each table, the name, value, format, word embedding and domain distribution are extracted from the table as the index of the table.

[0036] The name is the unique identifier of the table, which is used to distinguish different tables. The similarity of table names may mean that the tables have certain similar attributes or associations. Therefore, the name needs to be considered when measuring the relevance of tables.

[0037] The value is the data in the table, which can show whether the data between the tables is shared or has similar characteristics. If the values in the two tables are very similar, there may be some kind of relationship, for example, they may both come from the same data source.

[0038] The format is the data type in the table, including text, date, and number. The similarity of format between tables may mean that they have certain common properties, such as both tables contain date columns.

[0039] Word embeddings are numerical representations of text in a table. In this embodiment, the text is converted into corresponding numerical representations using a vector space model. If columns in two tables have similar word embeddings, there may be a certain relationship between them. For example, they may be data about similar topics.

[0040] Domain distribution refers to the distribution of data values in a table column. If columns in two tables have similar domain distributions, there may be a relationship between them. For example, they may both be data about similar domains.

[0041] The algorithm program code corresponding to the index construction process in this embodiment is as follows:

[0042]

[0043]

[0044] In this embodiment, the table name is Address, the attribute is a, and the attribute value corresponding to attribute a is {"18 Portland Street, M1 3BE", "41 Oxford Road, M13 9PL", "9 Mirabel Street, M3 1NN"}. The main functions in the above algorithm program code are described as follows:

[0045] get_qgrams(a) is used to obtain Q(a) of attribute a through the q-grams algorithm with q=4, so get_qgrams(a)={addr,ddre,dres,ress}.

[0046] H is histogram.new() that generates a count histogram, which saves the tokens of the value and is used to retrieve high-frequency and low-frequency words in the value.

[0047] get_tokens() is used to split the value into different parts by punctuation marks, mark the different parts as a series of words, and insert them into H to count the usage frequency of different words. Then get_tokens(a) = {portland,3BE,oxford,9PL,...}.

[0048] infrequent() and frequent() are used to tokenize the words obtained by get_tokens(), storing the low-frequency words in T(a) and the high-frequency words into word embedding vectors. middle.

[0049] get_regex_string() is used to perform regular expression classification on the attribute value format set. The regular expression classification is C = [AZ][az]+, N = [0-9]+, A = [A-Za-z0-9]+, p = [.,;: / -], where P can capture characters that cannot be captured by regular expressions in addition to punctuation marks. Given an attribute value, after tokenizing it, for each token, once its matching regular expression classification is found, the regular expression symbol (C, N, A, P) represented by it is added to the set representation R(a). If the same symbol appears consecutively, all symbols except the first one will be replaced with "+", for example {NC+P+A+}. If an attribute value matches multiple regular expressions, the first match is selected in the order listed above.

[0050] Step S2: searching and calculating the association degree between two associated tables according to the index as the correlation data of the two tables.

[0051] The calculation expression of the degree of association is:

[0052]

[0053]

[0054]

[0055]

[0056] Where D(T,S) is the degree of association between the target table T and the related table S, w t is the correlation of the t-th index, is a five-dimensional vector The tth value in D N (T, S) is the name distance between the target table T and the related table S, D V (T, S) is the value distance between the target table T and the related table S, D F (T, S) is the format distance between the target table T and the related table S, D E (T, S) is the word embedding distance between the target table T and the related table S, D D (T, S) is the domain distribution distance between the target table T and the related table S, D t In (T, S), t∈{N,V,F,E,D}, m is the number of related indexes in the target table T and the related table S, is the distance between the target table T and the related table S on the t-th related index on the i-th attribute, R t It is the set of all distance values between the target table T and all other related tables in the data lake at the t-th related index.

[0057] In this embodiment, the weight Represents the observed distance value In the set R t The probability of the minimum value is set by setting the weight To compensate for the weak attributes associated with the target attribute, that is, when an attribute of the related table has no strong correlation with the target attribute of the target table, its weight for the related measurement with the target attribute will be smaller, thereby avoiding the influence of these weakly correlated attributes on the measurement results.

[0058] In this embodiment, the correlation w t By placing the corresponding The input is a pre-trained logistic regression classifier, which is trained by using multiple existing related (T, S) pairs as positive training samples and irrelevant (T, S) pairs as negative samples.

[0059] In this embodiment, the data lake is retrieved through index retrieval to obtain the target table T with the name of GPs and the related table S with the name of GP funding. The distance value is calculated based on the target table T and the related table S. Name distance value D N , value distance value D V , format distance value D F , word embedding distance value D E and domain distribution distance value D D The calculation process is described as follows:

[0060] Practice City Postcode Payment Dr E Cullen London W1G 6BW 73648 Blackfriars Salford M3 6AF 15530

[0061] The above table is the related table S. The first row contains the names of the attributes in the table, and the second and third rows contain the specific contents of the corresponding attributes. For example, the cell in the second row and second column indicates that the content corresponding to the City attribute in the data in this row is London.

[0062]

[0063]

[0064] The above table is the target table T. The first row contains the names of the attributes in the table, and the second and third rows contain the specific contents of the corresponding attributes. For example, the cell in the second row and second column indicates that the content corresponding to the Street attribute in this row of data is 69 Church St.

[0065] In this embodiment, attribute pairs are constructed based on the hash scheme for the related attributes in the target table T and the related table S, and the distance values of each attribute pair on each index are calculated. If the content corresponding to the index in the attribute pair is a numeric value, the KS Tongji metric is used to calculate the distribution-based metric. Otherwise, the calculation result is set to the maximum distance 1. The calculation results are shown in the following table:

[0066] Attribute pairs <![CDATA[D N ]]> <![CDATA[D V ]]> <![CDATA[D F ]]> <![CDATA[D E ]]> <![CDATA[D D ]]> (T.Practice,S.Practice) 0.0 0.9 0.6 0.2 1.0 (T.City,S.City) 0.0 0.2 0.2 0.3 1.0 (T.Postcode,S.Postcode) 0.0 0.6 0.1 0.8 1.0

[0067] The first column in the table above represents the attribute pair constructed by the target table T and the related table S, and the second to sixth columns are the name distance values D of the attribute pair. N , value distance value D V , format distance value D F , word embedding distance value D E and domain distribution distance value D D For example, the cell in the second row and third column represents the distance value D calculated based on the content corresponding to the attribute Practice of the target table T and the content corresponding to the attribute Practice of the related table S. V is 0.9. In this embodiment, the name distance value D of the attribute pairs with the same attribute name is N Taking the value as 1, the value with correlation distance D V And word embedding distance value D E Set the value to be less than 1. The above three attribute pairs all contain text values, so the domain distribution distance value D D The maximum distance is 1.

[0068] Step S3: construct multiple related subgraphs based on all the correlation data.

[0069] The relevant subgraph includes multiple vertices and edges. The vertices represent corresponding tables, and the edges are used to connect two vertices, indicating the correlation data between the tables corresponding to the two vertices. The greater the correlation data, the greater the weight of the edge.

[0070] In this embodiment, the related subgraph has an intuitive and clear graphical structure. On the one hand, it allows users to more intuitively understand the relationship between tables in the data lake, quickly locate the data tables they need to find, and then conduct subsequent data analysis and mining; on the other hand, the related subgraph can be easily searched and traversed, allowing users to quickly find the required data tables in the data lake, thereby improving the query efficiency of the data lake.

[0071] In summary, building related subgraphs can improve the visualization and perceptibility of the associations between tables in the data lake, enabling users to better understand the structure and characteristics of the data in the data lake, thereby improving the efficiency of data lake usage and the accuracy of data analysis.

[0072] Figure 3 It is a schematic diagram of the principle of generating search results based on related subgraphs in an embodiment of the present invention.

[0073] like Figure 3 As shown, all relevant subgraphs are processed to generate corresponding first virtual documents, the first virtual documents are screened according to the query keyword to obtain a primary screening graph, the primary screening graphs are merged to obtain multiple merged graphs, the corresponding second virtual documents are generated according to the merged graphs, the second virtual documents are screened according to the query keyword to obtain a secondary screening graph, all relevant subgraphs corresponding to the secondary screening graphs are sorted to obtain the relevant subgraph sorting results as the retrieval results, and the specific process is shown in steps S4 to S11:

[0074] Step S4, converting the relevant subgraphs into corresponding virtual documents as the first virtual document through the topological semantic aggregator TSA. In this embodiment, the topological semantic aggregator TSA aggregates the information in multiple related subgraphs into a graph, and uses a graph-based algorithm to analyze the relationship between entities and concepts in the data to obtain a virtual document. The virtual document can be used for information retrieval, semantic search and other tasks that require compact and semantically meaningful representation of the information in the graph, which can improve the accuracy, efficiency and scalability of information retrieval and analysis tasks.

[0075] Step S5: Calculate the similarity between each first virtual document and the query keyword using the BM25 function as the first similarity.

[0076] Step S6: sort all first similarities from large to small, and select the first m related subgraphs corresponding to the first similarities as the primary screening graph. In this embodiment, m=10000.

[0077] Step S7: Merge all the preliminary screening graphs through a merging function to obtain multiple merged graphs.

[0078] Among them, the merging function converts the preliminary screening graph into a set of multiple triples. For the preliminary screening graph G1 and the preliminary screening graph G2, when the value |G1∩G2| is greater than the threshold percentage h, the preliminary screening graph G1 and the preliminary screening graph G2 are merged into a merged graph according to the set. In this embodiment, h=30%.

[0079] Step S8: The merged graph is converted into a corresponding virtual document as a second virtual document by the topological semantic aggregator TSA.

[0080] Step S9: Calculate the similarity between each second virtual document and the query keyword using the BM25 function as the second similarity.

[0081] Step S10: sort all the second similarities from large to small, and select the merged graphs corresponding to the first n second similarities as the re-screened graph.

[0082] Step S11 , sorting the relevance of all relevant subgraphs corresponding to the n complex screening graphs using a cost function, and using the sorted relevant subgraphs as retrieval results.

[0083] In step S11, the cost function is constructed based on compactness, word frequency and weights of similar edges.

[0084] To verify the performance of the data discovery method for table data in a data lake, namely the method of the present invention, a performance test was conducted on the method of the present invention and the existing TUS method and Aurum method. The test hardware environment was a Linux server with a 3090 GPU. The above three methods were converted into corresponding models and trained using Pytorch. The corresponding performance test results were then obtained using data from the Synthetic dataset and the Smaller Real dataset.

[0085] Figure 4 It is a schematic diagram of the accuracy results of each method under the Synthetic dataset in an embodiment of the present invention.

[0086] like Figure 4As shown, the horizontal axis is the number k of relevant subgraphs that serve as search results, and the vertical axis is the accuracy of the search results. It can be seen that when k∈[5,140], the method of the present invention has relatively high accuracy, and it decreases linearly when k is greater than 140, indicating that the vast majority of relevant data sets are ranked at the top. For the Aurum method, the Aurum method is relatively accurate when k∈[5,50], but linearly degenerates in the rest of the interval, for example, when k=350, it drops to 0.49. For the TUS method, the accuracy results of this method show that 20% to 30% of the search results are false positives, and their ranking is always higher than that of truly relevant tables. In summary, as k increases, the method of the present invention can identify 20% more relevant tables than the TUS method and 10% more relevant tables than the Aurum method.

[0087] Figure 5 It is a schematic diagram of the accuracy results of each method under the Smaller Real dataset in an embodiment of the present invention.

[0088] like Figure 5 As shown in the figure, the horizontal axis is the number k of relevant subgraphs as retrieval results, and the vertical axis is the accuracy of the retrieval results. It can be seen that when k∈[5,110], the method of the present invention has higher accuracy. Compared with the TUS method, the accuracy is improved by 0.2~0.4, and compared with the Aurum method, the accuracy is improved by 0.05~0.3.

[0089] In summary, the method of the present invention can discover the association of multiple attribute information by constructing an index, and the correlation subgraph constructed by correlation degree calculation can better reflect the correlation between tables, thereby improving the accuracy of the retrieval results.

[0090] Figure 6 2 is a schematic diagram of the recall rate results of each method under the Smaller Real dataset in an embodiment of the present invention.

[0091] like Figure 6 As shown, the horizontal axis is the number k of relevant subgraphs as retrieval results, and the vertical axis is the recall rate of the retrieval results. When k<110, the method of the present invention recognizes more than 70% of the relevant data sets, while the TUS method and the Aurum method only recognize about 55%. It can be seen that the method of the present invention has a better recognition effect.

[0092] In summary, compared with the existing TUS method and Aurum method, the method of the present invention has better retrieval accuracy on the Synthetic dataset and Smaller Real dataset.

[0093] In this embodiment, the query system based on the method of the present invention is used as the system of the present invention, and the performance is compared with the query system based on the GOODS method, namely the GOODS system, and the query system based on the whole, namely the whole map system, to verify the effectiveness of the method of the present invention in the query system. Ten keyword queries are performed on each system respectively, and the keyword contents are shown in the following table:

[0094]

[0095]

[0096] The first column in the above table is the query terms Q1 to Q10, and the second column is the actual content corresponding to the query terms Q1 to Q10. For example, the cell in the second row and second column indicates that the actual content of the query term Q1 is Community Hospital.

[0097] Figure 7 1 is a schematic diagram comparing query processing time of various query systems in an embodiment of the present invention.

[0098] like Figure 7 As shown, the horizontal axis represents each query word, and the vertical axis represents the average query time, i.e., the average search time, used by each query system to perform 10 queries based on the query word. It can be seen that the whole graph system consumes the most time because it does not use any indexing mechanism and needs to traverse the graph to perform queries. In addition, in queries for ten different query words, the system of the present invention is faster than the GOODS system, with an average efficiency improvement of 15.48%.

[0099] Figure 8 3 is a schematic diagram comparing the query effectiveness of the system of the present invention and the GOODS system in an embodiment of the present invention.

[0100] like Figure 8 As shown, the horizontal axis represents each query word, and the vertical axis represents the query effectiveness of performing 10 queries based on the query word, and obtaining 10 related subgraphs as query results for each query. It can be seen that the average query effectiveness of the system of the present invention is 147.68% higher than that of the GOODS system, and the query effectiveness of the system of the present invention is greater than 0.75, further illustrating the effectiveness of the method of the present invention for querying table data in the data lake.

[0101] In summary, the system of the present invention has better query efficiency than the existing GOODS system and the whole graph system, indicating that the data discovery method for table data in the data lake of the present invention has better query effect when applied to the query system.

[0102] Functions and Effects of the Embodiments

[0103] According to the data discovery method for table data in a data lake involved in this embodiment, on the one hand, the degree of association is calculated through the constructed index, and a related subgraph is constructed based on the degree of association. This related subgraph contains the correlation of multiple attribute information between tables, which can fully reflect the correlation between tables. On the other hand, based on the query keyword, the related subgraph is screened multiple times in combination with the BM25 function and the topological semantic aggregator TSA to obtain retrieval results, which improves the retrieval speed while ensuring the retrieval accuracy. In short, this method can obtain more accurate retrieval results based on the search keyword.

[0104] The above embodiments are preferred examples of the present invention and are not intended to limit the scope of protection of the present invention.

Claims

1. A data discovery method for table data in a data lake, used to obtain search results from all tables in the data lake based on query keywords, characterized by: The following steps are involved: Step S1, for each of the tables, extracting the name, value, format, word embedding, and domain distribution from the table as an index of the table; Step S2, searching and calculating the degree of association between two associated tables according to the index as correlation data of the two tables; Step S3, constructing a plurality of related subgraphs based on all the related data; Step S4, converting the relevant subgraph into a corresponding virtual document as a first virtual document through a topological semantic aggregator TSA; Step S5: Calculating the similarity between each of the first virtual documents and the query keyword using the BM25 function as a first similarity; Step S6: sort all the first similarities from largest to smallest, and select the first m related subgraphs corresponding to the first similarities as the primary screening graph; Step S7, merging all the preliminary screening graphs using a merging function to obtain multiple merged graphs; Step S8, converting the merged graph into a corresponding virtual document as a second virtual document through a topological semantic aggregator TSA; Step S9, calculating the similarity between each of the second virtual documents and the query keyword using the BM25 function as a second similarity; Step S10: sorting all the second similarities from largest to smallest, and selecting the merged graphs corresponding to the first n second similarities as a re-screened graph; Step S11 , sorting the relevance of all the relevant subgraphs corresponding to the n complex screening graphs by using a cost function, and using the sorted relevant subgraphs as the search results.

2. The data discovery method for table data in a data lake according to claim 1, characterized in that: in, In step S1, the name is a unique identifier of the table, used to distinguish different tables. The value is the data in the table, The format is the data type in the table, including text, date and number, The word embeddings are numerical representations of the text in the table, The domain distribution is the distribution of data values in the columns of the table.

3. The data discovery method for table data in a data lake according to claim 1, characterized in that: in, In step S2, the calculation expression of the association degree is: Where D(T,S) is the degree of association between the target table T and the related table S, w t is the correlation of the t-th index, is a five-dimensional vector The tth value in D N (T, S) is the name distance between the target table T and the related table S, D V (T, S) is the value distance between the target table T and the related table S, D F (T, S) is the format distance between the target table T and the related table S, D E (T, S) is the word embedding distance between the target table T and the related table S, D D (T, S) is the domain distribution distance between the target table T and the related table S, D t In (T, S), t∈{N,V,F,E,D}, m is the number of related indexes in the target table T and the related table S, is the distance between the target table T and the related table S on the t-th related index on the i-th attribute, R t It is the set of all distance values between the target table T and all other related tables in the data lake at the t-th related index.

4. The data discovery method for table data in a data lake according to claim 1, characterized in that: in, In step S3, the relevant subgraph includes a plurality of vertices and edges. The vertices represent the corresponding tables, The edge is used to connect two vertices, and represents the correlation data between the tables corresponding to the two vertices. The greater the correlation data, the greater the weight of the edge.

5. The data discovery method for table data in a data lake according to claim 1, characterized in that: in, In step S7, the merging function converts the preliminary screening graph into a set of multiple triples. For the preliminary screening graph G1 and the preliminary screening graph G2, when the value |G1∩G2| is greater than the threshold percentage h, the preliminary screening graph G1 and the preliminary screening graph G2 are merged into the merged graph according to the set.

6. The data discovery method for table data in a data lake according to claim 1, characterized in that: in, In step S11, the cost function is constructed based on compactness, word frequency and weights of similar edges.

Images