A relevance set-oriented maximum-difference data table combination query method

Abstract

The application belongs to the field of data retrieval, and discloses a difference maximization data table combination query method for associated data sets. In the data processing stage, the given table data set is first processed, a feature-data column index is established, connectable tables in the table data set are found according to the index, a data table connection graph index is constructed, and the connection information between each table is pre-calculated. In the data query stage, the candidate connection column set is searched in the feature-data column index according to the given sample query table and the connection column set, the candidate data table set is obtained, and the data table set that can be connected with the sample query table and has the maximum difference is selected according to the given budget. The application first proposes a method for searching connectable data table combinations in associated data sets, filters connectable data tables by using a feature index, establishes a data table connection graph to find the connection paths between data tables, and returns the data table set that has the maximum difference under the budget constraint.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of data retrieval, and in particular relates to a difference maximization data table combination query method for associated data sets. BACKGROUND

[0002] Many data engines are widely used in data queries at present, among which table data queries have attracted widespread attention. As a kind of structured data, table data widely exists in various fields such as enterprises, scientific research and finance, and its query and management are of great significance to improve work efficiency and support decision-making. There is an urgent practical need to efficiently obtain the required connectable data table.

[0003] In order to more efficiently perform table data queries, some advanced technologies and tools have been proposed and applied. For example, table recognition technology based on deep learning can automatically detect tables, identify table structures and contents through semantic segmentation algorithms, object detection algorithms and text sequence generation algorithms, thereby improving the query and processing efficiency of table data.

[0004] In the process of table data connectable query, there are also some challenges such as diversity evaluation, value evaluation and multi-attribute connection query. In order to cope with these challenges, researchers and developers are constantly exploring new methods and technologies. For example, by constructing a document summary index structure and using a multi-modal LLM, table data can be efficiently retrieved and summarized, thereby improving the accuracy and efficiency of table data query. However, due to the variable row and column order of different tables, the complexity is high when calculating the overlapping area of table data, and existing literature has proposed a variety of methods. For example, Sloth generates a seed list by detecting attribute pairs of shared data element values in two tables, and gradually combines these seeds to calculate the maximum overlapping area between the two tables: Mate uses a Bloom filter to pre-screen rows that cannot overlap, and then obtains the overlapping area of the two tables. However, in the existing difference maximization data table query method, the complexity of table data in real data lake is not considered, such as proportion foreign key constraints and inconsistent namespaces. Therefore, if a new method can be proposed to find a data table combination in an associated data set that meets the budget limit, this problem can be effectively solved. SUMMARY

[0005] In order to solve the deficiencies of the prior art, the application provides a difference maximization data table combination query method for associated data sets. The query method is for a table data lake with multi-level connection relationship, establishes a table connection index based on a graph database, and proposes a difference maximization data table combination query method in an associated data set.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solution:

[0007] This invention is a method for maximizing the difference between related datasets and combined data tables. This method refers to querying data tables in a table dataset C that are related to table T. Q The goal is to combine data tables that satisfy multi-attribute join conditions on a specified set of join columns Q and maximize the difference between them. This includes both data processing and data query stages. Let the tabular dataset composed of the tabular data be denoted as C = {T1, T2, ... T}. n}, T q To query the sample data table, specify the set of join columns Q = {q1, q2, ..., q...} m Specifically:

[0008] Phase 1: Data Processing Phase, which includes the following steps:

[0009] Step 1.1: For each data table T in the tabular dataset C i Based on the data type of each column, extract the feature set F of each column. i，j For a tabular dataset C, for each feature type f l Construct feature-data column index I l This leads to the construction of a feature-data column index set I = {I1, I2, ..., I...} L};

[0010] Step 1.2: For each data table T in the tabular dataset C i Based on feature-data column index set According to data table T i Each data column c j The data columns in the tabular dataset C that satisfy the similarity threshold θ are used to form the data table T. i The candidate connection set R i ={ <T k ,c x ,FK(c x ))>|c x ∈T i ,FK(c x )∈T k ,e(c x ,FK(c x ))>θ}, where FK(c x ) represents column c x In candidate table T k In the joinable columns, e is the column matching score calculation function, which calculates the score for each candidate table T in the candidate join set R. k Select a set of optimal connections <T k cx , FK(c x )) >, and delete the candidate join set R i i and other joins of candidate table T k ;

[0011] Step 1.3, for the table dataset C, according to the candidate join set R i of each data table T i in the table dataset C, use the graph database to build a graph index G representing the connection relationship between each table in the table dataset C;

[0012] The second stage: the data query stage, specifically including the following steps:

[0013] Step 2.1, according to the given query example data table T q and the join column set Q, retrieve all data columns in each feature-data column index set I in the join column set Q that satisfy the similarity threshold θ, and further construct the sample candidate join set R q of the query example data table T;

[0014] Step 2.2, according to the sample candidate join set R q , obtain all data table combinations p q that can be connected to the query example data table T k on all data columns in the join column set Q, have a connection path in the graph index G, and satisfy , and further construct the candidate data table combination set PT where PC k is a set of all connection paths that can connect all data tables in the data table combination p k ;

[0015] Step 2.3, according to each data table combination p q in the candidate data table combination set PT k and the connection path set PC k corresponding to p k , obtain the difference row tuple set Diff(T k , p q ) of the data table combination p q and the query example data table T k ;

[0016] Step 2.4, according to the given budget B, find a subset RP of the candidate data table combination set PT q , so that the subset RP satisfies the following conditions:

[0017] ​

[0018] Then construct the result set R = {(p k PC k )|p∈RP}, and return the result set R.

[0019] A further improvement of the present invention is that: in step 1.1, for each data table T in the tabular dataset C... i Construct a feature-data column index set Specifically, the steps are as follows;

[0020] Step 1.1.1: For the tabular dataset C, for the data table T i In the j-th column, data from table T is extracted using a pre-defined feature extraction algorithm. i The feature set of the j-th column Where L is the number of feature types;

[0021] Step 1.1.2: For the tabular dataset C, for each feature type f l Constructing a feature-data column index I based on locality-sensitive hashing l ;

[0022] Step 1.1.3: For each column of data characteristics Construct a data column information (T) i (j) and the index item of the corresponding feature value, and add it to the corresponding feature-data column index I. l Then, a feature-data column index set I = {I1, I2, ..., I...} is constructed. L}

[0023] A further improvement of the present invention is that, in step 1.2, for each data table T in the tabular dataset C... i and candidate connection set R i Therefore, the candidate connection set R i Each candidate table T in k Select a set of optimal connections <T k c x ,FK(c x The specific steps include the following:

[0024] Step 1.2.1: For the candidate connection set R i Each candidate table T in k In the candidate connection set R i Filter out all data from table T i The data columns in the table point to candidate table T. k Joining data columns in R ik ={ <T kc x ,FK(c x )>|c x ∈T i ,FK(c x )∈T k};

[0025] Step 1.2.2, for connecting R ik Each group of connections <c x ,FK(c x )>, calculate its connection score:

[0026] FK-SCORE(c x ,FK(c x ))=(e(c x ,FK(c x ))+j(c x ,FK(c x ))*max{u(c x ), u(FK(c x ))}

[0027] Where e(c) x ,FK(c x ) is the column matching score calculation function, j(c x ,FK(c x The matching score for each data element in the two columns is calculated as follows:

[0028]

[0029] Where u(c) x ), u(FK(c x The uniqueness scores of the two columns are calculated as follows:

[0030]

[0031] Among them, unique(c x ) represents c x The set of unique values ​​in a column, u(FK(c x )) represents FK(c x The set of unique values ​​in a column;

[0032] Steps 1, 2, and 3: Select FK_SCORE(c x ,FK(c x The largest connection pair <c x ,FK(c x As data table T i To candidate table T k The best connection.

[0033] A further improvement of the present invention is that in step 2.3, the set PT is combined according to the candidate data table. q Each data table combination p in k and p k The corresponding set of connection paths PC k Get data table combination p k Query sample data table T q The set of differing row tuples, Diff(T) q ,p k The specific steps include the following:

[0034] Step 2.3.1: For the set of connection paths PC k Each link path in x Select the connection path in sequence. x The included edges <T i ,T j >and connect data table T i T j The connection conditions and connection methods are as follows:

[0035]

[0036] Then obtain the connection path. x The set of row tuples PT(path) on the join column set Q after combining all the data tables x ):

[0037]

[0038] Where |V(path) x | indicates the number of vertices on the path;

[0039] Step 2.3.2: Based on the set of connection paths PC k The set of row tuples PT(path) formed on each path x ), calculate the combination p of the data tables. k Query sample data table T q The set of differing row tuples:

[0040]

[0041] A further improvement of the present invention is that: in step 2.4, based on the given budget B, a candidate data table combination set PT is constructed. q A subset RP that satisfies the given conditions is constructed as follows:

[0042] Step 2.4.1: Initialize the remaining budget b = B, and the set of selected row tuples RS = ∏ Q (Tq ), Selected data table set TS, candidate data table combination list PS = { <p k ,g(p k )>|p k ∈PT q} and the selected data table combination set RP, where g(p k The marginal benefit of this data table combination is calculated as follows:

[0043]

[0044] Step 2.4.2: Select the data table combination p from the candidate data table combinations in turn, ensuring that the marginal revenue is the highest and the cost is lower than the remaining budget b. k Add the selected data table set RP to the data table set RP, and update the selected data table set TS = TS∪p in sequence. k Remaining budget b = B - ∑ T∈TS price(T), the set of selected row tuples RS = RS∪Diff(RS, p) k This updates the marginal revenue of each remaining candidate table combination in the candidate data table combination list PS.

[0045] Step 2.4.3: If there is no data table combination that meets the conditions in the candidate data table combination list PS, return the data table combination set RP.

[0046] The beneficial effects of this invention are:

[0047] This invention uses feature indexes to perform preliminary filtering of connectable data tables, which can effectively reduce the number of connecting edges in the table connection graph, reduce the amount of computation, and thus shorten the query time.

[0048] This invention uses a specific join score calculation function to determine the connectability between data tables without requiring pre-providing join information from the data lake, thus improving the scalability of the table join graph.

[0049] This invention selects a combination of data tables that meet the constraints through a greedy selection method, which ensures query efficiency without significantly reducing accuracy. Attached Figure Description

[0050] Fig. 1 This is a flowchart of the data table combination query method for maximizing the difference of the present invention.

[0051] Fig. 2 This is a schematic diagram of the table connection graph index of the present invention.

[0052] Fig. 3 This is a schematic diagram of the data table query engine of the present invention. Detailed Implementation

[0053] The following will disclose embodiments of the present invention. For clarity, many practical details will be described in the following description. However, it should be understood that these practical details are not intended to limit the present invention. That is, in some embodiments of the present invention, these practical details are not essential.

[0054] For ease of description, the relevant symbols are defined as follows: Tabular dataset C = {T1, T2, ... T} n} contains n data tables, query sample data table T q Specify the set of columns to join: Q = {q1, q2, ..., q} m}

[0055] like Figs. 1-3 As shown, this invention is a data table combination query method for maximizing the difference of related datasets. This method refers to querying data in a table dataset C that is related to T. q The goal is to combine data tables that satisfy multi-attribute join conditions on a specified set of join columns Q and maximize the difference between them. This includes both data processing and data query stages. Let the tabular dataset composed of the tabular data be denoted as C = {T1, T2, ..., T}. n}, T q To query the sample data table, specify the set of join columns Q = {q1, q2, ..., q...} m Specifically:

[0056] Phase 1: Data Processing Phase, which includes the following steps:

[0057] Step 1.1: For each data table T in the tabular dataset C i Based on the data type of each column, extract the feature set F of each column. i，j For a tabular dataset C, for each feature type f l Construct feature-data column index I l This leads to the construction of a feature-data column index set I = {I1, I2, ..., I...} L}

[0058] Among them, constructing a feature-data column index set Specifically, the steps are as follows;

[0059] Step 1.1.1: For the tabular dataset C, for the data table T i In the j-th column, data from table T is extracted using a pre-defined feature extraction algorithm. i The feature set of the j-th column Where L is the number of feature types.

[0060] The feature extraction algorithm in this application can be flexibly selected according to factors such as data type and application scenario. For example, statistical features such as mean and variance can be extracted for numerical data, and text features such as word frequency, TF-IDF, and word embedding can be extracted for text data.

[0061] Step 1.1.2: For the tabular dataset C, for each feature type f l Constructing a feature-data column index I based on locality-sensitive hashing l ;

[0062] Step 1.1.3: For each column of data characteristics Construct a data column information (T) i (j) and the index item of the corresponding feature value, and add it to the corresponding feature-data column index I. l Then, a feature-data column index set I = {I1, I2, ..., I...} is constructed. L}

[0063] Step 1.2: For each data table T in the tabular dataset C i Based on feature-data column index set According to data table T i Each data column c j The data columns in the tabular dataset C that satisfy the similarity threshold θ are used to form the data table T. i The candidate connection set R i ={ <T k ,c x ,FK(c x ))>|c x ∈T i ,FK(c x )∈T k ,e(c x ,FK(c x ))>θ}, where FK(c x ) represents column c x In candidate table T k In the joinable column, e is the column matching score calculation function, which applies to each candidate table T in the candidate join set Ri. k Select a set of best connections <T k c x ,FK(c x ))>, and delete the candidate connection set R i Data table T i With candidate table T k Other connections specifically include the following steps:

[0064] Step 1.2.1: For the candidate connection set R ieach candidate table T k , in the candidate connection set R i , all connections R i pointing from a data column in data table T k to a data column in candidate table T ik = { <T k , c x , FK(c x ) | c x ∈ T i , FK(c x )∈ T k} ;

[0065] Step 1.2.2, for each group of connection <c x , FK(c x )> in the connection set R ik , calculate its connection score:

[0066] FK_SCORE(c x , FK(c x )) = (e(c x , FK(c x )) + j(c x , FK(c x )) * max{u(c x ), u(FK(c x ))}

[0067] where e(c x , FK(c x )) is the column matching score calculation function, j(c x , FK(c x )) is the matching score of each data element in the two columns, calculated as follows:

[0068]

[0069] where u(c x ), u(FK(c x )) are the uniqueness scores of the two columns respectively, calculated as follows:

[0070]

[0071] where unique(c x ) represents the set of unique values in column c x , u(FK(c x )) represents the set of unique values in column FK(c x );

[0072] Step 1.2.3, select FK_SCORE(c x,FK(c x The largest connection pair <c x ,FK(c x As data table T i To candidate table T k The best connection.

[0073] Step 1.3: For the tabular dataset C, based on each data table T in the tabular dataset C... i The candidate connection set R i Use graph databases such as Neo4j to construct a graph index G that represents the connection relationships between tables in a tabular dataset C.

[0074] The second stage: the data query stage, which includes the following steps:

[0075] Step 2.1: Based on the given query sample data table T q Join the column set Q, and the set of indexes for each feature-data column in the column set Q. Retrieve all data columns that satisfy the similarity threshold θ, and then construct a candidate join set R of the query sample data table Tq. q ;

[0076] Step 2.2: Based on the sample candidate connection set R q Retrieves all data columns in the join column set Q that are matched with the data in the query sample table T. q For columns that can be joined, a join path exists in the graph index G, and the following conditions are met: Data table combination p k This leads to the construction of a candidate data table combination set. PC k It is all combinations of data tables that can be joined. k The set consisting of the join paths of all data tables in the dataset;

[0077] Step 2.3: Combine the candidate data table into set PT. q Each data table combination p in k and p k The corresponding set of connection paths PC k Get data table combination p k Query sample data table T q The set of differing row tuples, Diff(T) q p k The specific steps include the following:

[0078] Step 2.3.1: For the set of connection paths PC k Each link path in x, select connection path path x containing edges <T i , T j > and connect data table T i , T j , the connection condition and connection mode are as follows:

[0079]

[0080] Further obtain all data table combinations on connection path path x After combination, the row tuple set PT(path x ) on the connection column set Q:

[0081]

[0082] Wherein |V(path x )| represents the number of vertices on the path;

[0083] Step 2.3.2, according to the row tuple set PT(path x ) formed on each path in the connection path set PC k , calculate the difference row tuple set between the data table combination p k and the query sample data table T q :

[0084]

[0085] Step 2.4, according to the given budget B, find a subset RP of the candidate data table combination set PT q , so that the subset RP satisfies the following conditions:

[0086]

[0087] Further construct the result set R={(p k , PC k )|p∈RP} and return the result set R.

[0088] Wherein, a subset RP of the candidate data table combination set PT q satisfying the limited condition is constructed, which specifically includes the following construction process:

[0089] Step 2.4.1, initialize the remaining budget b=B, the selected row tuple set RS=∏ Q (T q ), the selected data table set TS, and the candidate data table combination list PS={<p k , g(p k )>|p k ∈ PT q} and a selected data table combination set RP, where g(p k is the marginal revenue of the data table combination, which is calculated as follows:

[0090]

[0091] Step 2.4.2, select a data table combination p with the highest marginal revenue and a cost lower than the remaining budget b from the candidate data table combination list PS in turn k is added to the data table combination set RP, and the selected data table set TS = TS U p is updated in turn k , the remaining budget b = B - å T∈TS price(T), the selected row tuple set RS = RS U Diff(RS, p k ), and the marginal revenue of each remaining candidate table combination in the candidate data table combination list PS is updated in turn.

[0092] Step 2.4.3, when there is no data table combination satisfying the condition in the candidate data table combination list PS, return the data table combination set RP.

[0093] The above merely describes the embodiments of the present application but should not be taken as limiting the present application. The present application can be modified and changed in various ways by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A method for maximizing the difference of a data table combination query for a related dataset, characterized in that: This method refers to querying data in tabular dataset C that matches data in table T. q A combination of data tables that satisfy multi-attribute join conditions on a specified set of join columns Q and maximize the difference, including the data processing stage and the data query stage. Let the tabular dataset composed of the tabular data be denoted as C = {T}. 1, T2,…T n }, T q To query the sample data table, specify the set of join columns Q = {q1, q2, ..., q}. m Specifically: Phase 1: Data Processing Phase, which includes the following steps: Step 1.1: For each data table T in the tabular dataset C i Based on the data type of each column, extract the feature set F of each column. i,j For a tabular dataset C, for each feature type f l Construct feature-data column index I l This leads to the construction of a feature-data column index set. Step 1.2: For each data table T in the tabular dataset C i Based on feature-data column index set According to data table T i Each data column c j The data columns in the tabular dataset C that satisfy the similarity threshold θ are used to form the data table T. i The candidate connection set R i ={ <T k ,c x ,FK(c x ))>|c x ∈T i ,FK(c x )∈T k ,e(c x ,FK(c x ))>θ}, where FK(c x ) represents column c x In candidate table T k In the set of candidate joins R, the joinable columns are defined, and 'e' is a function that calculates column matching scores. i Each candidate table T in k Select a set of best connections <T k ,c x ,FK(c x ))>, and delete the candidate connection set R i Data table T i With candidate table T k Other connections; Step 1.3: For the tabular dataset C, based on each data table T in the tabular dataset C... i The candidate connection set R i Use a graph database to construct a graph index G that represents the connection relationships between tables in a tabular dataset C; The second stage: the data query stage, which includes the following steps: Step 2.1: Based on the given query sample data table T q Join the column set Q, and the set of indexes for each feature-data column in the column set Q. The system retrieves all data columns that meet the similarity threshold θ, and then constructs a query sample data table T. q The sample candidate connection set R q ; Step 2.2: Based on the sample candidate connection set R q Retrieves all data columns in the join column set Q that are matched with the data in the query sample table T. q For columns that can be joined, a join path exists in the graph index G, and the following conditions are met: Data table combination p k This leads to the construction of a candidate data table combination set. PC k It is all combinations of data tables that can be joined. k The set consisting of the join paths of all data tables in the dataset; Step 2.3: Combine the candidate data table into set PT. q Each data table combination p in k and p k The corresponding set of connection paths PC k Get data table combination p k Query sample data table T q The set of differing row tuples, Diff(T) q ,p k The specific steps include the following: Step 2.3.1: For the set of connection paths PC k Each link path in x Select the connection path in sequence. x The included edges <T i ,T j >and connect data table T i T j The connection conditions and connection methods are as follows: Then obtain the connection path. x The set of row tuples PT(path) on the join column set Q after combining all the data tables x ): Where |V(path) x | indicates the number of vertices on the path; Step 2.3.2: Based on the connection path set PC k The set of row tuples PT(path) formed on each path x ), calculate the combination p of the data tables. k Query sample data table T q The set of differing row tuples: Step 2.4: Based on the given budget B, find the candidate data table combination set PT. q A subset RP is given such that the subset RP satisfies the following condition: Then construct the result set R = {(p k, PC k )|p∈RP}, and return the result set R.

2. The method for maximizing the difference of a data table combination query for a related dataset according to claim 1, characterized in that: In step 1.1, for each data table T in the tabular dataset C... i Construct a feature-data column index set Specifically, the steps are as follows; Step 1.1.1: For the tabular dataset C, for the data table T i In the j-th column, data from table T is extracted using a pre-defined feature extraction algorithm. i The feature set of the j-th column Where L is the number of feature types; Step 1.1.2: For the tabular dataset C, for each feature type f l Constructing a feature-data column index I based on locality-sensitive hashing l ; Step 1.1.3: For each column of data characteristics Construct a data column information (T) i The index entries of the corresponding feature values ​​(j) are added to the corresponding feature-data column index I. l In this process, a feature-data column index set is then constructed.

3. The method for maximizing the difference of a data table combination query for a related dataset according to claim 1, characterized in that: In step 1.2, for each data table T in the tabular dataset C... i and candidate connection set R i Therefore, the candidate connection set R i Each candidate table T in k Select a set of best connections <T k ,c x ,FK(c x The specific steps include the following: Step 1.2.1: For the candidate connection set R i Each candidate table T in k In the candidate connection set R i Filter out all data from table T i The data columns in the table point to candidate table T. k Joining data columns in R ik ={ <T k ,c x ,FK(c x )>|c x ∈T i ,FK(c x )∈T k }; Step 1.2.2, for connecting R ik Each group of connections <c x ,FK(c x )>, calculate its connection score: FK_SCORE(c x ,FK(c x ))=(e(c x ,FK(c x ))+j(c x ,FK(c x ))*max{u(c x ),u(FK(c x ))} Where e(c) x ,FK(c x ) is the column matching score calculation function, j(c x ,FK(c x The matching score for each data element in the two columns is calculated as follows: Where u(c) x ), u(FK(c x The uniqueness scores of the two columns are calculated as follows: Among them, unique(c x ) represents c x The set of unique values ​​in a column, u(FK(c x )) represents FK(c x The set of unique values ​​in a column; Steps 1, 2, and 3: Select FK_SCORE(c x ,FK(c x The largest connection pair <c x ,FK(c x As data table T i To candidate table T k The best connection.

4. The method for maximizing the difference of a data table combination query for a related dataset according to claim 1, characterized in that: Step 2.4 Based on the given budget B, construct the candidate data table combination set PT. q A subset RP that satisfies the given conditions is constructed as follows: Step 2.4.1: Initialize the remaining budget b = B, and the set of selected row tuples RS = ∏ Q (T q ), Selected data table set TS, candidate data table combination list PS = { <p k ,g(p k )>|p k ∈PT q } and the selected data table combination set RP, where g(p k The marginal benefit of this data table combination is calculated as follows: Step 2.4.2: Select the data table combination p from the candidate data table combinations in turn, ensuring that the marginal revenue is the highest and the cost is lower than the remaining budget b. k Add the selected data table set RP to the data table set RP, and update the selected data table set TS = TS∪p in sequence. k Remaining budget b = B - ∑ T∈TS price(T), the set of selected row tuples RS = RS∪Diff(RS, p) k This updates the marginal revenue of each remaining candidate table combination in the candidate data table combination list PS. Step 2.4.3: If there is no data table combination that meets the conditions in the candidate data table combination list PS, return the data table combination set RP.

Images