Entity matching prediction method, device and storage device for relational database

Abstract

The present invention provides a relational database entity matching prediction method, device, and storage device. Two relational databases are arbitrarily selected, and each data in the relational database is converted into a string to obtain corresponding string sets A and B; a string set C is defined and initialized; an element in set C is randomly selected, and a TF-IDF operation is performed on it with all elements in set B to obtain several candidate data pairs. The element is deleted from set C, and the operation of step S3 is repeated until set C is empty, thereby obtaining a set of candidate data pairs; all candidate data pairs are feature-encoded to construct training samples; a prediction model is constructed using a maximum entropy algorithm, and a final prediction model is obtained after training. The data pairs are judged to generate a weighted graph, and the minimum cut set of the weighted graph is calculated. The edges of the cut set are the pairing results. The beneficial effect of the present invention is that entity matching can be performed efficiently and accurately.

Description

Technical Field

[0001] The present invention relates to the field of data fusion technology, and in particular to a relational database entity matching prediction method, device and storage device. Background Art

[0002] Relational databases are widely used in modern information systems. As an information management tool, they store structured data in a tabular format, providing powerful query and data manipulation capabilities, making data storage and use more efficient. Data fusion is the process of combining data from different data sources and formats to form a unified, complete dataset. Data from different data sources may cover different aspects and dimensions. Merging data from two different relational databases into a new dataset presents a challenge, and entity matching plays a key role in this data fusion process.

[0003] Entity matching refers to the process of identifying identical or similar entities across multiple databases. Its results are of great significance for tasks such as data cleaning and data integration. However, due to the particularity of relational data, accurate and efficient entity matching in relational databases is very challenging. Differences in naming conventions, data redundancy, errors, and other issues pose significant difficulties for entity matching. For example, the entity "China University of Geosciences" in one database may refer to the same location as the entity "Geological University" in another database. Due to the difference in naming, it is difficult to accurately match the two entities. In addition, entity matching often requires processing data from large-scale databases. Modern relational databases typically store large amounts of data. Therefore, in the entity matching process, efficient algorithms and technologies are needed to process this large amount of data. If the entity matching algorithm is inefficient, the matching task will not meet the timeliness requirements.

[0004] Nowadays, natural language processing methods have been widely used in word matching and semantic understanding of articles. However, due to the particularity of relational databases, each record is an independent data without contextual connection. It is difficult for natural language processing methods to judge the matching of two data from different databases.

[0005] Therefore, to address the accuracy and efficiency issues of entity matching in relational databases, this paper proposes a relational database entity matching prediction method. This method overcomes challenges such as data quality, large-scale data computation, and semantic differences, improving both accuracy and efficiency in database entity matching tasks. Summary of the Invention

[0006] In order to solve the above problems, for entities in relational databases (one data is an entity), the present invention provides a relational database entity matching prediction method, device and storage device. First, data conversion and synthesis are performed, each data in the relational database is converted into a string, each string is feature-encoded, a one-dimensional vector set is generated, and candidate data pairs are generated based on the one-dimensional vector set. The data is judged by the pairing function, and the matching data pairs are added to the sample data set. The prediction model is constructed using the maximum entropy algorithm to obtain a prediction function, which is used to predict whether a new data pair matches. Finally, matching prediction is performed, and the data pair is judged according to the prediction function, and the probability of matching the two data is used as a weight to generate a weighted graph with vertices as data and edges as matching probabilities. The minimum cut set of the graph is calculated, and the edges of the cut set are returned as pairing results to complete the matching prediction of the relational database.

[0007] A relational database entity matching prediction method mainly includes:

[0008] S1: Randomly select two relational databases, convert each data in the relational database into a string, and obtain the corresponding string sets A and B;

[0009] S2: Define a string set C and initialize it so that it contains all elements in set A.

[0010] S3: Randomly select an element from set C and perform a TF-IDF operation on it with all elements in set B to obtain several candidate data pairs. Then delete the element from set C. Repeat step S3 until set C is empty, and obtain a set of candidate data pairs.

[0011] S4: performing feature encoding on all candidate data pairs in the candidate data pair set to construct training samples;

[0012] S5: Use the maximum entropy algorithm to build a prediction model, and use the training samples to train the prediction model to obtain the final prediction model;

[0013] S6: According to the final prediction model, the data pairs are judged, a weighted graph is generated, and the minimum cut set of the weighted graph is calculated. The edges of the cut set are the pairing results, and the matching prediction of the relational database is completed.

[0014] 7. Furthermore, the process of generating the candidate data pair set is as follows:

[0015] 1) Define a candidate data pair set CandidatePairs and set it to an empty set;

[0016] 2) Define a distance T, which is the threshold of the TF-IDF cosine similarity between two strings, to determine whether the data corresponding to the two strings can be used as a candidate data pair;

[0017] 3) Loop through the set C. When C is not an empty set:

[0018] (1) Randomly select a data c from the set C i The corresponding string;

[0019] (2) The string is compared with all the strings in set B based on TF-IDF, and the inverted index is used to improve the operation efficiency. The data b1, b2, ... corresponding to the elements in set B that meet the conditions, that is, the operation results are less than T, are compared with data c. i Generate candidate data pairs (c i ,b1)、(c i ,b2)..., where b1, b2...∈B1;

[0020] (3) All candidate data pairs (c i ,b1)、(c i ,b2)...Add to the CandidatePairs set;

[0021] (4) Delete data c from set C i The corresponding string;

[0022] 4) When the set C is an empty set, the candidate data pair set CandidatePairs is obtained.

[0023] Furthermore, the prediction model is h(a,b), where a is the data corresponding to the string in set A, and b is the data corresponding to the string in set B. h(a,b)∈{0,1} indicates whether the data pair (a,b) matches. When the pairing function is 1, the data pair (a,b) matches, otherwise, the data pair (a,b) does not match.

[0024] Furthermore, the specific process of generating the weighted graph G is as follows: take the elements in the data pair as vertices, use the prediction model h(a,b) to judge the data pair, if h(a,b) is true, then add an edge between vertices a and b, and use the probability c(a,b) of (a,b) matching as the weight of the edge, thereby generating a weighted graph G with vertices as data and edges as matching probabilities based on the elements in database A and database B, and calculating the minimum cut set of the weighted graph G, and the edges of the cut set are the pairing results.

[0025] A storage device stores instructions and data for implementing a relational database-oriented entity matching prediction method.

[0026] A relational database-oriented entity matching prediction device comprises: a processor and the storage device; the processor loads and executes instructions and data in the storage device to implement a relational database-oriented entity matching prediction method.

[0027] The technical solution provided by the present invention has the following beneficial effects: the present invention arbitrarily selects two relational databases, converts each data in the relational databases into a string, and obtains corresponding string sets A and B; defines a string set C and initializes it; randomly selects an element from set C, performs a TF-IDF operation on it with all elements in set B, obtains several candidate data pairs, deletes the element from set C, repeats step S3 until set C is empty, and obtains a set of candidate data pairs; performs feature encoding on all candidate data pairs to construct training samples; uses the maximum entropy algorithm to construct a prediction model, obtains the final prediction model after training, judges the data pairs, generates a weighted graph, calculates the minimum cut set of the weighted graph, and the edges of the cut set are the pairing results. First, when constructing the prediction model, candidate data pairs are used instead of all data pairs, which greatly improves computational efficiency. Second, confidence is used to generate weights for the edges of the graph, transforming the unweighted graph into a weighted graph. Finally, the edges of the cut set are returned as pairs, improving computational accuracy. Such improvements help cope with matching tasks for large-scale datasets and improve the performance of the entire matching system. The present invention can perform entity matching efficiently and accurately, and provides an effective solution for entity matching prediction in relational databases. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0029] Figure 1 This is a flowchart of a relational database entity matching prediction method in an embodiment of the present invention.

[0030] Figure 2 It is a schematic diagram of the operation of the hardware device in the embodiment of the present invention. DETAILED DESCRIPTION

[0031] In order to have a clearer understanding of the technical features, purposes and effects of the present invention, specific embodiments of the present invention are now described in detail with reference to the accompanying drawings.

[0032] Embodiments of the present invention provide a relational database entity matching prediction method, device and storage device.

[0033] Please refer to Figure 1 , Figure 1 This is a flowchart of a relational database entity matching prediction method according to an embodiment of the present invention, which specifically includes:

[0034] S1: Randomly select two relational databases, convert each data in the relational database into a string, and obtain the corresponding string sets A and B;

[0035] S2: Define a string set C and initialize it so that it contains all elements in set A. Set C is used to store strings that may generate candidate data pairs.

[0036] S3: Randomly select an element from set C and perform a TF-IDF operation on it with all elements in set B to obtain several candidate data pairs. Then delete the element from set C. Repeat step S3 until set C is empty, and obtain a set of candidate data pairs.

[0037] S4: performing feature encoding on all candidate data pairs in the candidate data pair set to construct training samples;

[0038] S5: Use the maximum entropy algorithm to build a prediction model, use the training samples to train the prediction model, and obtain the final prediction model to predict whether the data pair matches;

[0039] S6: According to the final prediction model, the data pairs are judged, a weighted graph is generated, and the minimum cut set of the weighted graph is calculated. The edges of the cut set are the pairing results, and the matching prediction of the relational database is completed.

[0040] The specific operation process is as follows:

[0041] 1. Data Preprocessing

[0042] Preprocessing consists of three steps: data conversion and synthesis, feature encoding, and candidate data pair set generation. Conversion and synthesis converts the data in the relational database into the data format required by this invention to facilitate feature encoding. Feature encoding converts text data into numerical features that can be processed by machine learning algorithms. Feature encoding is then performed on candidate data pairs to improve computational efficiency.

[0043] 1.1 Conversion and synthesis of data in relational databases

[0044] First, each data item in the relational database is converted into a string. All the data in a database constitutes a string set. In the subsequent entity matching, the strings in the two string sets generated by the two databases are matched against each other.

[0045] 1.2 Generation of candidate data pair sets

[0046] A data entry in database A1 and a data entry in database B1 create a data pair. The purpose of building a prediction model is to determine whether this data pair matches. Without candidate data pairs, the computational complexity would be enormous, resulting in low efficiency. For example, if there are n data entries in database A1 and m data entries in database B1, there would be n*m ​​data pairs. If candidate data pairs are generated, the number of candidate pairs will be less than n*m, significantly improving computational efficiency.

[0047] The process of generating the candidate data pair set is as follows:

[0048] 1) Define a candidate data pair set CandidatePairs and set it to an empty set;

[0049] 2) Define a distance T, which is the threshold of the TF-IDF cosine similarity between two strings, to determine whether the data corresponding to the two strings can be used as a candidate data pair;

[0050] 3) Loop through the set C. When C is not an empty set:

[0051] (1) Randomly select a data c from the set C i The corresponding string;

[0052] (2) The string is compared with all the strings in set B based on TF-IDF, and the inverted index is used to improve the operation efficiency. The data b1, b2, ... corresponding to the elements in set B that meet the conditions, that is, the operation results are less than T, are compared with data c. i Generate candidate data pairs (c i ,b1)、(c i ,b2)..., where b1, b2...∈B1;

[0053] (3) All candidate data pairs (c i ,b1)、(c i ,b2)...Add to the CandidatePairs set;

[0054] (4) Delete data c from set C i The corresponding string;

[0055] 4) When the set C is an empty set, the candidate data pair set CandidatePairs is obtained.

[0056] 1.3 Feature Encoding

[0057] In the present invention, the string pairs corresponding to the data pairs are feature-encoded according to the features in Table 1, a data pair (a, b) is encoded into a one-dimensional vector (true is 1, false is 0), and all data pairs (i.e., data pair sets) are programmed into a one-dimensional vector set.

[0058] Table 1 Features used to learn pairing functions

[0059] feature coding Substring matching True if one of the two strings is a substring of the other, otherwise false Prefix matching True if one string is a prefix of the other, false otherwise Edit distance True if the edit distance between two strings is less than the set value, otherwise false Tag Matching True if the nth token in string a matches a token in string b, false otherwise Jaccard distance True if the Jaccard distance between two strings is less than the set value, otherwise false Number Matching True if both strings contain the same number, false otherwise

[0060] In tag matching, you can set multiple tags or joint tag matching. When the nth tag and the mth tag in string a both match two tags in string b, the condition is true, otherwise it is false.

[0061] 2. Building a Prediction Model

[0062] After feature encoding of the generated candidate data pairs, training samples can be constructed.

[0063] The maximum entropy algorithm is used to construct a prediction model h(a,b), where a represents the data corresponding to the elements in set A and b represents the data corresponding to the elements in set B. h(a,b)∈{0,1} represents whether the data pair (a,b) matches. The prediction model is trained using training samples. After training, the final prediction model is obtained, which is used to predict whether two pieces of data match.

[0064] The training process is as follows:

[0065] (1) Construct training samples S for the prediction model h(a,b). The process of constructing training samples S is as follows:

[0066] (1.1) Set the training sample S to an empty set;

[0067] (1.2) Based on the previously obtained method for generating candidate data pair sets, databases A1 and B1 generate a candidate data pair set CandidatePairs. For each element in the set CandidatePairs, if zi(a) = zi(b), the matching value label(a, b) is 1, and the candidate data pair matches; otherwise, it is 0, and the candidate data pair does not match. zi is a function that determines whether a and b match; i represents the i-th loop, i = 1, 2, ..., m, and m represents the maximum number of loops.

[0068] (1.3) Add the data pairs (a, b) with label (a, b) value 1 to the training sample S, and make the samples in S have labels.

[0069] (2) Use the training sample S to train the prediction model h(a, b), and mark the data pair (a, b) as 1 or 0, where 1 indicates a match and 0 indicates a mismatch.

[0070] When h(a,b) is 1 during the calculation process, the probability of matching the data pair (a,b) is recorded as c(a,b) for subsequent predictions.

[0071] 3. Matching Prediction

[0072] Matching prediction is the core task of the present invention, which aims to match corresponding entities in two relational databases A and B. The present invention adopts a series of improvement measures to improve the efficiency and accuracy of matching prediction.

[0073] A weighted graph G is generated based on the set of candidate data pairs. G takes the elements in all candidate data pairs as vertices, and then judges all candidate data pairs based on the final prediction model h(a,b). If h(a,b) is true, an edge is added between vertices a and b, and the probability c(a,b) of matching (a,b) is used as the weight of the edge. In this way, a weighted graph G is generated based on the elements in database A and database B.

[0074] Calculate the minimum cut set of graph G and return the edges of the cut set as the pairing results. At this point, the matching data in database A and database B can be returned.

[0075] To facilitate understanding, the following example illustrates the complete process: Assume that relational database A contains four data items (a1, b1, c1, d1) and relational database B contains three data items (x1, y1, z1). First, convert the data in database A and database B into string sets A and B. Assume that set A is (a, b, c, d) and set B is (x, y, z). Then, define a string set C containing all elements in set A. Then, randomly select an element a from set C and perform a TF-IDF calculation on it with all elements in set B. Assume that the elements that meet the requirements are x and y, resulting in the data pairs (a1, x1) and (a1, y1). Then, remove element a from set C, leaving the remaining elements in set C as b, c, and d. Then, select another element from set C and perform a TF-IDF calculation on it with all elements in set B until set C is empty. All the pairs (a1, x1), (a1, y1), (b1, z1), (c1, x1) generated in this way are candidate data pairs, and feature encoding is performed on all candidate data pairs.

[0076] Then construct the training samples, train the prediction model, get the final prediction model, judge the data pairs according to the final prediction model, generate the weighted graph, and use the fact that the graph is a bipartite graph. Using greedy algorithm Calculate the minimum cut set of the weighted graph, and the edges of the cut set are the pairing results.

[0077] 4. Experimental Data and Analysis

[0078] In this embodiment, the data set and processing process are first described, and then the experimental results are described.

[0079] 4.1 Dataset and Processing

[0080] This experiment uses two datasets. The first dataset is the restaurant dataset, which contains 533 restaurants from one restaurant guide and 331 restaurants from a second guide. The second dataset is the park dataset, which contains two sets: the first set contains 388 national park names, and the second set contains 258 names. There are 241 matching names in these two sets.

[0081] Each data set is divided into two parts. First, the prediction model is built with the first part and tested with the second part. Then the prediction model is built with the second part and tested with the first part.

[0082] 4.2 Experimental Results

[0083] In this experiment, a comparative experiment is used to replace the edge weight c(a, b) of the weighted graph G with the TF-IDF distance and edit distance of the two data respectively. The evaluation indicators include accuracy, recall, and F1 value, where:

[0084]

[0085]

[0086]

[0087] Among them, TP is True positives (TP), which means that the prediction is positive and the actual result is positive; TN is True negatives (TN), which means that the prediction is negative and the actual result is negative; FP is False positives (FP), which means that the prediction is positive and the actual result is negative; FN is False negatives (FN), which means that the prediction is negative and the actual result is positive; P is Accuracy, R is Recall, and PR means P multiplied by R.

[0088] The experimental results are shown in Table 2 and Table 3:

[0089] Table 2 Experimental results: precision and recall

[0090]

[0091]

[0092] Table 3 Experimental results: F1 value

[0093]

[0094] It can be seen from the experimental results that the method of the present invention has comparable or better results than the other two alternative methods.

[0095] See Figure 2 , Figure 2 4 is a schematic diagram of the working of the hardware device of an embodiment of the present invention, wherein the hardware device specifically comprises: a relational database entity matching prediction device 401, a processor 402 and a storage device 403.

[0096] A relational database-oriented entity matching prediction device 401: The relational database-oriented entity matching prediction device 401 implements the relational database-oriented entity matching prediction method.

[0097] Processor 402: The processor 402 loads and executes the instructions and data in the storage device 403 to implement the relational database entity matching prediction method.

[0098] Storage device 403: The storage device 403 stores instructions and data; the storage device 403 is used to implement the relational database-oriented entity matching prediction method.

[0099] The beneficial effects of the present invention are as follows: the present invention arbitrarily selects two relational databases, converts each data in the relational database into a string, and obtains corresponding string sets A and B; defines a string set C and initializes it; randomly selects an element from set C, performs a TF-IDF operation on it with all elements in set B, obtains several candidate data pairs, deletes the element from set C, repeats the operation of step S3 until set C is empty, and obtains a set of candidate data pairs; performs feature encoding on all candidate data pairs and constructs training samples; uses the maximum entropy algorithm to construct a prediction model, obtains the final prediction model after training, judges the data pairs, generates a weighted graph, calculates the minimum cut set of the weighted graph, and the edges of the cut set are the pairing results. First, when constructing the prediction model, candidate data pairs are used instead of all data pairs, which greatly improves the computational efficiency. Secondly, the edges of the graph are weighted using confidence, and the unweighted graph is converted into a weighted graph. Finally, the edges of the cut set are returned as pairs, which improves the accuracy of the computation. Such improvements help to cope with the matching task of large-scale data sets and improve the performance of the entire matching system. The present invention can perform entity matching efficiently and accurately, and provides an effective solution for entity matching prediction in relational databases.

[0100] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A relational database entity matching prediction method, characterized by: include: S1: Randomly select two relational databases A1 and B1, convert each data in the relational database into a string, and obtain the corresponding string sets A and B; S2: Define a string set C and initialize it so that it contains all elements in set A. S3: Randomly select an element from set C and perform a TF-IDF operation on it with all elements in set B to obtain several candidate data pairs. Then delete the element from set C. Repeat step S3 until set C is empty, and obtain a set of candidate data pairs. S4: performing feature encoding on all candidate data pairs in the candidate data pair set to construct training samples; S5: Use the maximum entropy algorithm to build a prediction model, and use the training samples to train the prediction model to obtain the final prediction model; S6: According to the final prediction model, the data pairs are judged, a weighted graph is generated, and the minimum cut set of the weighted graph is calculated. The edges of the cut set are the pairing results, and the matching prediction of the relational database is completed.

2. The relational database entity matching prediction method according to claim 1, wherein: The process of generating the candidate data pair set is as follows: 1) Define a candidate data pair set CandidatePairs and set it to an empty set; 2) Define a distance T, which is the threshold of the TF-IDF cosine similarity between two strings, to determine whether the data corresponding to the two strings can be used as a candidate data pair; 3) Loop through the set C. When C is not an empty set: (1) Randomly select a data c from the set C i The corresponding string; (2) The string is compared with all the strings in set B based on TF-IDF, and the inverted index is used to improve the operation efficiency. The data b1, b2, ... corresponding to the elements in set B that meet the conditions, that is, the operation results are less than T, are compared with data c. i Generate candidate data pairs (c i ,b1)、(c i ,b2)..., where b1, b2...∈B1; (3) All candidate data pairs (c i ,b1)、(c i ,b2)...Add to the CandidatePairs set; (4) Delete data c from set C i The corresponding string; 4) When the set C is an empty set, the candidate data pair set CandidatePairs is obtained.

3. The relational database entity matching prediction method according to claim 1, wherein: The prediction model is h(a,b), where a is the data corresponding to the string in set A, and b is the data corresponding to the string in set B. h(a,b)∈{0,1} indicates whether the data pair (a,b) matches. When the pairing function is 1, the data pair (a,b) matches; otherwise, the data pair (a,b) does not match.

4. The relational database-oriented entity matching prediction method according to claim 3, wherein: The specific process of generating the weighted graph G is as follows: take the elements in the data pair as vertices, use the prediction model h(a,b) to judge the data pair, if h(a,b) is true, then add an edge between vertices a and b, and use the probability c(a,b) of (a,b) matching as the weight of the edge, thereby generating a weighted graph G with vertices as data and edges as matching probabilities based on the elements in database A and database B, and use the greedy algorithm to calculate the minimum cut set of the weighted graph G, and the edges of the cut set are the pairing results.

5. A storage device, characterized in that: The storage device stores instructions and data for implementing the relational database-oriented entity matching prediction method according to any one of claims 1 to 4.

6. A relational database entity matching prediction device, characterized by: include: A processor and a storage device; the processor loads and executes instructions and data in the storage device to implement the relational database entity matching prediction method according to any one of claims 1 to 4.

Images