A candidate data screening method for assisting multi-source duplicate data deduplication

Abstract

The present invention discloses a candidate data screening method for assisting in deduplication of multi-source duplicate data, comprising constructing a multi-source duplicate data set; dividing it into a training data set and a test data set; extracting data samples from the training data set and capturing similarity features between the data samples; performing predictions to obtain predicted labels; establishing a connection graph between matching points; cleaning the connection graph; setting graph labels on the cleaned connection graph, where points in the same graph are regarded as similar duplicate data pointing to the same entity, comparing the graph predicted labels of the data with the labels predicted by the xgboost model, selecting data with inconsistent labels for manual re-labeling; until the number of cycles reaches a set value; and deduplicating redundant data. The invention can, in practical applications, well characterize the similarity of different length attribute values in data, and can also have a good recognition effect on duplicate data in terms of deduplication of multi-source duplicate data.

Description

Technical Field

[0001] The present invention specifically relates to a candidate data screening method for assisting multi-source duplicate data deduplication. Background Art

[0002] The advent of the big data era has enabled its application in various industries, undoubtedly bringing new opportunities and changes. However, the increase in data volume also presents challenges in how to utilize and process this data. For example, how to deduplicate data, supplement missing data, and correct data errors are all factors that must be considered in big data processing. This massive amount of data often requires specific processing, and only data that meets the requirements can be valuable. Choosing the right method for processing big data not only eliminates unvalued data, reducing the workload during data analysis, but also reduces errors caused by it in model training. In real-world production, duplicate data from multiple sources often leads to various problems in data analysis. Therefore, selecting the most valuable data for model training from massive amounts of data can reduce the consumption of manpower and resources. Therefore, selecting the most valuable data for model training through active learning is of great significance to big data processing.

[0003] In recent years, active learning-based models have shown significant advantages in solving big data processing problems. Committee-based active learning and margin-based active learning have been shown to significantly improve the efficiency of big data processing.

[0004] Reference [1] "Heterogeneous committee-based active learning for entity resolution" (Chen X, Xu Y, Broneske D, et al. European Conference on Advances in Databases and Information Systems.) shows that committee-based active learning has two shortcomings. First, the selected initial training data is usually not balanced enough and the amount of information is insufficient. Second, the committee is composed of homogeneous classifiers, including their accuracy to achieve the diversity of the committee, that is, the classifiers are not trained with all available training data or optimal parameter settings. Therefore, a committee-based active learning method HeALER is proposed. This method overcomes these two shortcomings by using a more effective initial training data selection method and a more effective heterogeneous committee, but this method does not solve the problem of applying active learning in a multi-source environment.

[0005] Document [2] "Graph-boosted active learning for multisource entity resolution"

[0006] (Primpeli A, Bizer C. International Semantic Web Conference. Springer, Cham, 2021: 182-199.) Graph active learning is proposed to solve entity recognition of duplicate data in multi-source situations. However, when there is insufficient data overlap, the graphs built by graph active learning often fail to achieve good screening results. Furthermore, when faced with multi-source duplicate data, how to retain and delete the identified duplicate data is also an issue that has not been widely discussed.

[0007] In terms of similarity calculation, the paper [3] “DEEPACTIVE LEARNING FOR NAMEDENTITY RECOGNITION” (Shen Y, Yun H, Lipton ZC, et al. arXiv preprint arXiv:1707.05928, 2017.) introduced a lightweight architecture, namely the CNN-CNN-LSTM model, which consists of a convolutional character and word encoder and a long short-term memory (LSTM) label decoder. However, there are still three major areas that need to be improved in deduplication when facing multi-source data: First, how to fully characterize the entity attribute values of multi-source data. A single similarity calculation method often does not perform well when facing data with both short and long attribute values; Second, multi-source data often has more complex matching results than single-source data. Choosing which active learning method to match data in a multi-source environment is a question worth considering; Third, there are many examples of frameworks for duplicate data identification that can be used as reference, but choosing which appropriate data to retain and delete in duplicate data is a problem that many current algorithms do not address.

[0008] In summary, the existing deduplication models have shortcomings in practical applications, such as incomplete feature extraction, little overlap in multi-source data and inability to filter data, due to incomplete characterization of attribute value similarity, the adaptability of active learning methods under multi-source data, and the uncertainty of deduplication methods. Summary of the Invention

[0009] The present invention mainly overcomes the shortcomings of the prior art and provides a candidate data screening method for assisting in deduplication of multi-source duplicate data.

[0010] The present invention solves the above technical problems and provides a technical solution: a candidate data screening method for assisting multi-source duplicate data deduplication, comprising the following steps:

[0011] Step 1: Preprocess the multi-source duplicate data and construct a multi-source duplicate dataset;

[0012] Step 2: Divide the multi-source repeated dataset into a training dataset and a test dataset;

[0013] Step 3: Extract data samples from the training data set and capture the similarity features between the data samples;

[0014] Step 4: Input the obtained similarity features into the xgboost model for prediction to obtain the predicted label;

[0015] Step 5: Establish a connection graph between matching points based on the predicted labels;

[0016] Step 6: Extract known mismatched data from the training dataset based on the connection graph and clean the constructed connection graph;

[0017] Step 7: Label the cleaned connection graph. Points in the same graph are considered to be similar duplicate data pointing to the same entity. Compare the graph prediction labels of the data with the labels predicted by the xgboost model. Select the data with inconsistent labels for manual re-labeling.

[0018] Step 8: The manually labeled data continues to be predicted by the xgboost model, and steps 4 to 8 are repeated until the number of cycles reaches the set value;

[0019] Step 9: Calculate the similarity between the identified duplicate data, select the best data to retain, and deduplicate the redundant data.

[0020] A further technical solution is that the preprocessing process in step 1 is to process outliers and fill in missing values.

[0021] A further technical solution is to use the maximum and minimum normalization method in step 2 to standardize the multi-source repeated data set, taking the first 70% of the multi-source repeated data set as the training data set and the last 30% as the test data set.

[0022] A further technical solution is that after setting the learning rate lr, the number of iterations n-epochs, the batch sample size batch-size in step 3, the maximum distance L between two points, and the weight M of the edge between the two points, batch-size samples are taken from the training data set in turn, and they are input into the similarity calculation framework to capture the similarity between attribute values.

[0023] A further technical solution is that the calculation formula for the similarity in step 3 is:

[0024]

[0025]

[0026]

[0027]

[0028] Where: X, Y represent the respective character strings; sim overlap 、sim jaccard 、sim jaro and sim lev There are four different similarity calculation formulas.

[0029] A further technical solution is to construct a connection graph between matching points based on the graph active learning framework in step 5: specifically, a data point is regarded as a point in the connection graph. If the two data points are predicted to have matching labels by the xgboost model, the two points are connected and the weight of the edge between the two points is calculated; if the predicted labels of the two points are not matched, there is no need to connect the two points.

[0030] A further technical solution is that the cleaning process in step 6 is: detecting whether there is a path between the unmatched data in the established connection diagram; if there is a connection, cutting the two points by the minimum cut method to ensure that there is no path between the two points.

[0031] A further technical solution is that in step 6, the distance threshold L set in advance and the edge weight threshold M are used to compare whether the distance between the points in the connection graph after the first cleaning exceeds the distance threshold L; if it exceeds the distance threshold L, then calculate whether the average weight of the edge on the route is lower than the set weight threshold M. If the above two conditions are met at the same time, the two points are considered to be suspicious points, and then the minimum cut method is used to cut the two points.

[0032] A further technical solution is that the formula for deleting duplicate data and selecting to retain in step 9 is:

[0033]

[0034] Where: w k is a point in the connection graph; k is the number of points in the connection graph.

[0035] Beneficial effects of the present invention:

[0036] First, we use a combination of traditional similarity algorithms and the deep learning Sentence-BERT model to characterize the similarity between data. When faced with inconsistent data lengths, traditional similarity algorithms are better at characterizing the similarity of short data, but when it comes to long attribute values, pre-trained deep learning models are more effective in characterizing semantic similarity.

[0037] Second, we applied the existing graph active learning framework to the new field of deduplication and made some innovations and improvements. The original graph active learning framework can only be used when there is duplication between training data and experimental data. We improved it for the case where the overlap is not high.

[0038] 3. In practical applications, the invention can well characterize the similarity of attribute values of different lengths in data, and can also have a good identification effect on duplicate data when deduplicating multi-source data. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is the SBERT architecture diagram with classification objectives;

[0040] Figure 2 This is an example diagram of using a graph to detect false positives;

[0041] Figure 3 This is the technical architecture diagram for the implementation of this method;

[0042] Figure 4 This is a flow chart of the method when deduplicating data;

[0043] Figure 5 is a flow chart for the implementation of this method;

[0044] Figure 6 This is the result of Experiment 1. DETAILED DESCRIPTION

[0045] The technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0046] like Figure 5 As shown, a candidate data screening method for assisting multi-source duplicate data deduplication of the present invention includes the following steps:

[0047] Step 1: Process the multi-source duplicate data, deal with outliers and fill in missing values, and construct a multi-source duplicate data set;

[0048] Step 2: Divide the multi-source repeated dataset into a training dataset and a test dataset;

[0049] The multi-source repeated data sets were normalized using the maximum and minimum normalization method, with the first 70% of the data set used as the training data set and the last 30% used as the test data set.

[0050] Step 3: Extract data samples from the training data set and capture the similarity features between the data samples;

[0051] Set the completion learning rate lr and the number of iterations n - epochs, after batch processing of sample size batch-size, the maximum distance L between two points, the weight M of the edge between two points, the number of samples of batch-size size are taken from the training dataset in sequence, and they are input into the similarity calculation framework to capture the similarity between attribute values. The conversion formula is as follows;

[0052]

[0053]

[0054]

[0055]

[0056] Where: X, Y represent the respective character strings; sim overlap 、sim jaccard 、sim jaro and sim lev There are four different similarity calculation formulas;

[0057] Step 4: Input the obtained similarity features into the xgboost model for prediction to obtain the predicted label;

[0058] Step 5: Establish a connection graph between matching points based on the predicted labels;

[0059] Based on the graph active learning framework, a connection graph between matching points is constructed. Specifically, a data point is regarded as a point in the connection graph. If the labels of two data points are predicted to match by the xgboost model, the two points are connected and the weight of the edge between the two points is calculated. If the predicted labels of the two points are not matched, there is no need to connect the two points.

[0060] Step 6: Extract known mismatched data from the training dataset based on the connection graph and clean the constructed connection graph;

[0061] The specific cleaning process is:

[0062] The first step is to select the data with mismatched labels in the labeled data and detect whether there is a path between the mismatched data in the established connection graph; if there is a connection, the two points are cut using the minimum cut method to ensure that there is no path between the two points;

[0063] Step 2: To ensure the effectiveness of cleaning the connection graph, if the data duplication between the labeled data and the established connection graph is low, the cleaning efficiency will be low;

[0064] Therefore, we use the pre-set distance threshold L and edge weight threshold M to compare whether the distance between points in the graph after the first cleaning exceeds the distance threshold L. If it exceeds the distance threshold L, we then calculate whether the average weight of the edges on the route is lower than the set weight threshold M. If both conditions are met, the two points are considered suspicious and the minimum cut method is used to cut the two points.

[0065] The Minimum Cut algorithm is a classic image segmentation algorithm. In a directed graph, the maximum flow from a source to a sink is equal to the minimum sum of the capacities of the edges that, if removed from the graph, would disrupt the network flow. In other words, in any network, the maximum flow is equal to the capacity of the minimum cut.

[0066] Step 7: Label the cleaned connection graph. Points in the same graph are considered to be similar duplicate data pointing to the same entity. Compare the graph prediction labels of the data with the labels predicted by the xgboost model. Select data with inconsistent labels for manual re-labeling. For data with inconsistent labels, we believe that such data has a high information content in active learning. To ensure fairness in data selection, we average the weights of false positive and false negative data to ensure that different data are selected fairly. For example, if there are ten false positive data and five false negative data, the probability of selecting a false positive data is one in ten, and the probability of selecting a false negative data is one in five.

[0067] Step 8: The manually labeled data continues to be predicted by the xgboost model, and steps 4 to 8 are repeated until the number of cycles reaches the set value;

[0068] Step 9: Calculate the similarity between the identified duplicate data, select the best data to retain, and deduplicate the redundant data;

[0069]

[0070] Where: w k is a point in the connection graph; k is the number of points in the connection graph.

[0071] Experiment 1:

[0072] This experiment evaluates the proposed model on four datasets, including the MusicBrainz dataset [4] “Using link features for entity clustering knowledge graphs” (Saeedi, A., Peukert, E., Rahm, E. ESWC 2018. LNCS, vol. 10843, pp. 576–592. Springer, Cham (2018). https: / / doi.org / 10.1007 / 978-3-319-93417-437), which is a classic dataset for multi-source entity recognition. The second dataset comes from the Magellan repository [5] "Magellan: toward building entity matching management systems over data science stacks" (Konda P, Das S, Doan AH, et al. Proceedings of the VLDB Endowment, 2016, 9 (13): 1581-1584), which provides a large number of matching tasks between two sources, and the last one [6] "The WDC training dataset and gold standard for large-scale product matching" (Primpeli, A., Peeters, R., Bizer, C. Companion Proceedings of WWW (2019)) comes from a subset of computer product records published by four e-commerce websites, as well as a variant with higher complexity and sparsity. These datasets consist of multiple tables from different data sources with the same attributes, and the F1 score and Precision score are used as evaluation indicators.

[0073] Experiment 1 mainly tests the advantages of combining traditional classic similarity algorithms with deep learning and traditional similarity algorithms in similarity calculation. Traditional similarity calculation only uses formulas such as edit distance, Jaccard coefficient or overlap. These formulas mainly calculate the similarity of attribute value characters, which has natural advantages for short attribute values, such as no need for training and convenient calculation. However, when faced with longer attribute values, it is impossible to calculate their semantic similarity well, so we propose to use a pre-trained deep learning model combined with the traditional similarity algorithm formula. The experimental results are as follows Figure 6 As shown:

[0074] Since only three of the four selected datasets have long attribute values, we chose three of them for the experiment. The three result graphs show that combining a deep learning model with a classic similarity algorithm yields superior F1 scores, achieving good results more quickly in fewer active learning rounds. Furthermore, for multiple rounds of active learning, the combined approach offers greater stability, avoiding significant fluctuations.

[0075] Experiment 2:

[0076] The data set used in this experiment is the same as that used in Experiment 1. The purpose of this experimental design is to test the similar repeated data matching effect of the method of the present invention. Based on the traditional similarity calculation formula combined with the pre-trained deep learning model, we use the conventional edge-based active learning method, the committee-based active learning method and the original graph active learning method to compare with our method, observe and record the repeated data matching experiments of our method, and obtain the experimental results in Tables 1 and 2 with a fixed number of iterations.

[0077] Table 1

[0078]

[0079]

[0080] Table 2

[0081]

[0082] As shown in Table 1, GDAL achieves the best accuracy scores on all three datasets as active learning cycles progress. As can be seen from the 75th and 150th iterations, GDAL converges faster than the other three baseline methods in most cases. For the fourth dataset, we also find that our model is not the best on this dataset due to the lower proportion of Matches data and a wider range of sparsity.

[0083] Experimental data shows that when the number of active learning operations reaches 200, the F1 scores for all tasks are 0 to 0.047 lower than the passive learning results obtained by training the matching model with all data, while the MB baseline methods are all lower than the GDAL method. Table 2 shows the accuracy scores of GDAL and the other three baseline methods across active learning iterations. At iteration 150, they outperform the ALMSER, QHC, and MB baseline methods by an average of 3.65%, 2.95%, and 3.85%, respectively.

[0084] In summary, this paper proposes for the first time a graph deep active learning framework for data deduplication, which extracts deep similarity features from multi-source data records based on a similarity algorithm combined with a BERT model, which is used to remove duplicate data that effectively retains the most information. Based on the experimental results on four multi-source tasks, it is shown that the feature extraction step combined with deep learning can better characterize the characteristics of different data records compared with the baseline method. At the same time, the improved graph active learning is more effective than the state-of-the-art committee-based query strategy and the baseline graph-enhanced active learning model. Future work will focus on improving model performance and optimizing model comparison experiments.

[0085] The above description does not limit the present invention in any form. Although the present invention has been disclosed through the above embodiments, it is not intended to limit the present invention. Any technician familiar with the profession can use the technical content disclosed above to make some changes or modifications to equivalent embodiments without departing from the scope of the technical solution of the present invention. However, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention are still within the scope of the technical solution of the present invention.

Claims

1. A candidate data screening method for assisting multi-source duplicate data deduplication, characterized in that: The following steps are involved: Step 1: Preprocess the multi-source duplicate data and construct a multi-source duplicate dataset; Step 2: Divide the multi-source repeated dataset into a training dataset and a test dataset; Step 3: Extract data samples from the training data set and capture the similarity features between the data samples; Step 4: Input the obtained similarity features into the xgboost model for prediction to obtain the predicted label; Step 5: Establish a connection graph between matching points based on the predicted labels; Specifically, a data point is regarded as a point in the connection graph. If the labels of two data points are predicted to match by the xgboost model, the two points are connected and the weight of the edge between the two points is calculated; if the predicted labels of the two points are not matched, there is no need to connect the two points. Step 6: Extract known mismatched data from the training dataset based on the connection graph and clean the constructed connection graph; The cleaning process is as follows: detecting whether there is a path between the unmatched data in the established connection graph; if there is a path, cutting the two points by the minimum cut method to ensure that there is no path between the two points; Step 7: Label the cleaned connection graph. Points in the same graph are considered to be similar duplicate data pointing to the same entity. Compare the graph prediction labels of the data with the labels predicted by the xgboost model. Select the data with inconsistent labels for manual re-labeling. Step 8: The manually labeled data continues to be predicted by the xgboost model, and steps 4 to 8 are repeated until the number of cycles reaches the set value; Step 9: Calculate the similarity between the identified duplicate data, select the best data to retain, and deduplicate the redundant data.

2. The candidate data screening method for assisting multi-source duplicate data deduplication according to claim 1, characterized in that: The preprocessing process in step 1 is to process outliers and fill in missing values.

3. The candidate data screening method for assisting multi-source duplicate data deduplication according to claim 1, characterized in that: In step 2, the multi-source repeated data set is standardized using the maximum and minimum normalization method, and the first 70% of the multi-source repeated data set is taken as the training data set, and the last 30% is taken as the test data set.

4. The candidate data screening method for assisting multi-source duplicate data deduplication according to claim 1, characterized in that: After setting the learning rate lr, the number of iterations n-epochs, the batch sample size batch-size, the maximum distance L between two points, and the weight M of the edge between two points in step 3, batch-size samples are taken from the training data set in turn and input into the similarity calculation framework to capture the similarity between attribute values.

5. The candidate data screening method for assisting multi-source duplicate data deduplication according to claim 1, characterized in that: The calculation formula of the similarity in step 3 is: Where: X, Y represent the respective character strings; sim overlap 、sim jaccard 、sim jaro and sim lev There are four different similarity calculation formulas.

6. The candidate data screening method for assisting multi-source duplicate data deduplication according to claim 1, characterized in that: In step 6, the distance threshold L set in advance and the edge weight threshold M are used to compare whether the distance between the points in the connection graph after the first cleaning exceeds the distance threshold L; if the distance threshold L is exceeded, the average weight of the edge on the route is calculated to see if it is lower than the set weight threshold M. If both conditions are met, the two points are considered suspicious points and the minimum cut method is used to cut the two points.

7. The candidate data screening method for assisting multi-source duplicate data deduplication according to claim 1, characterized in that: The formula for deleting duplicate data and selecting to retain in step 9 is: Where: w k is a point in the connection graph; k is the number of points in the connection graph.

Images