A duplicate data fusion detection method based on pre-trained active learning

Abstract

This invention discloses a duplicate data fusion detection method based on pre-trained active learning. The method includes generating candidate pairs in a labeled dataset and serializing these candidate pairs to obtain a serialized dataset; inputting the serialized dataset into a NER model to identify known types and using regular expressions to identify specific types; preprocessing the serialized dataset; inputting the preprocessed serialized dataset into a pre-trained BERT model, selecting the most uncertain data in the unlabeled dataset for labeling, and then inputting the labeled data into the labeled dataset; using the R-Drop strategy to augment the labeled dataset from step 4; and iterating through the augmented labeled dataset to obtain the final labeled dataset. This invention combines active learning algorithms, selecting the most valuable data for manual annotation, reducing the cost of manual annotation, and rapidly improving model quality with a small amount of data.

Description

Technical Field

[0001] This invention specifically relates to a method for detecting duplicate data fusion based on pre-trained active learning. Background Technology

[0002] In the era of big data, data quality issues are becoming increasingly prominent, particularly duplicate data entry and similar duplicate records resulting from merging multiple databases. These "dirty data" problems severely limit the effective application of big data. Therefore, duplicate data detection has become a widely discussed and researched topic. Commonly used algorithms for duplicate data detection include fuzzy matching algorithms, SNM-based algorithms, clustering-based algorithms, and neural network-based algorithms. Fuzzy matching algorithms primarily select one or more keywords, and for each data point, calculate its similarity to other data using a string similarity formula. If the similarity exceeds a threshold, it is considered duplicate data. Commonly used algorithms include the Smith-Waterman algorithm and the N-Gram algorithm. SNM-based algorithms mainly consist of three steps: keyword selection, sorting, and merging. However, both of these methods suffer from a problem: difficulty in selecting keywords. If the keywords are not chosen properly, it will not only directly affect the detection efficiency, but also greatly affect the accuracy of the results. Clustering-based methods, such as Dedupe, mainly use support vector machines to achieve accurate distance estimation between records composed of multiple fields. Finally, hierarchical clustering is used, and the distance metric adopts the idea of ​​"average metric" to cluster those that are close together. However, this method cannot understand the semantic information between data well.

[0003] In recent years, with the development of deep learning, many neural network-based algorithms for detecting duplicate records have been proposed, such as Sentencebert: this model is a variant of the BERT model, training sentence embeddings for sentence similarity search. The trained model generates a high-dimensional (e.g., 768 for BERT) vector for each record, using it and the vector's cosine similarity to find duplicate data. Transformer-based pre-trained language models (TPLMs), such as BERT and RoBERTa, have shown good performance in a wide range of NLP tasks. These models are pre-trained using large unlabeled text corpora, such as Wikipedia. Because the Transformer architecture computes token embeddings from all tokens in the input sequence, the embeddings generated by the pre-trained model are more context-sensitive and fully understand semantics than other methods, such as word2vec, GloVe, or FastText. In entity matching, the pre-trained model has been shown to understand contextual information well and perform well on dirty datasets. However, all of the above neural network-based methods require a large amount of labeled data for training.

[0004] In summary, traditional duplicate data identification models typically determine the similarity between different columns of data. This is not only time-consuming but also highly dependent on the similarity between attributes, without considering the semantic relationships between attributes. Furthermore, in real-world business scenarios, manually acquiring labeled duplicate data for training and validating the model is very costly. Summary of the Invention

[0005] The present invention aims to overcome the shortcomings of the existing technology and provides a method for detecting duplicate data fusion based on pre-trained active learning.

[0006] This invention addresses the aforementioned technical problems and provides a method for detecting duplicate data fusion based on pre-trained active learning, comprising the following steps:

[0007] Step 1: First, use the Block strategy to generate candidate pairs in the labeled dataset, and then serialize the candidate pairs to obtain the serialized dataset;

[0008] Step 2: Input the serialized dataset into the NER model to identify known types, and use regular expressions to identify specific types;

[0009] Step 3: Preprocess the serialized dataset;

[0010] Step 4: Input the preprocessed serialized dataset into the pre-trained model BERT, select the data that is most uncertain in the unlabeled dataset for the current model, label it, and then input the labeled data into the labeled dataset.

[0011] Step 5: Use the R-Drop strategy to augment the labeled dataset from Step 4;

[0012] Step 6: Repeat steps 1-6 for the enhanced labeled dataset and continue iterating until the iteration ends to obtain the final labeled dataset.

[0013] A further technical solution is that, in step 1, serialization is performed according to the following formula;

[0014] serialize(e,e′)=[CLS]serialize(e)[SEP]serialize(e′)[SEP]

[0015] In the formula: serialize(e,e′) is the serialized data.

[0016] A further technical solution is that the NER model in step 2 is:

[0017] {start,end,label}=NER(serialize(e,e′))

[0018] In the formula: serialize(e,e′) is the serialized data; start is the starting position of the entity; end is the ending position of the entity; label is the label of the entity being identified.

[0019] A further technical solution is that, in step 2, [LAST][ / LAST] is used to mark the identified entity fragments;

[0020] serialize(e,e′) start =[LAST]

[0021] serialize(e,e′) end =[ / LAST]

[0022] In the formula: serialize(e,e′) is the serialized data; start is the starting position of the entity; end is the ending position of the entity.

[0023] A further technical solution is that the preprocessing formula in step 3 is:

[0024] x=MAX_TFIDF(serialize(e,e′)) j ,0 <j<n

[0025] In the formula: serialize(e,e′) is the serialized data; x is the serialized entry obtained after preprocessing; n is the maximum string length of the model input.

[0026] A further technical solution is that the formula in step 4 is:

[0027] y cls = Bert_CLS(x i )

[0028] P = softmax(w*y) cls +b)

[0029] In the formula: x i This refers to the serialized data obtained after preprocessing; y cls [CLS] is the output of BERT; w and b are the weights and biases of the linear layer, respectively.

[0030] A further technical solution is that the training loss function in step 6 is:

[0031]

[0032]

[0033]

[0034] In the formula: α is a hyperparameter.

[0035] The beneficial effects of this invention are:

[0036] I. This invention proposes a pre-trained duplicate data detection model based on active learning. This model is based on a pre-trained Transformer language model and fine-tunes the model by treating duplicate data deletion as a sequence for classification problems, thereby enabling the model to have better semantic understanding.

[0037] Second, this invention also incorporates an active learning algorithm to select the most valuable data for manual annotation, thereby reducing the cost of manual annotation and quickly improving the quality of the model with a small amount of data.

[0038] Third, use R-Drop to augment the labeled data in each round. Compared with traditional methods such as deletion and modification, R-Drop can better ensure the semantic correctness of the augmented data, making the trained model more robust. Attached Figure Description

[0039] Figure 1 This is the model design architecture diagram of this method;

[0040] Figure 2 These are histograms of recall for duplicate data detection from different models;

[0041] Figure 3 It is an F1 histogram for detecting duplicate data from different models. Detailed Implementation

[0042] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0043] like Figure 1 As shown, the present invention provides a duplicate data fusion detection method based on pre-trained active learning, comprising the following steps:

[0044] Step 1: First, use the Block strategy to generate candidate pairs (which may contain two duplicate records) in the labeled dataset (the initial labeled dataset may be small and will be expanded as active learning iterates), and then serialize the candidate pairs to obtain the serialized dataset.

[0045] The Block strategy used in this invention requires that at least one word be common between candidate pairs. For the generated candidate pairs (e, e′), they need to be serialized into data that the model can use for classification. Each entity e = {attr} i ,val i} 0<i≤m Serialization is performed according to the following formula, where the [COL] prefix represents the attribute name and the [VAL] prefix represents the attribute value;

[0046] serialize(e)=[COL]attr1[VAL]val1...[COL]attr m [VAL]val m

[0047] Where m represents the number of attributes of e;

[0048] For candidate pairs (e, e′), serialization is performed using the following formula:

[0049] serialize(e,e′)=[CLS]serialize(e)[SEP]serialize(e′)[SEP]

[0050] In the formula: serialize(e,e′) is the serialized data;

[0051] [CLS] is a special label in BERT, and this vector will be input into a fully connected layer for classification.

[0052] Step 2: Input the serialized dataset into the NER model to identify known types, such as people, dates, or organizations, and use regular expressions to identify specific types, such as version information, phone number digits, etc.; for the serialized data serialize(e,e′), input it into the NER model and obtain the following output:

[0053] {start,end,label}=NER(serialize(e,e′))

[0054] In the formula: serialize(e,e′) is the serialized data; start is the starting position of the entity; end is the ending position of the entity; label is the label of the entity being identified;

[0055] For each entity identified by NER and regular expressions, the identified entity fragments are marked with [LAST][ / LAST], as shown in the following formula:

[0056] serialize(e,e′) start =[LAST]

[0057] serialize(e,e′) end =[ / LAST]

[0058] In the formula: serialize(e,e′) is the serialized data; start is the starting position of the entity; end is the ending position of the entity;

[0059] Step 3: Preprocess the serialized dataset;

[0060] Next, we summarize the serialized data. If the serialized entries are too long, it hinders the model from learning important knowledge, and the BERT model has its own input limitations (up to 512 sub-word tokens). This invention selects to retain words with higher TF-IDF scores, as shown in the following formula:

[0061] x=MAX_TFIDF(serialize(e,e′)) j ,0 <j<n

[0062] In the formula: serialize(e,e′) is the serialized data; x is the serialized entry obtained after preprocessing, which is used as the input of the BERT model; n is the maximum string length of the model input;

[0063] Step 4: Input the pre-processed serialized dataset into the pre-trained model BERT, select the data that is most uncertain in the unlabeled dataset (the closer the model's output probability p is to 0.5, the greater the uncertainty of the model), label it, and then input the labeled data into the labeled dataset.

[0064] The data in the labeled dataset, after the above preprocessing, can be represented as D = {x} i ,y i}, where x i These are preprocessed data pairs, y i The fourth step involves using labels indicating whether they are duplicates (category labels 0, 1) to process the preprocessed string x. i Input the pre-trained BERT model; to make the pre-trained model suitable for binary classification tasks, add a simple fully connected layer and a softmax output layer, and use the [CLS] output of the BERT model as the input to the fully connected layer, as shown in the following formula:

[0065] y cls = Bert_CLS(x i )

[0066] P = softmax(w*y) cls +b)

[0067] In the formula: xi This refers to the serialized data obtained after preprocessing; y cls [CLS] is the output of BERT; w and b are the weights and biases of the linear layer, respectively;

[0068] Step 5: Use the R-Drop strategy to augment the labeled dataset from Step 4, helping the model perform better on dirty data, such as null values ​​and spelling errors. For the training data D = {x} i ,y i}, each training sample x i It will propagate forward twice through the network. Then, two output predictions are obtained: p1(y) i |x i ) and p2(y i |x i Because dropout randomly discards a portion of neurons each time, p1(y) i |x i ) and p2(y i |x i ) are two different predicted probabilities obtained from two different subnetworks (from the same model). R-Drop uses KL divergence to constrain p1(y i |x i ) and p2(y i |x i );

[0069] Step 6: Repeat steps 1-6 for the enhanced labeled dataset and continue iterating until the iteration ends to obtain the final labeled dataset;

[0070] The training loss function in step 6 is:

[0071]

[0072]

[0073]

[0074] In the formula: α is a hyperparameter.

[0075] To verify the effectiveness of the present invention, comparative experiments were conducted on four datasets; the first two datasets used in the experiment came from public benchmark datasets for entity resolution; the third dataset was provided in DedeUPE; and the last dataset was database information table data collected internally; details of these datasets are shown in Table 1.

[0076] Table 1 Description of the experimental dataset

[0077]

[0078]

[0079] To verify the deduplication performance of the proposed algorithm, PDDM-AL was compared with three benchmarks: Sentencebert, Dedupe, and Field-based similarity. Recall and F1 scores were used as model evaluation metrics. The ratio of training, validation, and test sets was 6:2:2, a 12-layer BERT layer was used, and α was set to 0.8 in all experiments. The batch size was 34.

[0080] The experiment was divided into two parts. The first part used the same amount of labeled data to train each model, validating the effectiveness of the proposed method (PDDM-AL) in duplicate data detection. In the second part, a small amount of labeled data was input into PDDM-AL for training, followed by an active learning loop where a small amount of data was added until convergence, validating that active learning can rapidly improve the model's accuracy with a small number of labels.

[0081] First, in the first part of the experiment, the model was trained using all the data in the training set until convergence, and the results are shown in Table 2:

[0082] Table 2 Experimental Results

[0083]

[0084] As shown in Table 2, our method (PDDM-AL) achieved the highest F1 score and recall in most cases, except in the Education dataset, where the F1 score lagged behind Dedupe by 0.0065, but its recall was 0.0322 higher than Dedupe. Compared to Sentencebert, PDDM-AL's F1 score was 0.009, 0.294, 0.246, and 0.056 higher than the previous sentence, respectively. The recall increased by 0.023, 0.287, 0.267, and 0.192, respectively.

[0085] The second part of the experiment aimed to verify the effectiveness of active learning. The F1 score and Recall of the model were recorded sequentially after each round of active learning (each round of active learning involved selecting 1000 pairs of data labels and inputting them into the model for training). The main results are as follows:

[0086] Table 3 shows the F1 scores for the three datasets in each round of active learning.

[0087] Dataset Round 1 Second round Third round Fourth round Fifth round Musicbrainz-20-A01 0.674 0.948 0.973 0.973 0.974 GeographicSettlements 0.798 0.936 0.961 0.961 0.961 Education 0.846 0.939 0.887 0.887 0.887

[0088] Table 4 shows the Recall of the three datasets in each round of active learning.

[0089] Dataset Round 1 Second round Third round Fourth round Fifth round Musicbrainz-20-A01 0.970 0.924 0.989 0.989 0.992 GeographicSettlements 0.708 0.935 0.953 0.953 0.953 Education 0.891 0.941 0.900 0.900 0.900

[0090] As shown in Tables 3 and 4, active learning enables the model to be trained using a small amount of labeled data, which can rapidly improve F1 and Recall. For example, on the GeographicSettlements dataset, only 1000 pairs of data were added for training between the first and second rounds, resulting in an increase of 0.138 and 0.227 in F1 and Recall, respectively. Across the three datasets, this model outperformed other models using only 3000 pairs of data. This is primarily because PDDM-AL's active learning strategy effectively selects the most valuable data for the model and hands it over to experts for labeling via a selector, thus allowing training on a small amount of the most valuable data and rapidly improving model performance.

[0091] In summary, this invention outperforms other models in terms of recall and F1 score for duplicate data identification. The main reasons are: first, the pre-trained model understands contextual information and semantics better than other methods; second, PDDM-AL incorporates domain knowledge, enabling it to better capture key information, and employs the R-Drop data augmentation strategy, allowing the model to maintain high F1 and recall even with dirty data.

[0092] The above description is not intended to limit the present invention in any way. Although the present invention has been disclosed through the above embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some changes or modifications to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the present invention shall still fall within the scope of the present invention.

Claims

1. A duplicate data fusion detection method based on pre-trained active learning, characterized in that, Includes the following steps: Step 1: First, use the Block strategy to generate candidate pairs in the labeled dataset, and then serialize the candidate pairs to obtain the serialized dataset; Serialization is performed according to the following formula; In the formula: This is the serialized data; Step 2: Input the serialized dataset into the NER model to identify known types, and use regular expressions to identify specific types; The NER model is as follows: In the formula: This is the serialized data; This is the starting position of the entity; This is the end position of the entity; The label for the entity being identified; Step 3: Preprocess the serialized dataset; Step 4: Input the preprocessed serialized dataset into the pre-trained model BERT, select the data that is most uncertain in the unlabeled dataset for the current model, label it, and then input the labeled data into the labeled dataset. Step 5: Use the R-Drop strategy to augment the labeled dataset from Step 4; Step 6: Repeat steps 1-5 for the enhanced labeled dataset and continue iterating until the iteration ends to obtain the final labeled dataset.

2. The duplicate data fusion detection method based on pre-trained active learning according to claim 1, characterized in that, In step 2, the identified entity fragments are marked with [LAST] [ / LAST]; In the formula: This is the serialized data; This is the starting position of the entity; This indicates the end position of the entity.

3. The duplicate data fusion detection method based on pre-trained active learning according to claim 1, characterized in that, The preprocessing formula in step 3 is: In the formula: This is the serialized data; x These are the serialized entries obtained after preprocessing; n The maximum length of the string input to the model.

4. The duplicate data fusion detection method based on pre-trained active learning according to claim 1, characterized in that, The formula in step 4 is: In the formula: x i This is the serialized data obtained after preprocessing; y cls [CLS] is the output of Bert; w and b These represent the weights and biases of the linear layer, respectively.

5. The duplicate data fusion detection method based on pre-trained active learning according to claim 1, characterized in that, The training loss function in step 6 is: In the formula: α is a hyperparameter.

Images