Similarity evaluation method of approximately duplicate records

Abstract

The present invention discloses a similarity evaluation method of approximately duplicate records. The similarity evaluation method comprises: step 1, partitioning a large data set waiting for duplicate removing, and obtaining a plurality of smaller data blocks; step 2, with respect to each data block, initializing similarity between an attribute layer and a recording layer; step 3, if iteration stop conditions are not satisfied, using the similarity of the recording layer to update the similarity of the attribute layer, and using the similarity of the attribute layer to update the similarity of the recording layer; and step 4, outputting the similarity between the attribute layer and the recording layer. According to the similarity evaluation method provided by the present invention, the similarity is iteratively spread in the attribute and recording layers, so that the problem that the records have missing values and noise values in practical production is solved, and the similarity of records is evaluated more accurately. The similarity evaluation method provided by the present invention is unsupervised, the cost of marking data is reduced, and the output can be flexibly integrated in conventional duplicate removing system frames based on aggregation or distance.

Description

technical field [0001] The invention relates to a near-duplicate record recognition technology under big data, and to a method for evaluating similarity between records. Background technique [0002] In the era of big data, integrating data from various sources is the most basic part of generating data value, and the deduplication of near-duplicate record identification is the core step. Usually, a record usually consists of multiple attribute values. Existing recognition methods can be mainly classified into the following categories: (1) methods based on probability matching, which use conditional independence assumptions or generalized expectation maximization (EM, Expectation Maximization) algorithm to infer the probability of matching between a single record pair, each observation value is the value of the attribute in the record; (2) distance-based method, which uses different similarity measures to calculate the similarity between attribute layers And by setting diffe...

AI Technical Summary

Problems solved by technology

However, each method is only effective for specific variable types, and it does not work well for missing or noisy values, especially for data on the Internet.

Images