A method for quickly generating a pair of similar data object candidates

Abstract

The application discloses a kind of similar data object candidate pair fast generation method, comprising the following steps: to the structured data table of multiple need generation similar data object candidate pair is carried out mode alignment and inter-table joint, according to the characteristics of field value the field of joint table is divided into formatted field and unstructured field;Respectively for formatted field and unstructured field processing, respectively obtain block table and minhash vector;The information entropy of block table is calculated as the weight of intra-table block key, and the data object in joint table is taken as node, the sum of the weight of block key shared by object is taken as edge weight, and undirected graph is constructed;Based on the sum of block table information entropy, pruning threshold is calculated, the edge of undirected graph is pruned, and the node pair corresponding to the remaining edge is similar data object candidate pair.The application provides a kind of similar data object candidate pair fast generation method, can filter out more compact, higher quality candidate pair, significantly improve the efficiency of subsequent data object matching task.

Description

Technical Field

[0001] This invention relates to the field of big data processing, and in particular to a method for rapidly generating candidate pairs of similar data objects. Background Technology

[0002] In the era of big data, the amount of data carried by information systems of enterprises, governments, and various institutions is growing rapidly, and the sources of data are becoming increasingly diversified. To extract value from big data more efficiently, data cleaning and integration capabilities are indispensable for big data information systems. Entity resolution plays a crucial role in data integration, aiming to identify data objects pointing to the same real-world entity (such as users or products) within a single data source and across multiple data sources. Entity resolution requires pairwise matching of data objects in the data source, often resulting in extremely high computational costs and excessive time. Therefore, it is necessary to reduce the time consumed by entity resolution through pre-selection of similar data object candidate pairs. Current similar candidate pair selection techniques mainly fall into two categories: learning-based and non-learning-based. Learning-based methods often require pre-labeled datasets for training, and the models are difficult to transfer to data sources with different structures. Non-learning-based methods often directly use word or text slices as the standard for selecting candidate pairs, ignoring the information contained within them, and easily generating many redundant similar candidate pairs. Summary of the Invention

[0003] In view of the aforementioned shortcomings of existing technologies, the technical problem to be solved by this invention is that existing methods for selecting candidate pairs of similar data objects require pre-labeled datasets and training, and the models are difficult to transfer to data sources with different structures, or are prone to generating a large number of redundant candidate pairs. Therefore, this invention provides a method for rapidly generating candidate pairs of similar data objects based on locality-sensitive hashing and undirected graphs. This method enables the rapid generation of candidate pairs of similar data objects from multiple structured data sources, without relying on labeled data for training. It classifies fields according to the characteristics of values, uses locality-sensitive hashing to extract similarity features of data objects, and considers the differences in information content between different similarity features to make the similarity measurement more accurate. Furthermore, it uses undirected graph pruning to make the generated candidate pairs of similar data objects more concise and of higher quality, which helps to improve the efficiency of subsequent matching tasks.

[0004] To achieve the above objectives, the present invention provides a method for rapidly generating candidate pairs of similar data objects, comprising the following steps:

[0005] S1. Perform schema alignment and inter-table union on multiple structured data tables that are candidate pairs of similar data objects to be generated. Based on the characteristics of the field values, the fields of the union table are divided into two categories: formatted fields and unformatted fields.

[0006] S2. Process the formatted field and the unformatted field separately to obtain the minhash vector of all feature vectors under the formatted field and the unformatted field respectively;

[0007] S3. Based on the minhash vector, construct the minhash matrix of the field and divide it vertically into multiple local sensitive hash matrices. Use the index of the local sensitive hash matrix and the minhash segment as the block key and the list of object indexes as the value to construct a block table.

[0008] S4. Treat the set of block keys in the block table as the information source, calculate its information entropy, and use it as the weight of the block key;

[0009] S5. Traverse all block tables, using the objects in the list corresponding to the block key as nodes, and the sum of the weights of the co-occurring block keys between objects as the weights of the edges, to construct an undirected graph.

[0010] S6. Calculate the pruning threshold based on the sum of the information entropy of all block tables, prune the edges of the undirected graph, and the node pairs corresponding to the remaining edges in the graph are candidate pairs of similar data objects.

[0011] Furthermore, the structured data tables containing multiple candidate pairs of similar data objects are subjected to schema alignment and inter-table joins. Based on the characteristics of the field values, the fields of the joined table are divided into two categories: formatted fields and unformatted fields, specifically including:

[0012] The pattern alignment is set to filter out common fields with the same meaning across multiple structured data tables based on the field information of multiple structured data tables. For each structured data table, only the common fields and the primary key of that table are retained.

[0013] A table join is configured to combine multiple data tables vertically, merging the primary key columns of the original tables into a new column. A primary key is added to the joined table, representing the index of the data objects in the table.

[0014] The fields in the table are divided into two categories: fields whose values ​​follow a uniform format and have few category values ​​are called formatted fields, and fields whose values ​​have no fixed format and have a high degree of variation are called unformatted fields.

[0015] Furthermore, processing is performed separately for the formatted and unformatted fields to obtain the minhash vectors of all feature vectors under the formatted and unformatted fields, respectively. Specifically, this includes:

[0016] S21. For formatted fields, construct a partitioned table using the category value of the field value as the partitioning key and the list of object indexes as the value;

[0017] S22. For unformatted fields, treat the field values ​​as short texts to construct a corpus, use one-hot encoding to extract the bag-of-words vector of the field value, use the non-zero subscripts of the bag-of-words vector to construct the feature vector of the field value, and calculate the minhash vector of all feature vectors.

[0018] Furthermore, S22, for unformatted fields, the field values ​​are treated as short texts to construct a corpus. One-hot encoding is used to extract the bag-of-words vector of the field values. The non-zero indices of the bag-of-words vectors are used to construct the feature vector of the field value. The minhash vector of all feature vectors is calculated. Specifically, for each unformatted field, the field values ​​are treated as short texts. The set of field values ​​constitutes a corpus. After removing punctuation marks, special characters, and stop words, a keyword dictionary is extracted. Each keyword is represented using one-hot encoding. The sum of the keyword one-hot vectors is used to represent the bag-of-words vector of the field value. The indices of the non-zero elements in the bag-of-words vector of the field value are extracted to construct the feature vector of the field value.

[0019] The parameters of minhash are calculated using the following formula:

[0020] Where K represents the length of a locality-sensitive hash segment, p J p represents the lowest Jaccard similarity between two field values ​​in the bag-of-words system. T This represents the similarity between two field values ​​of p. J At this time, there is at least one locally sensitive hash segment with expected probability. Given K, p T p J Then, L can be calculated, representing the total number of locally sensitive hash segments;

[0021] Select a random seed s, and use the Mersenne rotation to generate two pseudo-random sequences α and β of length L*K, where the range of α is [1, p) and the range of β is [0, p), and p is a large Mersenne prime; convert α and β into column vectors;

[0022] For an eigenvector f, horizontally expand β into a matrix B with |f| columns, where |f| represents the length of the eigenvector; then hash the eigenvector L*K times using the following global hash function:

[0023]

[0024] Where M hash (f) is the global hash matrix of the eigenvector f, for M hash The minimum value of each row of (f) is used to form the minhash vector of the feature vector f.

[0025] Furthermore, in step S21, for the formatted fields, a partitioned table is constructed using the category value of the field value as the partitioning key and the list of object indexes as the value. Specifically, for each formatted field of the join table, the field values ​​are traversed and the unique category values ​​are extracted. A hash table, called the partitioned table, is constructed using the category value as the key and the list of object indexes corresponding to the category value as the value. The key in the table is called the partitioning key.

[0026] Furthermore, S3, based on the minhash vector, constructs a minhash matrix for the field and vertically divides it into multiple locality-sensitive hash matrices. Using the index of the locality-sensitive hash matrix and the minhash segment as the block key, and the list of object indices as the value, a block table is constructed, specifically including:

[0027] For an unstructured field, the S22 method is used to calculate the minhash vector of each value and construct the minhash matrix of the field by row; the minhash matrix is ​​vertically divided into L N*K local sensitive hash matrices, with each row being a minhash segment, where N represents the total number of rows in the join table;

[0028] Traverse the minhash segment of a locality-sensitive hash matrix, extract the unique category values, use the matrix index and category value as the block key, and the list of corresponding object indices as the values ​​to construct a block table.

[0029] Furthermore, S4, treating the set of block keys in the block table as the information source, calculates its information entropy, which is used as the weight of the block key, specifically including:

[0030] For each block table, its information entropy is calculated using the following formula:

[0031]

[0032] Where |BT| represents the total number of data objects corresponding to all block keys in the block table, |BT(k i | represents the block key k i The total number of corresponding data objects, where m is the total number of block keys in the block table; information entropy is used as the weight of the block keys within the table.

[0033] Further, in S5, traverse all the partitioned tables, using the objects in the list corresponding to the partition key as nodes, and the sum of the weights of the co-occurring partition keys between objects as the weights of the edges, to construct an undirected graph, specifically including:

[0034] Treat each data object in the union table as a node, construct edges between each pair of nodes, and the weight of the edge is the sum of the weights of the block key that the two data objects share in all the block tables. Remove edges with zero weight and nodes with zero degree to obtain an undirected graph.

[0035] Furthermore, S6, using the sum of the information entropy of all block tables as a benchmark, calculates the pruning threshold and prunes the edges of the undirected graph. The node pairs corresponding to the remaining edges in the graph are candidate pairs of similar data objects, specifically including:

[0036] The maximum similarity S is obtained by summing the information entropy of the block tables generated in S4. max Choose a coefficient c and calculate the pruning threshold S. p =c×S max ;

[0037] In the undirected graph generated by S5, all weights less than S p After removing all edges, the remaining edge pairs corresponding to the nodes are the candidate pairs of similar data objects.

[0038] Technical effect

[0039] This invention provides a method for rapidly generating candidate pairs of similar data objects, enabling the rapid generation of candidate pairs from multiple structured data sources. It does not rely on labeled data for training, classifies fields based on value characteristics, uses locality-sensitive hashing to extract similarity features of data objects, and considers the differences in information content between different similarity features to make the similarity measurement more accurate. Furthermore, it uses a pruning method for undirected graphs to produce candidate pairs of similar data objects with a more streamlined scale and higher quality, thus improving the efficiency of subsequent matching tasks.

[0040] The following will further explain the concept, specific structure, and technical effects of the present invention in conjunction with the accompanying drawings, so as to fully understand the purpose, features, and effects of the present invention. Attached Figure Description

[0041] Figure 1 This is a flowchart illustrating a preferred embodiment of a method for rapidly generating candidate pairs of similar data objects according to the present invention.

[0042] Figure 2 This is the output undirected graph of a method for rapidly generating candidate pairs of similar data objects according to a preferred embodiment of the present invention. Detailed Implementation

[0043] To make the technical problems to be solved, the technical solutions, and the beneficial effects of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present invention and are not intended to limit the present invention.

[0044] In the following description, specific details, such as particular internal procedures and techniques, are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of the invention. However, those skilled in the art will appreciate that the invention may be practiced in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of the invention with unnecessary detail.

[0045] like Figure 1 As shown, this invention provides a method for rapidly generating candidate pairs of similar data objects, specifically including the following steps:

[0046] Step S1: Perform schema alignment and inter-table union on multiple structured data tables that are to be generated as candidate pairs of similar data objects. Based on the characteristics of the field values, the fields of the union table are divided into two categories: formatted fields and unformatted fields.

[0047] In this embodiment, two heterogeneous data tables, shown in Table 1 and Table 2, are used as input.

[0048] 1 "w1 w2 w3" v1 v3 2 "w4 w5 w6" v2 v4

[0049] Table 1 Structured Data

[0050] 1 "w1 w2 w3" v1 2 "w7 w8 w9" v2

[0051] Table 2 Structured Data

[0052] In this table, T1.a1 and T2.a1, T1.a2 and T2.a2 are common fields with the same meaning in both tables. T1.id and T2.id are the primary keys of both tables. These fields are retained, while T1.a3 is discarded. A vertical join is then performed on the two tables, resulting in the joined table shown in Table 3.

[0053] 1 1 "w1 w2 w3" v1 2 2 "w4 w5 w6" v2 3 1 "w1 w2 w3" v1 4 2 "w7 w8 w9" v2

[0054] Table 3 Joint Table

[0055] Specifically, T1.a1 and T2.a1, T1.a2 and T2.a2 are merged into Tu.a1 and Tu.a2 fields, respectively. The primary keys of the original tables are merged into Tu.originid field, where Tu.id represents the primary key of the join table and the index of the data objects in the table. The total number of data objects N in the join table is 4. Based on field characteristics, Tu.a1 is classified as an unformatted field, and Tu.a2 is classified as a formatted field.

[0056] Step S21: For the formatted field, construct a block table with the category value of the field value as the block key and the list of object indexes as the value;

[0057] In this embodiment, the category values ​​of the formatted field Tu.a2 are v1 and v2, and the block table is constructed as follows:

[0058] v1 [1,3] v2 [2,4]

[0059] Table 4 Block Table 1

[0060] v1 and v2 are used as keys, and the value of the Tu.a2 field is a list of indexes of the data objects v1 and v2.

[0061] Step S22: For unformatted fields, the field values ​​are treated as short texts to construct a corpus. One-hot encoding is used to extract the bag-of-words vector of the field value. The non-zero subscripts of the bag-of-words vector are used to construct the feature vector of the field value. The minhash vector of all feature vectors is calculated.

[0062] In this embodiment, the value of field Tu.a1 is treated as short text. After unifying the capitalization, removing punctuation and stop words, the corpus is constructed as follows:

[0063] ["w1 w2 w3","w4 w5 w6","w7 w8 w9"]

[0064] Keywords are extracted from the corpus and represented using one-hot vectors. A bag-of-words vector is then derived for each field value. The indices of the non-zero elements in the bag-of-words vector are used as the feature vectors for the field values, as follows:

[0065]

[0066]

[0067] Table 5 Feature vectors of the Tu.a1 field

[0068] The parameters of minhash are calculated using the following formula:

[0069] Where K represents the length of a locality-sensitive hash segment, p J p represents the lowest Jaccard similarity between two field values ​​in the bag-of-words system. T This represents the similarity between two field values ​​of p. J At this time, there is at least one locally sensitive hash segment with expected probability. Given K, p T p J Then, L can be calculated, representing the total number of locally sensitive hash segments;

[0070] In this embodiment, K = 1, p T =0.95, p J =0.5, then L=5;

[0071] Select a random seed s, and use the Mersenne rotation to generate two pseudo-random sequences α and β of length L*K, where the range of α is [1, p) and the range of β is [0, p), and p is a large Mersenne prime; convert α and β into column vectors;

[0072] In this embodiment, p = 7 (note that in practical applications, p should be a large Mersenne prime, such as 2). 61 -1) The two generated random sequences are:

[0073] α=(1,2,3,5,6), β=(0,1,3,4,6);

[0074] For an eigenvector f, horizontally expand β into a matrix B with |f| columns, where |f| represents the length of the eigenvector; then hash the eigenvector L*K times using the following global hash function:

[0075]

[0076] Where M hash (f) is the global hash matrix of the eigenvector f, for M hash The minimum value of each row of (f) is used to form the minhash vector of the feature vector f.

[0077] In this embodiment, m = 2 32 -1, the minhash vector corresponding to the feature vector in Table 5 is calculated as follows:

[0078] 1 (1,0,2,0,3) 2 (4,2,0,1,0) 3 (1,0,2,0,3) 4 (0,1,2,0,4)

[0079] Table 6. minhash value of the Tu.a1 field

[0080] Step S3: For an unstructured field, calculate the minhash vector of each value using the method described in S22 and construct the minhash matrix of the field by row; divide the minhash matrix vertically into L N*K local sensitive hash matrices, with each row being a minhash segment, where N represents the total number of rows in the join table;

[0081] Traverse the minhash segment of a locality-sensitive hash matrix, extract the unique category values, use the matrix index and category value as the block key, and the list of corresponding object indices as the values ​​to construct a block table.

[0082] In this embodiment, the five block tables constructed using the minhash matrix are as follows:

[0083] 1-0 [4] 1-1 [1,3] 1-4 [2]

[0084] Table 7 Block Table 2

[0085] 2-0 [1,3] 2-1 [4] 2-2 [2]

[0086] Table 8 Block Table 3

[0087] 3-0 [2] 3-2 [1,3,4]

[0088] Table 9 Block Table 4

[0089] 4-0 [1,3,4] 4-1 [2]

[0090] Table 10 Block Table 5

[0091] 5-0 [2] 5-3 [1,3] 5-4 [4]

[0092] Table 11 Block Table 6

[0093] Step S4: For each block table obtained in S21 and S3, calculate its information entropy using the following formula:

[0094]

[0095] Where |BT| represents the total number of data objects corresponding to all block keys in the block table, |BT(k i | represents the block key k i The total number of corresponding data objects, where m is the total number of block keys in the block table; information entropy is used as the weight of the block keys within the table.

[0096] In this embodiment, the information entropy corresponding to Table 4 is 0.5, and the information entropies corresponding to Tables 7 to 11 are 0.75, 0.75, 0.405, 0.405, and 0.75, respectively.

[0097] Step S5: Treat each data object in the union table as a node, construct edges between each pair of nodes, and the weight of the edge is the sum of the weights of the block key that the two data objects share in all the block tables. Remove edges with zero weight and nodes with zero degree to obtain an undirected graph.

[0098] In this embodiment, the undirected graph constructed according to Tables 4 and 7 is as follows: Figure 2 As shown;

[0099] Step S6: Sum the entropy of the block table information generated in S4 to obtain the maximum similarity S. max Choose a coefficient c and calculate the pruning threshold S. p =c×S max ;

[0100] All weights in the undirected graph generated in S5 that are less than S p After removing all edges, the remaining edge pairs corresponding to the nodes are the candidate pairs of similar data objects.

[0101] In this embodiment, S max =3.56, c takes 0.5, then S p =1.78. Discard all edges in the undirected graph with values ​​less than 1.78. The node pairs corresponding to the remaining edges are <1,3>. These two data objects constitute a similarity candidate pair, and their indexes in their respective original tables are both 1.

[0102] This invention provides a method for rapidly generating candidate pairs of similar data objects based on locality-sensitive hashing and undirected graphs. This method enables the rapid generation of candidate pairs of similar data objects from multiple structured data sources, without relying on labeled data for training. It categorizes fields based on the characteristics of the values, uses locality-sensitive hashing to extract similarity features of data objects, and considers the differences in information content between different similarity features to make the similarity measurement more accurate. Furthermore, it uses a pruning method for the undirected graph to produce candidate pairs of similar data objects with a more streamlined scale and higher quality, which helps improve the efficiency of subsequent matching tasks.

[0103] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A method for rapidly generating candidate pairs of similar data objects, characterized in that, Includes the following steps: S1. Perform schema alignment and inter-table union on multiple structured data tables that are candidate pairs of similar data objects to be generated. Based on the characteristics of the field values, the fields of the union table are divided into two categories: formatted fields and unformatted fields. S2. Process the formatted and unformatted fields separately to obtain the minhash vectors of all feature vectors under the formatted and unformatted fields, respectively; specifically including: S21. For formatted fields, construct a partitioned table using the category value of the field value as the partitioning key and the list of object indexes as the value; S22. For unformatted fields, the field values ​​are treated as short texts to construct a corpus. One-hot encoding is used to extract the bag-of-words vector of the field values. The non-zero indices of the bag-of-words vectors are used to construct the feature vector of the field value. The minhash vector of all feature vectors is calculated. The content includes: For each unformatted field, the field values ​​are treated as short texts. The set of field values ​​constitutes a corpus. After removing punctuation marks, special characters, and stop words, a keyword dictionary is extracted. Each keyword is represented by one-hot encoding. The sum of the keyword one-hot vectors is used to represent the bag-of-words vector of the field value. The indices of the non-zero elements in the bag-of-words vector of the field value are extracted to construct the feature vector of the field value. The parameters of minhash are calculated using the following formula: , in, K This represents the length of a locally sensitive hash segment. The Jaccard similarity represents the lowest bag-of-words similarity between two field values. This represents the similarity between two field values. At that time, there is at least one expected probability of a locally sensitive hash segment; given K , , Then, it can be calculated L , representing the total number of locally sensitive hash segments; Select a random seed s, and use the Mason tween rotation to generate two pairs of lengths s and s. L*K Pseudo-random sequences , ,in The range of values ​​is , The range of values ​​is , For a large Mersenne prime; , Convert to a column vector; For a feature vector ,Will The number of columns to be horizontally expanded is matrix ,in This represents the length of the feature vector; the feature vector is hashed using the following global hash function. L*K Secondary hash: ， in For feature vectors The global hash matrix, for The minimum value of each row is taken to form the feature vector. minhash vector S3. Based on the minhash vector, construct the minhash matrix of the field and divide it vertically into multiple local sensitive hash matrices. Use the index of the local sensitive hash matrix and the minhash segment as the block key and the list of object indexes as the value to construct a block table. S4. Treat the set of block keys in the block table as the information source, calculate its information entropy, and use it as the weight of the block key; specifically including: For each block table, its information entropy is calculated using the following formula: ， in, This represents the total number of data objects corresponding to all partition keys in the partitioned table. Represents block key The total number of corresponding data objects The total number of block keys in the partitioned table; information entropy is used as the weight of the block keys within the table; S5. Traverse all block tables, using the objects in the list corresponding to the block key as nodes, and the sum of the weights of the co-occurring block keys between objects as the weights of the edges, to construct an undirected graph. S6. Calculate the pruning threshold based on the sum of the information entropy of all block tables, prune the edges of the undirected graph, and the node pairs corresponding to the remaining edges in the graph are candidate pairs of similar data objects.

2. The method for rapidly generating candidate pairs of similar data objects as described in claim 1, characterized in that, This involves schema alignment and inter-table joining of multiple structured data tables containing candidate pairs of similar data objects. Based on the characteristics of the field values, the fields in the joined table are categorized into formatted and unformatted fields, specifically including: The pattern alignment is set to filter out common fields with the same meaning across multiple structured data tables based on the field information of multiple structured data tables. For each structured data table, only the common fields and the primary key of that table are retained. Inter-table joins are configured to combine multiple data tables vertically, merging the primary key columns of the original tables into a new column; a primary key is added to the joined table, representing the index of the data objects in the table; The fields in the table are divided into two categories: fields whose values ​​follow a uniform format and have few category values ​​are called formatted fields, and fields whose values ​​have no fixed format and have a high degree of variation are called unformatted fields.

3. The method for rapidly generating candidate pairs of similar data objects as described in claim 1, characterized in that, S21. For formatted fields, construct a partitioned table using the category value of the field value as the partitioning key and the list of object indexes as the value. Specifically, for each formatted field in the join table, iterate through the field values ​​and extract the unique category values. Construct a hash table, called the partitioned table, using the category value as the key and the list of object indexes corresponding to the category value as the value. The key in the table is called the partitioning key.

4. The method for rapidly generating candidate pairs of similar data objects as described in claim 3, characterized in that, S3 involves constructing a minhash matrix based on the minhash vector as the field, dividing it vertically into multiple local sensitive hash matrices, and using the index of the local sensitive hash matrix and the minhash segment as the block key, and the list of object indexes as the value, to build a block table, specifically including: For an unstructured field, calculate the minhash vector for each value using the method described in S22 and construct the minhash matrix for that field row by row; divide the minhash matrix vertically into equal parts. L indivual N * K Locality-Sensitive Hash (LSH) matrix, where each row is a minhash segment, where N Represents the total number of rows in the join table; Traverse the minhash segment of a locality-sensitive hash matrix, extract the unique category values, use the matrix index and category value as the block key, and the list of corresponding object indices as the values ​​to construct a block table.

5. The method for rapidly generating candidate pairs of similar data objects as described in claim 1, characterized in that, S5. Traverse all partitioned tables, using the objects in the list corresponding to the partition key as nodes, and the sum of the weights of the co-occurring partition keys between objects as the edge weights, constructing an undirected graph, specifically including: Treat each data object in the union table as a node, construct edges between each pair of nodes, and the weight of the edge is the sum of the weights of the block key that the two data objects share in all the block tables. Remove edges with zero weight and nodes with zero degree to obtain an undirected graph.

6. The method for rapidly generating candidate pairs of similar data objects as described in claim 1, characterized in that, S6. Calculate the pruning threshold based on the sum of the information entropy of all block tables, prune the edges of the undirected graph, and the node pairs corresponding to the remaining edges in the graph are candidate pairs of similar data objects, specifically including: The maximum similarity is obtained by summing the entropy of the block table information generated in S4. Choose a coefficient c and calculate the pruning threshold. ; All weights in the undirected graph generated in S5 are less than After removing all edges, the remaining edge pairs corresponding to the nodes are the candidate pairs of similar data objects.

Images