A signature-based set semantic similarity connection method

Abstract

The application relates to a signature-based set semantic similarity connection method, and belongs to the fields of databases and information retrieval. The method comprises four parts: firstly, a classification tree construction step: a classification tree is constructed according to a WordNet knowledge base given a data set; secondly, a set signature step: each set in the data set is signed to obtain a corresponding signature data set; thirdly, a data preprocessing step: the sets in the signature data set are sorted to obtain a sorted data set; and finally, a connection processing step: self-connection is performed on the sets in the sorted data set to obtain a set of semantic similarity results. The method is based on signature prefix filtering technology and length filtering technology, and finally realizes the set semantic similarity connection method, so that the set semantic connection efficiency can be effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to a signature-based set semantic similarity join method, belonging to the field of database and information retrieval. Related knowledge includes computer technology, database technology, document and document processing technology, information retrieval technology, etc. BACKGROUND

[0002] Set similarity join aims to find all pairs of sets from two sets whose similarity is greater than or equal to a certain threshold θ. At present, many research works adopt the filter and verify framework, which filters out the sets that are not likely to be similar in the filtering stage, thereby generating a candidate set. In the verification stage, the candidate set is verified to obtain the final result. Fast generation of as small a candidate set as possible is the key to an efficient set similarity join algorithm, and for this purpose, many filtering techniques such as length filtering, prefix filtering, position filtering, etc. have been proposed and applied to candidate set generation.

[0003] There are many researches on set similarity join. Roughly speaking, they can be divided into prefix filtering-based methods, partition-based methods and LSH-based methods. The prefix filtering-based methods mainly include SSJoin, AllPair, PPJoin, MPJoin, TopkJoin, MKJoin, etc. The partition-based methods mainly include PartEnum, BFJoin, PartJoin, etc. The LSH-based methods mainly include MinHash and BayesLSH, etc.

[0004] The current methods have the following problems: first, set similarity join mainly focuses on the matching degree of elements between sets, while set semantic similarity join needs to understand the semantic relationship of elements in the set. This involves in-depth understanding of natural language, including synonyms, antonyms, hyponym-hypernym relationships, etc., which is a great technical challenge. Especially the processing of synonyms and polysemous words is difficult to achieve in string-based methods, because they do not take into account the meaning of words and the relationship between words; second, there is a need for appropriate evaluation indicators to evaluate element similarity and set similarity. Therefore, the present application introduces a classification tree to evaluate element semantic similarity and set semantic similarity by constructing a classification tree. SUMMARY

[0005] The purpose of the present application is to provide a signature-based set semantic similarity join method, aiming to solve the technical problem that semantic word processing needs to perform Hungarian algorithm, which corresponds to high processing cost, in set semantic similarity join.

[0006] To achieve the above purpose, the technical scheme of the present application is: a signature-based set semantic similarity join method, the specific steps of which are:

[0007] Step 1: For a given dataset D, construct a classification tree T according to the WordNet knowledge base;

[0008] This step is the classification tree construction phase;

[0009] Step 2: Sign each set S in the dataset D to obtain the corresponding signature set D sig ;

[0010] This step is the set signature phase;

[0011] Step 3: Sort the sets in the signature set D sig in ascending order of set length, and sort the elements in the set in ascending order of signature frequency, to obtain the sorted dataset D O ;

[0012] This step is the data preprocessing phase;

[0013] Step 4: Perform self-join on the sets in the sorted dataset D O to obtain the semantic similarity result pair set R.

[0014] This step is the connection processing phase;

[0015] The Step 1 is specifically:

[0016] Step 1.1: Given the dataset D, extract all the words appearing in the WordNet knowledge base from the dataset D;

[0017] Step 1.2: For each word w appearing in WordNet, identify the path P from word w to the root node in WordNet, and construct a new path P according to the order of all words appearing in the dataset D along the path P from the leaf to the root in WordNet. * ;

[0018] Step 1.3: Insert the nodes in the new path P * into the classification tree in turn to construct a simplified classification tree T.

[0019] The Step 2 is specifically:

[0020] Step 2.1: Given the element semantic similarity threshold and the set semantic similarity threshold θ, calculate

[0021] Step 2.2: For each set S in the dataset D, generate the signature S* of S according to the following steps:

[0022] For each element e in S, if e is not in the classification tree T or its depth in the classification tree T is less than then set the element signature g of e e = e, otherwise set g where is the ancestor node of e in the classification tree T with depth ;

[0023] replace each element e in S with the corresponding element signature g e , and obtain the signature set S* of all collections, which finally constitutes the signature dataset D sig .

[0024] The Step 3 is specifically:

[0025] Step 3.1: count the frequency of all independent signature elements in the signature dataset D sig ;

[0026] Step 3.2: sort all collections in the signature dataset D sig in ascending order of collection length, and sort the signature elements in each collection in ascending order of frequency, to obtain the sorted dataset D O ;

[0027] Step 3.3: assign a number ID to each collection in the sorted dataset D O from 0 to |D O |-1 in turn.

[0028] The Step 4 is specifically:

[0029] Step 4.1: set the result pair set R = NULL, and the signature inverted list I = NULL;

[0030] Step 4.2: j from 1 to |D|-1, and execute Step 4.3 to Step 4.7 in a loop;

[0031] Step 4.3: set the candidate set C = NULL;

[0032] Step 4.4: for each signature element g e in the prefix with length of , execute Step 4.5 to Step 4.6 in a loop;

[0033] Step 4.5: for each ID corresponding to g e in the inverted list I[g e ], execute Step 4.6; Step 4.6: if , add the pair of collection ID to the candidate set C;

[0034] ​Step 4.6: Add to the end of inverted list I[g e ];

[0035] Step 4.7: Verify each pair of set ID in C Call Hungarian algorithm to compute and the maximum matching and further compute the semantic similarity of two sets If then add to the result set R.

[0036] The beneficial effects of the present application are: to provide a signature-based set semantic similarity connection method, in the filtering stage, using the signature inverted index, using signature prefix filtering and length filtering, filtering a large number of impossible similar set pairs, reducing the number of Hungarian algorithm execution in the verification stage, and being able to obtain higher efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 is a flowchart of the present application;

[0038] Figure 2 is an example classification tree of the present application;

[0039] Figure 3 is a comparison of the running time of the present application and other methods;

[0040] Figure 4 is a comparison of the number of candidates of the present application and other methods. DETAILED DESCRIPTION

[0041] To make the purpose, technical scheme and advantages of the present application clearer, the embodiments of the present application will be further described in detail with self-connection by combining with the drawings and examples.

[0042] Example 1: As shown in Figure 1 , a signature-based set semantic similarity connection method, the specific steps are:

[0043] Step 1: Given the example dataset D as shown in Table 1, and given the example classification tree T as shown in Figure 2 , which expresses the IS-A relationship between elements;

[0044] Table 1 Example dataset

[0045]

[0046]

[0047] Step 2: Given a set semantic similarity threshold of θ = 0.8 and an element semantic similarity threshold is 0.6, calculate

[0048]

[0049] Step 3: For the first set S1{B, D, I, M, a, b} in D, for element B, in the classification tree, obtain the depth d of B in T B =1, So B's signature g B For B, for element D, which is in the classification tree, obtain the depth d of D in T D =2, Its signature should be its ancestor node with a depth of 2, so D's signature g D For D, for element I, which is in the classification tree, obtain the depth d of I in T I =3, Its signature should be its ancestor node with a depth of 2, so the signature of I is g I For F, for element M, which is in the classification tree, obtain the depth d of M in T M =4, Its signature should be its ancestor node with a depth of 2, so the signature of M is g M For E, for element a, it is not in the classification tree, so the signature of a is g a For a, for element b, it is not in the classification tree, so the signature of b is g b For b, we get

[0050] Sign all sets in D and get D sig As shown in Table 2;

[0051] Table 2 Signature dataset

[0052]

[0053] Step 4: D sig The frequencies of all independent signatures are B: 3 times, D: 7 times, F: 4 times, E: 7 times, a: 4 times, b: 4 times, c: 2 times, d: 2 times, and the signature frequencies from low to high are c, d, B, F, a, b, D, E;

[0054] Step 5: Get D sig The length of each set in , Sort the element signature set Dsig in ascending order according to the length of the set, and e ), according to the signature g e Sort in ascending order of frequency, for example: Since the signature frequencies are F, D, E from low to high, we have After sorting, we have {(N, F), (K, D), (L, E)} and the sorted dataset D O is shown in Table 3.

[0055] Table 3 Sorted Dataset

[0056]

[0057] Step 6: Compute D O the prefix length of all sets, for Take the first elements of , we get Take the prefix of D O all sets, we get S * .pref, which is shown in Table 4.

[0058] Table 4 Prefix of S *

[0059]

[0060] Step 7: Initialize the semantic similarity result pair set R and the signature inverted list I to be empty. For each set O in D , we do the following. At the beginning of the iteration, the first set to be iterated is Initialize the candidate set C to be empty. We iterate through which has only one element At this time, the inverted list of is empty, we add the ID: 1 of to the inverted list of F; the second set to be iterated is We iterate through which has only one element At this time, the inverted list of is empty, we add the ID: 2 of to the inverted list of B; the third set to be iterated is We iterate through which has only one element At this time, the inverted list of is empty, we add the ID: 3 of to the inverted list of a; the fourth set to be iterated is We iterate through which has only one element We iterate through the inverted list of and get the ID: 3, we add to the candidate set C, and add ​The ID: 4 is added to the inverted list of a; the fifth traversed set is Traversal Traversal Inverted list, get ID: 2, Terminate the current loop and The ID: 5 is added to the inverted list of B. Traversal Inverted list, get ID: 1, Terminate the current loop and ID: 5 is added to the inverted list of F; the sixth traversed set is Traversal at this time The inverted list of is empty, The ID: 6 is added to the inverted list of c. at this time The inverted list of is empty, ID: 6 is added to the inverted list of d; the set of the 7th traversal is Traversal Traversal Inverted list, get ID: 6, Will Add candidate set C, The ID: 7 is added to the inverted list of c. Traversal Inverted list, get ID: 6, because Already in the candidate set, terminate the current loop, ID: 7 is added to the inverted list of d; the signature inverted list is shown in Table 5;

[0061] Table 5 Signature inverted list

[0062]

[0063] Step 8: Verify each set pair in the candidate set C. Enter the Hungarian algorithm to get the maximum matching is 3.75, and its semantic similarity is calculated Will Add R; for Enter the Hungarian algorithm to get the maximum matching is 4, calculate its semantic similarity prune

[0064] Finally, the set semantic similarity threshold θ of dataset D is 0.8, and the element semantic similarity threshold 0.6, the result of the set semantic similar join is

[0065] The advantages of the present application compared with the prior art are:

[0066] The platform environment used for testing is:

[0067] CPU: 11th Gen Intel(R) Core(TM) i5-1135G7@2.40GHz 2.42GHz;

[0068] Memory: 16GB;

[0069] Operating system: Windows 11;

[0070] Programming language: Python 3.8.8;

[0071] IDE: Spyder 4.2.5;

[0072] The dataset OHSUMED is used for testing, which corresponds to a classification tree MeSH. The information of the dataset and the classification tree is shown in Table 6 and Table 7.

[0073] Table 6 Information of the dataset

[0074]

[0075] Table 7 Information of the classification tree

[0076]

[0077] The OHSUMED dataset contains 100,000 sets, each set is obtained by dividing the MeSH index term in the document from the medical information database MEDLINE. The MeSH classification tree matched therewith is used for indexing and cataloging biomedical information, which describes the concept classification in the biomedical field.

[0078] The present application designs an efficient set semantic similar join method SPL-Join. In this method, the signature prefix filtering technology SPF and the length filtering technology LF are used to quickly and efficiently exclude impossible similar set pairs, thereby obtaining high join efficiency. To verify the performance of the present method relative to other methods, it is compared with the method APJoin.

[0079] In the test, the number of candidate sets and the connection time of SPL-Join are compared. The results of the comparison experiment are shown in Figure 3 and Figure 4 As shown in Figure 3 and Figure 4It can be seen that the number of candidates and efficiency of SPL-Join are far superior to APJoin on OHSUMED dataset. When θ = 0.9, the time spent by AP-Join on OHSUMED is 13563.4s, while SPL-Join only needs 19.904s. Analyzing the reason for the performance difference of different methods, when θ = 0.9, the number of candidates of AP-Join on OHSUMED is 353940172, while the number of candidates of SPL-Join is only 208355, and the number of candidates of SSC-Join is only 0.06% of that of AP-Join, so its performance is much higher than that of AP-Join.

[0080] The specific embodiments of the present application are described in detail above with reference to the accompanying drawings, but the present application is not limited to the above-described embodiments, and various changes can be made within the knowledge of those skilled in the art without departing from the spirit of the present application.

Claims

1. A signature-based set semantic similarity connection method, characterized by: Step 1: For a given dataset D, construct a classification tree T based on the WordNet knowledge base; Step 2: Sign each set S in the dataset D to obtain the corresponding signature set D sig ; Step 3: Signature set D sig The set is sorted in ascending order by set length, and the elements in the set are sorted in ascending order by signature frequency to obtain the sorted data set D O ; Step 4: sort the data set D O Perform self-join on the set in to obtain the semantically similar result set R; The Step 1 is specifically as follows: Step 1.1: Given a dataset D, extract all the words that appear in the WordNet knowledge base from the dataset D; Step 1.2: For each word w that appears in WordNet, identify the path P from word w to the root node in WordNet, and construct a new path P based on all the words in path P that appear in dataset D along the order from leaf to root in WordNet. * ; Step 1.3: The new path P * The nodes in are inserted into the classification tree in sequence to build a simplified classification tree T; The Step 2 is specifically as follows: Step 2.1: Given an element semantic similarity threshold and the set semantic similarity threshold θ, calculate Step 2.2: For each set S in the dataset D, generate the signature S* of S by following these steps: For each element e in S, if e is not in the classification tree T or the depth of the classification tree T is less than Then set the element signature of e to g e =e, otherwise set in The depth of e in the classification tree T is Ancestor node of Replace each element e in S with the corresponding element signature g e , get the collective signature set S*, and the signature sets of all the collections finally constitute the signature data set D sig ; The Step 3 is specifically as follows: Step 3.1: Statistical signature dataset D sig The frequencies of all independent signature elements in ; Step 3.2: Signature dataset D sig All sets in are sorted in ascending order by set length, and the signature elements in each set are sorted in ascending order by frequency, and the sorted data set D is obtained. O ; Step 3.3: sort the data set D O The set in ranges from 0 to |D O |-1 is assigned a number ID in sequence.

2. The signature-based set semantic similarity connection method according to claim 1, characterized in that: The Step 4 is specifically as follows: Step 4.1: Set the result pair set R = NULL and the signature inverted list I = NULL; Step 4.2: j increases from 1 to |D|-1, and the loop continues from Step 4.3 to Step 4.

7. Step 4.3: Set candidate set C = NULL; Step 4.4: The length is Each signature element g in the prefix of e , loop through Step 4.5 to Step 4.6; Step 4.5: g e The corresponding inverted list I[g e ] corresponds to the set of each ID in like Then the collection ID Add to candidate set C; Step 4.6: Append to the inverted list I[g e ]’s rear; Step 4.7: Verify each set ID pair in C Call Hungarian algorithm to calculate and Maximum matching of Then calculate the semantic similarity of the two sets like Then Add to the result pair set R.

Images