A novel efficient filtering method for accelerating similar text search

Abstract

The application discloses a novel efficient filtering method for accelerating similar text search, comprising the following steps: for the user-submitted text data set, distinguishing high and low frequency elements, and performing hashing on the low frequency elements to establish a to-be-searched data set; dividing the signatures in the data set into different partitions according to length, using a clustering method for the signatures in each partition, forming different clusters for the similar signatures in the new to-be-searched data set, and ensuring that the similarity between the text in the cluster and the cluster head text is greater than a set similarity threshold; constructing a complete binary tree for the text in each cluster in each partition; and deploying the tree filter to the system to accelerate the similar text search through fast filtering. On the basis of using K minimum value technology to estimate the intersection potential, the filter is constructed into a tree according to the distribution of the data set, repeated traversal is avoided, the retrieval efficiency is improved, and the problem of reduced filtering accuracy is avoided.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of database, in particular to a new efficient filtering method for accelerating similar text search. BACKGROUND

[0002] With the advent of the open data era, more and more databases are being published, and these databases become very large as they are constantly updated. These large databases are widely used in many applications (such as data mining, data cleaning, document search), so people have become increasingly interested in how to quickly search similar text in large databases.

[0003] Similar text search can be converted into a threshold intersection potential query problem, and the GB-KMV sketch technique can be used to efficiently complete the threshold intersection potential search problem. The GB-KMV sketch method fully considers the distribution relationship characteristics of the data (i.e. the size distribution of the set and the frequency distribution of the element), and while ensuring fast response in executing the threshold intersection potential query, it also greatly improves the accuracy of the threshold intersection potential query compared to the KMV sketch technique.

[0004] Since the GB-KMV sketch needs to compare the high-frequency part and the low-frequency part of each text with the query text, it is relatively time-consuming. A filter needs to be designed to filter out part of the elements for accelerated processing. SUMMARY

[0005] The purpose of the present application is to provide a new efficient filtering method for accelerating similar text search, which solves the problem of low performance of similar text search.

[0006] Technical scheme: In order to achieve the above-mentioned purpose of the present application, the present application provides a new efficient filtering method for accelerating similar text search query, comprising the following steps:

[0007] (1) High and low frequency element block uniform hashing: For the text data set to be queried, the frequency of all elements in the text data set is counted, the first B elements are selected as the high frequency part of the text, and the remaining elements are selected as the low frequency part. The high frequency part is represented by a bitmap, and the low frequency part is truncated to a K minimum value, which is a KMV signature. The text represented in step (2) is transmitted to step (2);

[0008] (2) Partition Similarity Signature Clustering Phase: using the low-frequency part of the KMV signature, according to the KMV signature length, it is divided into different partitions, and each partition contains texts with equal low-frequency part signature length; in each partition, a clustering algorithm is used, a cluster center is randomly selected, the similarity between other texts and the cluster center is calculated, the high-frequency part and the low-frequency part of the KMV signature of the text with high similarity are combined into a cluster, if the size of the cluster meets the requirements, the cluster is retained, otherwise the cluster center is reselected, and the clustered cluster is transmitted to step (3);

[0009] (3) KMV Forest Construction Phase: for each cluster in each partition, a complete binary tree is constructed, the leaf nodes of the binary tree are composed of a plurality of texts in the cluster, and the high-frequency part and the low-frequency part of the text are recorded; each parent node contains all the text information of the leaf node, the high-frequency part bitmap is the union of the two leaf nodes, and the construction is sequentially performed upwards, and finally the root node contains the information of all the texts in the cluster;

[0010] (4) Threshold Intersection Potential Query: according to each text to be queried and the corresponding similarity threshold, in each tree in the KMV forest, the root node is recursively queried downwards, if the intersection potential of the current node does not meet the similarity threshold, subsequent queries are not performed; if the recursion reaches the leaf node and still meets the intersection potential requirement, the leaf node that meets the intersection potential requirement is transmitted to step (5);

[0011] (5) Verification Phase: for each text contained in each leaf node, an intersection potential calculation operation is performed to determine whether the text is a similar text that meets the similarity requirement.

[0012] Further, in step (2), the texts are divided into different partitions according to the low-frequency KMV signature length, so that the number of low-frequency elements of the low-frequency KMV signature of all texts in the same partition is equal; and in each partition, the clustering method is used to group the signatures into different clusters according to the similarity between the signatures with the same length; the similarity of the signatures refers to the similarity of the high-frequency part bitmap and the low-frequency part, that is, the estimated intersection potential / the length of the text, and the calculation method of the estimated intersection potential is the sum of the high-frequency part bitmap intersection potential and the low-frequency part intersection potential multiplied by the correction coefficient.

[0013] Further, in step (2), different similarity thresholds can be set for the clustering method, and clusters with higher similarity and clusters with slightly lower similarity are clustered respectively, so that more elements are added to the cluster to the greatest extent, while ensuring that the similarity between clusters is as high as possible.

[0014] Further, in step (3), the different clusters are constructed into a complete binary tree, each node of the binary tree contains: a pointer array storing the high frequency part bitmap of the corresponding multiple texts and the pointer of the KMV signature; an array recording all different elements contained in the low frequency KMV signature of the text saved by itself; a bitmap used for saving the high frequency part of all texts; the construction of the complete binary tree includes the following steps: first, for each cluster in each partition, starting from the leaf node, select several texts in the cluster to form a leaf node, the method of forming the leaf node is that the high frequency part bitmap is the or of the high frequency part bitmap of all texts, and the element array is all different elements contained in the KMV signature of all texts. Finally, these leaf nodes are merged upwards in turn to form a complete binary tree.

[0015] Further, in step (3), the method of merging two binary tree nodes upwards to form a parent node is that the pointer array of the parent node is the union set of the two child nodes, the element array is the sorted union set without repeated elements of the element arrays of the two child nodes, and the high frequency part bitmap is still the or (union set) of the two child node bitmaps.

[0016] Further, in step (3), the height of the tree and the granularity of filtering can be adjusted by adjusting the maximum number of stored texts of the binary tree and the number of texts contained in each leaf node, which produces the best effect for different data sets and also controls the memory overhead.

[0017] Further, in step (3), in order to reduce Cache miss in the query process, the memory of KMV in all leaf nodes in the KMV forest should be allocated as closely as possible to obtain the highest query efficiency in the verification stage.

[0018] Further, in step (4), when the KMV forest is used for filtering operation, the specific steps are as follows: 1. Start comparing the high frequency part bitmap and the element array recursively from the root node, and calculate the estimated intersection potential of the two with the query text. 2. If the estimated intersection potential is less than the similarity threshold, filter out all the records contained in the node. 3. Otherwise, recursively perform steps 1 and 2 on the two child nodes. The filtering process can be accelerated using prefix filtering. 4. If the current node is a leaf node and cannot be filtered, perform individual verification, and the verification process is to compare the estimated intersection potential of the text to be verified with the query text with the similarity threshold.

[0019] Beneficial effects: the application uses clustering method in similar text search problem, and combines with KMV forest construction, and proposes an effective and hierarchical method using these clusters to accelerate threshold intersection potential query operation: first, the data distribution in the data set and the similarity between texts are fully considered, the filter is designed as a forest constructed according to the similarity, so that multiple similar elements can be filtered out at the tree node at one time, and the query delay of similar text search is reduced. Second, considering the cache miss problem, the memory occupied by KMV in the leaf node of the KMV forest is closely allocated, the verification efficiency is improved, and the query delay of similar text search is further reduced. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 The overall flowchart of the application is shown in the figure;

[0021] Figure 2 The partitioned similar signature clustering stage described in the application is shown in the figure;

[0022] Figure 3 The KMV forest structure described in the application is shown in the figure;

[0023] Figure 4 The threshold intersection potential query stage described in the application is shown in the figure. DETAILED DESCRIPTION

[0024] The application will be further illustrated below in combination with the drawings and specific embodiments, and it should be understood that these embodiments are only used to illustrate the application and not to limit the scope of the application, and after reading the application, those skilled in the art can make various equivalent modifications of the application, which all fall within the scope defined by the appended claims of the application.

[0025] The application provides a new efficient filtering method for accelerating similar text query, which solves the problem of low performance of similar text search. As shown in the figure, Figure 1 The complete flow of the application includes high and low frequency element block uniform hashing stage, partitioned similar signature clustering stage, KMV forest construction stage, threshold intersection potential query stage and verification stage. The specific embodiments are described as follows:

[0026] The high and low frequency element block uniform hashing stage corresponds to technical solution step (1). The specific embodiment is as follows: first, the frequency of all elements in the data set is counted, and the top B high frequency elements are selected as the high frequency part according to the frequency from high to low, and the remaining elements constitute the low frequency part. For the B high frequency elements, select the high frequency elements in each text to form a high frequency buffer (bitmap). The high frequency part bitmap of each text is stored in the memory, and the low frequency part bitmap of each text is stored in the disk. Figure 1Each bit indicates whether the high frequency element exists in the text. Secondly, we use a uniform hash function to hash the low frequency elements in each text into a fixed interval [0, 1]. Finally, we sort the low frequency elements in each text according to the size of the hash value to form a low frequency area.

[0027] The partition similarity signature clustering stage corresponds to technical solution step (2). The specific implementation is: first, divide the text into different partitions according to the length of the low frequency part of the signature, and each partition contains signatures with equal length. Randomly select a signature in each partition as the cluster head, and calculate the similarity between the other texts and the cluster head, for example, the specific calculation method of the similarity sim(A, B) between text A and text B is

[0028]

[0029] Where H A ,H B are the high frequency part bitmaps of texts A and B respectively, is the estimated intersection potential of the low frequency parts of A and B, The calculation method is as follows,

[0030]

[0031] Where L A ,L B are the low frequency parts of texts A and B, k is the length of the low frequency part in the partition, and f(k) is the hash value of the kth element.

[0032] Then find all the texts with a similarity greater than or equal to the set similarity threshold. If the total number is greater than or equal to the set minimum cluster size, we form a cluster with this part of the signature. Otherwise, reselect the cluster head and repeat the above operation. After this stage, we get multiple partitions, each containing several clusters, and each cluster directly guarantees that the similarity between the cluster center and other texts is greater than the set similarity threshold. In other words, the elements in each cluster are relatively similar to each other.

[0033] Figure 2 An example is used to show the general situation of the final obtained partitions, including a cluster head and multiple similar texts.

[0034] The KMV forest construction stage corresponds to technical solution step (3). The specific implementation is: first, determine the size of the leaf node, that is, how many texts it contains, which is determined by the input parameter. Then, for each cluster obtained in step (2), we combine it into multiple leaf nodes, and then use these leaf nodes to construct a complete binary tree.

[0035] Where the leaf nodes of the binary tree are composed of different texts in the cluster, recording all the signature elements and high frequency parts information. The content of the node is shown as Figure 3

[0036] 1) An array of pointers, which stores the corresponding high frequency part bitmap of the corresponding multiple texts, and the pointer of KMV signature

[0037] 2) An array, which records all the different elements contained in the low frequency KMV signature of the multiple texts saved by itself

[0038] 3) A bitmap, which is used to save the elements in the high frequency part of the multiple texts.

[0039] In order to reduce the Cache miss problem in the verification of the last leaf node, we try to allocate the memory of the leaf node of the KMV forest as closely as possible. After constructing each leaf node, we construct a complete binary tree upwards. The parent node contains all the text information of the child nodes, and the high frequency part bitmap is the union of the two child nodes. We build upwards in turn, and finally the root node will contain the high frequency and low frequency part information of all the texts in the cluster.

[0040] Finally, we get several partitions, each of which contains several complete binary trees composed of texts.

[0041] The threshold intersection potential query stage corresponds to technical scheme step (4). The specific implementation is: for each text to be queried and the corresponding similarity threshold, we also divide the text to be queried into high frequency part bitmap and low frequency KMV signature according to step (1). For each binary tree in the KMV forest, we start from the root node and recursively query the high frequency intersection and low frequency intersection in turn to estimate the total intersection potential of the text, and the calculation method is as follows:

[0042]

[0043] Where A is the text to be queried, TreeNode represents the node of the tree, L TreeNode and H TreeNode represent the low frequency part and high frequency part bitmap saved in the node respectively, L A and H A represent the low frequency part and high frequency part bitmap of the text to be queried respectively.

[0044] If the similarity threshold requirement is not met, all the texts contained in the whole node will be filtered, and if it is met, the child nodes will be queried.

[0045] The specific steps of filtering are as follows, Figure 4 The flowchart of the filtering operation process is shown in the following figure: ​

[0046] 1. Start from the root node, recursively compare the high frequency part bitmap and the element array with the query text to estimate the intersection size.

[0047] 2. If the estimated intersection size is smaller than the similarity threshold, filter out all the text contained in the node.

[0048] 3. Otherwise, recursively apply step 1 and 2 to the two child nodes. The filtering process can be accelerated by using prefix filtering.

[0049] 4. If the current node is a leaf node, go to step 5 for verification.

[0050] The verification phase corresponds to technical solution step (5). The specific implementation is: taking the intersection size of the high frequency part and the low frequency part corresponding to the text that cannot be filtered out in step (4) with the high frequency and low frequency parts corresponding to the query text respectively, estimating the intersection size between the texts to finally confirm whether the text is a similar text.

Claims

1. A novel and efficient filtering method for accelerating similar text search characterized in that The similarity distribution characteristics of data are utilized, including the following steps: (1) high and low frequency element block uniform hashing: for the text data set to be queried, the frequency of all elements in the text data set is counted, the first B elements are selected as the high frequency part of the text, and the remaining elements are selected as the low frequency part of the text, the high frequency part is represented by a bitmap, a part of the low frequency part is selected as a K minimum value, the K minimum value is a KMV signature, and the re-represented text is transmitted to step (2); (2) partition similarity signature clustering stage: the KMV signature of the low frequency part is utilized, different partitions are divided according to the length of the KMV signature, and each partition contains texts with equal low frequency part signature lengths; in each partition, a cluster center is randomly selected, the similarity between other texts and the cluster center is calculated, the high frequency part and the KMV signature of the low frequency part of the text with high similarity are combined to form a cluster, if the size of the cluster meets the requirement, the cluster is retained, otherwise the cluster center is reselected, and the clustered cluster is transmitted to step (3); (3) KMV forest construction stage: for each cluster in each partition, a complete binary tree is constructed, the leaf nodes of the binary tree are composed of a plurality of texts in the cluster, and the high frequency part and the low frequency part of the text are recorded; each parent node contains all the text information of the leaf nodes, the high frequency part bitmap is the union of the two leaf nodes, and the construction is sequentially performed upwards, and finally the root node contains the information of all the texts in the cluster; (4) threshold intersection potential query: according to each text to be queried and the corresponding similarity threshold, each tree in the KMV forest is queried recursively from the root node downwards, if the intersection potential of the current node does not meet the similarity threshold, subsequent queries are not performed; if the recursion reaches the leaf node and still meets the intersection potential requirement, the leaf node that meets the intersection potential requirement is transmitted to step (5); (5) verification stage: the intersection potential calculation operation is performed on the text contained in each leaf node to determine whether the text is a similar text that meets the similarity requirement.

2. A novel and efficient filtering method for accelerating search of similar texts as claimed in claim 1, wherein: In step (2), the texts are divided into different partitions according to the length of the KMV signature of the low frequency part, and the texts in each partition are grouped into different clusters according to the similarity between the texts by using a clustering method; if the number of texts contained in a cluster is greater than or equal to a set cluster size threshold, the cluster is retained, otherwise the cluster center is reselected, and the clustering operation is repeated.

3. The novel and efficient filtering method for accelerating search of similar text according to claim 1, wherein: In step (3), different clusters are constructed into a complete binary tree, each node of the complete binary tree includes: a pointer array that stores the high frequency part bitmap corresponding to a plurality of texts and the pointer of the low frequency KMV signature; an array that records all different elements contained in the low frequency KMV signature corresponding to a plurality of texts saved by itself; and a bitmap for saving the high frequency part of all texts.

4. The novel and efficient filtering method for accelerating search of similar text according to claim 1, wherein: In the step (3), the different clusters are constructed into a complete binary tree, including the following steps: firstly, for each cluster in each partition, a number of texts are selected to form a leaf node from the leaf node, the method of forming a leaf node is that the bitmap of the high frequency part is the or of the bitmaps of all texts, and the element array is all different elements contained in the KMV signatures of all texts; finally, a complete binary tree is constructed from the leaf nodes to the top; the method of merging two leaf nodes to form a parent node is that the pointer array of the parent node is the union of the two leaf nodes, the element array is the sorted union of the element arrays of the two leaf nodes without duplicate elements, and the high frequency part bitmap is the or of the bitmaps of the two leaf nodes.

5. The novel and efficient filtering method for accelerating search of similar text as claimed in claim 1, wherein: In the step (4), when the KMV forest is used for filtering operation, the high frequency part bitmap and the element array are compared with the query text from the root node, if the estimated intersection potential is less than the similarity threshold, all texts contained in the node are filtered out, otherwise the same filtering operation is performed on the leaf node; the prefix filtering is used to accelerate the query process.

Images