Set similarity calculation method and system based on minhash

Abstract

The invention discloses a set similarity calculation method and system based on minhash. The method includes the steps that each element in a set is mapped into a first hash value with an m-bit length through a hash function, and 2k class groups are established, wherein each class group corresponds to one tag which is a second hash value with a k-bit length, and different class groups correspond to different tags; for any set, the first hash values corresponding to the elements in the set are distributed into the class groups corresponding to the tags with the same first k bits as the first hash values; minhash values, corresponding to the class groups, of the set are determined according to the distribution result; the minhash values, corresponding to the class groups, of the set form an array serving as a minhash signature of the set; according to the minhash signatures of any two sets, the similarity of the two sets is calculated. By means of the technical scheme, the minhash signature speed can be greatly increased, and thus the set similarity calculation speed is greatly increased.

Description

technical field [0001] The invention relates to the technical field of computer processing, in particular to a minhash-based set similarity calculation method and system. Background technique [0002] Given two sets A and B, Jaccard similarity is an algorithm widely used to describe the similarity between sets, and its formula is expressed as follows: [0003] [0004] To calculate the similarity between two sets of N sets, you need to calculate N(N-1) / 2 times, and the complexity is O(n 2 ). The calculation speed of a single Jaccard similarity will be more critical, especially if the set is relatively large, the calculation of the Jaccard similarity will be relatively time-consuming, and there will be greater pressure on computing resources. For example, according to the relationship between two programs Viewers to calculate the similarity between programs, each program may have millions of viewers, at this time the calculation of Jaccard similarity will be time-consumi...

AI Technical Summary

Problems solved by technology

[0016] However, in actual operation, it is found that the generation process of the minhash function is a process of randomly rearranging rows. If the number of rows (the overall number of elements) is large, the calculation process will be more complicated. At the same time, this rearrangement The calculation time required for permutation will be longer, that is, the minhash signature process will take longer

Images