A method and system for detecting duplicate video based on correlation characteristics

Abstract

The invention belongs to the technical field of computer data storage and discloses a repeated video detection method and system based on correlation properties in a mass storage system. The method uses position-sensitive hash functions to map repeated videos or similar videos to the same set of an index table, and accordingly the corresponding set can be directly located to perform finding when querying, traversing is not needed, the querying range is greatly shortened, and the querying speed of the repeated video detection method is improved. In the set, a Cuckoo Hashing mechanism is used for performing flat addressing in the whole index table, Hash collision in the set is reasonably solved, the index table maintains the correlation properties on the whole, the querying speed is not influenced, and accordingly the load balance is realized, and the use ratio of the index table is effectively improved.

Description

technical field [0001] The invention belongs to the technical field of computer data storage, and more particularly relates to a method and system for detecting repeated video based on correlation characteristics. Background technique [0002] With the emergence of video websites, the number and types of videos on the Internet are increasing rapidly, and users can easily upload or download videos through the Internet. However, in these massive online videos, there are a large number of repeated or nearly repeated videos. Existing studies have shown that by analyzing the search results of video search engines with specific keywords, it is found that there are 27% of repeated or nearly repeated videos in the search results on average. For individual query results, the repetition rate is even as high as 93%. These a large number of repeated videos not only waste storage space, but also bring inconvenience to the effective management of videos, and return too many repeated res...

AI Technical Summary

Problems solved by technology

[0004] (1) The current research is still on the issue of improving the accuracy of query results. Most of the methods use the tree structure as the index structure, and there is no more effective index structure to support fast query

[0005] (2) The scalability is poor. When the number of data increases with the expansion of the system, the query response time of the existing method will increase significantly

[0006] (3) The load is unbalanced. For different types of videos, the index structure is very different, which makes the load of each part of the system unbalanced.

Images