Big data redundancy detection method

Abstract

The invention discloses a big data redundancy detection method. According to the method, all files are subjected to hash value detection, whether the files are redundant data are judged, then a fixed-length data block is selected randomly from the files and extended, whether the extended data block is the redundant data is judged, and finally, remaining data blocks of the files are detected. With the adoption of the big data redundancy detection method, not only can the redundant data be found out rapidly in a big data system, but also data matching strength can be effectively reduced for possibly emerging similar files, and redundant data blocks in the similar files can be found out as many as possible.

Description

【Technical field】 [0001] The invention belongs to the field of computers and the Internet, and in particular relates to the field of big data, in particular to a method for detecting redundancy of big data. 【Background technique】 [0002] With the rapid development of computer and Internet technology, we are in an era of information explosion. In order to process a large amount of information, the concept of big data has emerged. The so-called big data refers to a collection of data that cannot be captured, managed and processed by conventional software tools within an affordable time frame. , high growth rates and diverse information assets. [0003] Due to the massive nature of data, it is difficult for people to analyze these data on their own. However, under the backdrop of technological innovation represented by cloud computing, these data that were difficult to collect and use have become easy to use Yes, through continuous innovation in all walks of life, big data w...

AI Technical Summary

Problems solved by technology

However, simply using files as a unit to detect redundancy is too granular to handle some of the same data in similar files. Therefore, it is necessary to consider reducing the granularity to be able to detect some redundant data between similar files as much as possible.

Images