Method for deleting repeated data by using double-fingerprint hash check

Abstract

The invention provides a method for deleting repeated data by using double-fingerprint hash check. The method comprises the following steps of: dividing an object to be backed up into data blocks to be backed up with equal lengths; judging whether the weak fingerprint of one of the data blocks to be backed up is the same as the weak fingerprint of any data block in a server; if the weak fingerprint of one of the data blocks to be backed up is different from the weak fingerprint of any data block in the server, backing up the data block; if the weak fingerprint of one of the data blocks to be backed up is the same as the weak fingerprint of any data block in the server, judging whether the strong fingerprint of the data block is the same as the strong fingerprint of any data block in the server; if the strong fingerprint of the data block is different from the strong fingerprint of any data block in the server, backing up the data block; and performing the operation on all data blocks to be backed up. In the method for deleting the repeated data, a hash function with low calculation amount is adopted at the first time to perform weak check on every data block and a has function with high calculation amount is adopted then to perform strong check, so the problem that all the data are checked by the hash function with the high calculation amount is avoided, the fingerprint calculation amount during checking is reduced greatly, the series performance is enhanced, and the appreciable transmission performance is provided for data backup based on mass data storage.

Description

technical field [0001] The invention belongs to the field of computer storage technology and data backup technology, and in particular relates to a duplicate data deletion method using double-fingerprint hash verification. Background technique [0002] With the improvement of computer informatization, human society has entered the information age, and computers have penetrated into all walks of life in society. More and more applications are combined with computers, and people's work and life rely more and more on the Internet. The stronger it is, the more important the security of the network system and its data becomes. At the same time, the infinite expansion of the Internet has led to the explosive growth of data information in a geometric progression. Turing Award winner Jim Gray pointed out that the amount of new data added every 18 months in the network environment is equal to the sum of the amount of data in history. Almost all business activities of enterprises are...

AI Technical Summary

Problems solved by technology

If the number of fingerprints is small, the probability of collision during data block retrieval is high; if the number of fingerprints is large, the corresponding probability of collision is low, and it will cost more computation

And for KB-level fixed-length blocks, the number of blocks is very large, which leads to more complex calculations

In practical applications, in order to achieve a lower collision probability, a hash function with a high amount of calculation will be selected for fingerprint operation to reduce the probability of data block retrieval collision, which also greatly increases the calculation amount of data block fingerprints

Images