File data replication method for quick deduplication

Abstract

The invention discloses a file data replication method for quick deduplication. The method comprises the following steps of: before a first target file in a storage medium is wholly replicated to a target file directory for the first time, performing fingerprint calculation on metadata information of the first target file to form identity (ID1) by a processor, extracting contents of a plurality of file data segments from the first target file according to an interval with preset size and performing the fingerprint calculation to form ID2, performing the fingerprint calculation on the metadata information of a second target file to form ID3, and extracting contents of a plurality of file data segments from the second target file according to an interval with preset size and performing the fingerprint calculation to form ID4; comparing the ID1 with the ID3; if the ID1 is the same as the ID3, saving replication; if the ID1 is different from the ID3 and the ID2 is the same as the ID4, updating metadata of the first target file; and if the ID2 is different from the ID4, wholly replicating the second target file.

Description

technical field [0001] The invention relates to a method for duplicating data, in particular to a method for rapidly deduplicating file data. Background technique [0002] There are mainly two methods for existing file data copying: one is to deduplicate the entire file, and the other is to deduplicate internal data blocks of the file. The former refers to the improvement of the overall file deduplication algorithm. The traditional algorithm is called whole file detection (WFD). WFS technology uses files as the granularity to find duplicate data. First, fingerprint calculation (hash calculation) is performed on the entire file, and then the value is compared with the hash value of other files that have been stored. For comparison, if the same value is detected, only the file is replaced with a pointer, and if it is different, the entire file is transferred. The latter splits the file into smaller data segments and performs fingerprint calculation on the contents of the data...

AI Technical Summary

Problems solved by technology

In addition, even if the content of the file is changed, however, for some industries, such as the film and television industry, IT industry, etc., the number of files that need to be copied regularly is large, and the size of a single file is also large, and the modified part is usually large, and The distribution of the modified position in the file is not certain either. For example, the content after a large number of modifications exists in the middle or the end of the entire file. , to back up the file data, it is necessary to compare all the contents. Therefore, a lot of information that has not been modified is extracted for comparison, and the detection time in the early stage is greatly increased, which greatly reduces the efficiency of copying, and even worse It will cause the stagnation of industrial production, which will cause greater economic losses.

Images