Fingerprint-based block granularity data deduplication system and method

Abstract

The invention provides a block granularity data deduplication system and method based on fingerprints. The method comprises the following steps: carrying out preliminary blocking on original input data; respectively calculating fingerprints, head fingerprints and tail fingerprints of the blocks obtained by partitioning and the heads and the tails of the blocks, and storing the fingerprints, the head fingerprints and the tail fingerprints in corresponding data structures; when the physical block address is read and input, requesting data from a lower-layer storage system, and returning the data after the data is read; when the data is written and input, outputting the data to the lower storage system, and returning the physical block address allocated by the storage system after the writing is completed; delta data are created, and original data are recovered according to the delta data; and adding, deleting, modifying and checking entries of the deduplication table, the head deduplication table, the tail deduplication table and the delta table in the system. According to the method, the deduplication effect of data with similar contents is focused on, and high I / O throughput and low system resource overhead are kept through design strategies of the I / O module and the index module.

Description

technical field [0001] The invention relates to the technical field of data De duplication and storage, in particular to a block granularity data De duplication system and method based on fingerprint. Background technology [0002] In the data storage task, the amount of data to be processed is increasing, which increases the space pressure of the storage system and the I / O throughput. In addition, the expansion of storage media increases the money cost, the service life of high-performance storage media is short, and a large amount of data writing further shortens the service life of the media. In order to solve the above problems, the usual practice is to de duplicate the data of the storage system. Specifically, the input data is processed in blocks, the fingerprint of each block is calculated, and the fingerprint to be written is compared with the fingerprint of the written block. If it is the same, it indicates that the data of the two blocks are repeated, the block is no ...

AI Technical Summary

Problems solved by technology

In short, the previous data deduplication methods faced two problems: (1) some blocks with some data changes could not be deduplicated (2) the block length was small, which increased the storage system overhead

Images