Convergence blocking method and device for data deduplication

Abstract

The invention discloses a convergence blocking method for data deduplication. The method comprises the following steps: recording the initial position of a data stream as a blocking position, and enabling a sliding window to gradually move forwards from the initial position; when the sliding window moves one step each time, judging whether the current position of the sliding window is the end position of the data stream; and if not, dynamically selecting judgment conditions to judge whether the current position is the blocking position according to the current blocking length. The invention also discloses a convergence blocking device for data deduplication. According to the convergence blocking method and device disclosed by the invention, the current blocking length is introduced as the blocking parameter during blocking, the extension or tightening of the blocking judgment conditions is dynamically selected, and thus the average block length of data blocks can be controlled, the total number of the data blocks can also be reduced, and the deduplication blocking efficiency can be increased.

Description

technical field [0001] The invention relates to the field of computers, in particular to a convergent block method and device for data deduplication in a storage system. Background technique [0002] At present, in the field of deduplication of data streams, blocks are used to obtain hash values ​​for comparison to determine whether the data stream is repeated. After the block is divided, the larger the average block length, the greater the granularity of deduplication of data streams, and the deduplication rate decreases; the average block The smaller the length, the lower the deduplication efficiency. However, the currently widely used content-based block method uses a byte-by-byte sliding window, and then calculates and judges the hash value of the window content. Due to the randomness of the hash value in the content-based block algorithm, after block The number of generated block lengths and the block length are exponentially distributed: there are a lot of ultra-small...

AI Technical Summary

Problems solved by technology

[0002] At present, in the field of deduplication of data streams, blocks are used to obtain hash values ​​for comparison to determine whether the data stream is repeated. After the block is divided, the larger the average block length, the greater the granularity of deduplication of data streams, and the deduplication rate decreases; the average block The smaller the length, the lower the deduplication efficiency

However, the currently widely used content-based block method uses a byte-by-byte sliding window, and then calculates and judges the hash value of the window content. Due to the randomness of the hash value in the content-based block algorithm, after block The number of generated block lengths and the block length are exponentially distributed: there are a lot of ultra-small data blocks and very large data blocks; among them, a lot of small data blocks are divided into blocks, which will lead to a large number of block data blocks. Large, resulting in an increase in the number of times to calculate and judge the hash value of the window content, reducing the efficiency of block

Images