Differential compression method based on data de-duplication

Abstract

The invention discloses a differential compression method based on data de-duplication. The differential compression method includes steps of partitioning files in data flow to obtain multiple data blocks; computing data block fingerprint of each data block for searching duplicate data; grouping all the data blocks to establish data block groups and double link lists thereof; searching the fingerprint of each data block in each data block group for realizing data de-duplication so as to determine whether the data block is duplicated or not; searching similar data locally to the data block group which is subjected to the data de-duplication process according to the duplicated data information in the double link lists of the data block groups, namely, determining the non-duplicated data blocks adjacent to the duplicated data blocks as potential similar data blocks; verifying the similarity of the similar data blocks by differential compression; and finally complementarily searching similarity data to the data block groups according to the similarity. The differential compression method based on data de-duplication has the advantages of rapidness in similar data searching, low computing and indexing overhead and high data compression efficiency.

Description

technical field [0001] The invention belongs to the field of data compression for computer storage, and more specifically relates to a data difference compression method based on deduplication. Background technique [0002] In recent years, with the development and popularization of computer technology and networks, the amount of data information storage in the world has shown an explosive growth trend. Although the price of storage devices has been declining, it is far behind the speed of data expansion. Data deduplication (Data Deduplication), as a technology to effectively eliminate redundant data on a large scale, has become a hotspot in storage system research in recent years. To put it simply, the emerging data deduplication is an important technology to reduce the cost of data storage by effectively eliminating redundant data on a large scale. For example: Now a core department has 200GB of data that needs to be backed up every day, so it needs to back up 73TB a yea...

AI Technical Summary

Problems solved by technology

[0006] However, the existing differential compression technology has the following problems: its calculation speed is slow, the indexing overhead is large, the data compression efficiency is low, and the scalability is poor. If it is to support PB-level similar data retrieval, it will generate a 10TB-level similar data information index , these metadata cannot be put into the memory because they are too large, and at the same time, because they are put into the disk storage management, it brings the bottleneck of slow indexing

Such metadata management and indexing severely limit the promotion and development of differential compression algorithms.

Images