Approximately optimal data fragment block rewriting method for data deduplication technology

Abstract

The invention discloses an approximately optimal data fragment block rewriting method for a data deduplication technology. With the arrival of the data era, the total amount of the data is increased in an explosion mode, and the data storage and management requirements of the information world reach the PB level and even the EB level. It is found through IDC research that nearly 75% of data in thedigital world is redundant, and ESG indicates that the data redundancy in a backup and archiving system exceeds 90%. The data deduplication technology can effectively identify and eliminate repeateddata, and the data storage cost is reduced. However, data fragmentation can be caused by deleting repeated data, and data recovery performance is seriously damaged by data fragmentation. According tothe method, the fragment blocks in the data blocks can be accurately identified, and the fragmentation degree of the data is relieved by rewriting the fragment blocks; meanwhile, rewriting strategies(the optimal rewriting strategy and the aggressive rewriting strategy) can be switched in a self-adaptive mode according to different workloads, and therefore the data recovery performance and the data deduplication rate are greatly improved.

Description

technical field [0001] The invention relates to the technical field of data storage and data deduplication, in particular to an approximate optimal data fragment block rewriting method oriented to data deduplication technology. Background technique [0002] With the advent of the data age, the total amount of data in the world is showing an explosive growth trend. IDC research shows that by 2020, the world's annual data growth will increase 44 times from 0.8ZB in 2009 to 35ZB. With the increase of devices such as mobile devices and sensors, the growth of data is intensified. These data sources include human genes, social networks, financial analysis, environmental protection, energy exploration, video games, and medical health. More and more applications field. These data are not only huge in amount, but also complex and diverse in data structure, which brings new challenges to data storage and management, and also increases the risk of data management. How to effectively...

AI Technical Summary

Problems solved by technology

[0003] However, data deduplication technology introduces data fragmentation, which seriously damages the read performance of the data deduplication system, that is, data recovery performance.

The main method to alleviate data fragmentation is to rewrite fragmented blocks, but the current method of rewriting fragmented blocks (referred to as the rewriting method) has the disadvantages of a large deduplication rate of lost data and a small improvement in data recovery performance.

There are two main reasons for this shortcoming: (1) The current rewriting method only sorts the containers referenced by a data segment according to the container reference rate, and selects the container with the "lowest" container reference rate one-sidedly and arbitrarily. As a result, the containers selected by them are not optimal, and thus the rewritten fragment blocks are not optimal; (2) The current rewriting algorithm cannot adaptively adjust the rewriting strategy according to different workloads

Images