Method and system for global deduplication and storage device

Abstract

The invention discloses a method and system for deleting global repeating data and a storage device. The method comprises the following steps that 1.1, initialization is conducted; 1.2, a data stream is divided into a plurality of read-in data segments through a receiving layer, a read-in data segment fingerprint value is calculated for each read-in data segment, a management node corresponding to each read-in data segment fingerprint value is looked up, and the read-in data segment fingerprint values are sent to the management nodes; 1.3, whether the received read-in data segment fingerprint values exist in a fingerprint value index structure or not is judged through the management nodes, if yes, the read-in data segment is directly written in the storage device or a storage medium through the receiving layer, and the fingerprint value index structure is updated; if not, updating is conducted directly. The system is used for achieving the method. The storage device comprises the storage medium and a storage controller. The storage controller comprises the system for deleting the global repeating data. According to the method and system for deleting the global repeating data and the storage device, only the fingerprint values of the data segments need to be transmitted, not all the data segments need to be transmitted, and the interactive operation performance is greatly improved through establishment of the fingerprint value index structure and partition of a fingerprint value management range.

Description

technical field [0001] The present invention mainly relates to the field of data storage, in particular to a method and system for global deduplication of data storage devices. Background technique [0002] With the explosive growth of data volume, the amount of data stored in the storage system is increasing. According to the statistics of IDC, the total amount of global data reached trillions of GB in 2012, and more than 95% of the data is unstructured data; In many data-centric computing centers, the amount of new data generated every day has reached 100GB or even 1TB. At the same time, new storage media and their technologies, such as flash memory and phase change memory, are also developing. As a typical new type of storage medium, flash memory storage medium has the characteristics of high density, light weight, and low energy consumption, and is an ideal storage medium to replace the disk in the main storage system. The minimum read / write unit of flash memory is a f...

AI Technical Summary

Problems solved by technology

This method still has the following problems: 1. This method requires the transfer of all data to be written between the data redirection module and the controller, so that the controller can complete the detection of duplicate data and the redundant operation

Since frequent large-scale data transmission will occupy a large amount of controller bandwidth, this method will greatly reduce the read and write access performance of the system

2. This method does not efficiently organize and utilize the fingerprint value index in the metadata, and read requests need to access the fingerprint value index frequently. Therefore, in a storage system with frequent read requests, this method will reduce system access performance

Images