Method, device and system for data processing

Abstract

The invention relates to the technical field of data processing, in particular to a method, a device and a system for data processing. The method for the data processing includes providing a first interface which receives a file write operation request from an application program; portioning the data of a file into at least one sub-data block, obtaining an unique identification of each of the at least one sub-data block, and comparing unique identifications with a reserved unique identification; establishing a chaining between a sub-data block identical to the reserved unique identification and the data corresponding to the reserved unique identification if the unique identifications are identical to the reserved unique identification; and otherwise, reserving the unique identifications of the sub-data block, which are different from the reserved unique identification, sending a data write operation request including the data of the sub-data block, which are different from the reserved unique identification, to a storage system by means of a second interface, and enabling the storage system to store the data of the sub-data block. According to the method, the device and the system for the data processing, the repeated data are reduced, and the storage space is saved.

Description

technical field [0001] The present invention relates to the technical field of data processing, in particular to a data processing method, device and system. Background technique [0002] After the data deduplication technology was proposed at the beginning of this century, it was quickly applied in various storage enterprises and became a symbolic function of storage technology. Data deduplication means storing only one copy of the same data to save space. Data deduplication can effectively reduce data and reduce storage costs. [0003] The existing data deduplication module is generally integrated in the storage system. The process is generally to slice the database, calculate the hash value (hash value) of each slice, and generate a logical address according to the hash value, and map from the logical address Stored as a physical address. This method is suitable for newly developed storage systems, but it is not suitable for some old storage systems. It needs to make a...

AI Technical Summary

Problems solved by technology

[0005] In the process of realizing the present invention, the inventor found at least the following problems in the prior art: the solution provided by the prior art can only support post-deduplication, but cannot perform real-time deduplication

This method makes repeated data read and write operations performed multiple times, takes up a lot of system resources, and causes a lot of storage space to be wasted, and cannot effectively save storage space

Images