File semantics and system real-time state based redundant data deduplication method

Abstract

The invention discloses a file semantics and system real-time state based redundant data deduplication method. The method is mainly realized by three functional modules: a multi-semantic dimension division based deduplication priority calculation module (an MPD module), a hierarchical data deduplication module (a deduplicator) and a system real-time state based deduplication control module (a controller). The MPD module is based on multi-dimensional file semantics and outputs a file object subjected to deduplication operation in priority; the deduplicator, according to the output, sequentially executes hierarchical deduplication policies including global file-level deduplication and local chunk-level deduplication; and at the same time, in the operational process of the deduplicator, the controller, according to the system real-time state, dynamically adjusts the deduplicator so as to reduce more storage space costs while ensuring the read request response performance of a distributed main storage system.

Description

technical field [0001] The invention belongs to the technical field of computer information management, and in particular relates to a redundant data deduplication method based on file semantics and system real-time status. Background technique [0002] With the further popularization and in-depth application of cloud computing and mobile Internet, various online applications and services are playing a more important role in all walks of life. is growing at an explosive rate. The distributed storage system is the background support system of various cloud services. All service data are stored in the distributed storage system. The storage system provides a unified read-write interface for users or upper-level application services to access or modify and store in the disk. The data. Corresponding to different application scenarios and main purposes, distributed storage systems can be roughly divided into two categories: backup storage systems and primary storage systems. T...

AI Technical Summary

Problems solved by technology

Although these two solutions can meet certain deduplication requirements to a certain extent, their design does not consider the balance and trade-off between the application scenario of the primary storage system and other factors.

First of all, the data deduplication technology will cause the loss of data reading performance. GFD may cause the data accessed by the user to be transmitted from the remote server, which will cause delays in network transmission, and LCD will cause local disks. Fragmentation, which may require multiple disk seeks when reading a certain data, resulting in a delay in response to read operations

The greater the deduplication, the more the overhead on the storage device can be saved, but the read performance will be affected more severely, so the balance and trade-off between deduplication and read performance is a primary consideration.

Images