A data deduplication method, apparatus, device, and storage medium

Abstract

The present application discloses a data deduplication method, apparatus, device and storage medium, which relates to the field of data deduplication technology and is applied to a distributed storage system. The method includes: when a data write request is detected, the written data is divided into blocks to obtain a number of data blocks; according to the storage information, the data blocks are divided into metadata objects and block objects and the metadata objects are stored in the metadata pool and the block objects are stored in the block pool; based on the block mapping between the metadata pool and the block pool, the target data identifier whose reference count in the reference count information reaches a preset threshold is screened out from the pre-built virtual fingerprint table, so that the target data identifier is submitted to the actual fingerprint table, and the corresponding data block position is determined according to the target data identifier to perform data deduplication. Through the technical solution of the present application, the efficiency of data deduplication can be improved, while the amount of data actually stored in the distributed storage system can be reduced, saving the operating costs of storage facility construction.

Description

Technical Field

[0001] The present invention relates to the field of data deduplication technology, and in particular to a data deduplication method, apparatus, device and storage medium. Background Art

[0002] Currently, distributed storage systems can expand capacity on demand, and their scale-out storage capabilities are being used in various computing fields. To more efficiently store massive amounts of data, data reduction technologies such as deduplication have become a research focus. Reducing data volume is an important method for reducing the overall cost of distributed storage systems. However, storing and managing exponentially growing massive amounts of data is a challenge. One solution to reducing data storage volume is deduplication, which removes redundant data from cluster nodes in a storage system. However, in existing technologies, applying traditional deduplication designs to distributed storage systems is not easy due to the following fundamental reasons.

[0003] Due to the metadata management and deduplication metadata processing inherent in distributed storage systems, applying deduplication to existing software stacks is extremely complex. In typical deduplication techniques, data streams or files are typically sliced and stored in a certain manner, with each shard having the same data block size. A hash calculation is then performed on the data block ID (identifier). The hash value is called the fingerprint (FP) information, and fingerprint comparison is used to identify identical data blocks for deletion. Designing deduplication in distributed storage systems faces the following challenges:

[0004] First, based on the horizontal scalability of the distributed storage system, the scalability of the fingerprint index table needs to be considered and managed. In the distributed storage system, redundancy solutions such as replicas or erasure codes are needed to achieve high availability to cope with failures of the underlying storage devices. Using traditional data deduplication methods will affect the integrity of redundant copy data in the distributed storage system.

[0005] Secondly, ensure the compatibility between the deleted fingerprint information and the existing stored metadata. Storing the fingerprint information of the deleted data will be challenging. If it is not stored in conjunction with a distributed storage metadata service, the indexing performance of the fingerprint table will be slow and difficult to manage.

[0006] Finally, the processing required by the data deduplication process, such as the use of CPU (Central Processing Unit) and memory resources, and the additional I / O (Input / Output) traffic, will degrade storage system performance.

[0007] In summary, how to improve the efficiency of data deduplication, solve the limitation of fingerprint tables that cannot be horizontally expanded, reduce the occupation of system resources and I / O traffic amplification, and be compatible with the existing functions of distributed storage systems are currently unresolved issues. Summary of the Invention

[0008] In view of this, the present invention aims to provide a data deduplication method, apparatus, device, and storage medium that can improve deduplication efficiency, address the limitation of fingerprint tables that cannot be horizontally expanded, reduce system resource usage and I / O traffic amplification, and be compatible with existing functions of distributed storage systems. The specific solution is as follows:

[0009] In a first aspect, the present application discloses a data deduplication method, which is applied to a distributed storage system, comprising:

[0010] When a data write request is detected, the write data is divided into blocks to obtain a number of data blocks;

[0011] The data block is divided into a metadata object and a block object according to the storage information, and the metadata object is stored in a metadata pool, and the block object is stored in a block pool; wherein the metadata object is used to record the block identifier of the data block and the metadata information of the written data; the block object is used to record the data identifier of the data block and the reference count information of the data identifier; the data identifier is an identifier of the data content corresponding to the block identifier determined by using a hash algorithm based on the block identifier;

[0012] Based on the block mapping between the metadata pool and the block pool, the target data identifiers whose reference counts in the reference count information reach a preset threshold are filtered out from the pre-built virtual fingerprint table, so that the target data identifiers are submitted to the actual fingerprint table, and the corresponding data block positions are determined according to the target data identifiers for data deduplication.

[0013] Optionally, the block mapping includes the offset range of the data block, the data identifier, the cache bit and the dirty bit.

[0014] Optionally, the block mapping between the metadata pool and the block pool includes:

[0015] Determining status information of the cache bit;

[0016] If the status information of the cache bit is TRUE, the data block is cached in the metadata object;

[0017] If the status information of the cache bit is FALSE, the data block is cached in the block object.

[0018] Optionally, the block mapping between the metadata pool and the block pool includes:

[0019] Determining status information of the dirty bit;

[0020] If the status information of the dirty bit is TRUE, the data block is determined to be a deduplication target;

[0021] If the status information of the dirty bit is FALSE, it is determined that the data block is not the deduplication target.

[0022] Optionally, the process of determining the corresponding data block location according to the target data identifier and performing data deduplication includes:

[0023] Acquire a dirty metadata object in the metadata object according to the metadata information, and determine a dirty block object corresponding to the dirty metadata object according to the block mapping;

[0024] When the dirty block object is the deduplication target, determining whether the block entry corresponding to the dirty block object already has the block identifier;

[0025] If the block identifier already exists in the block entry corresponding to the dirty block object, dereference the block object, and after dereferencing the block object, send the generated new block object to the block pool so as to generate a new block identifier according to the new block object;

[0026] If the block entry corresponding to the dirty block object does not have the block identifier, the generated new block object corresponding to the dirty block object is directly sent to the block pool so as to generate a new block identifier according to the new block object.

[0027] Optionally, the process of determining the corresponding data block location according to the target data identifier and performing data deduplication includes:

[0028] Performing a CRUSH mapping calculation on the new block identifier to determine a block position of the new block object;

[0029] Determine whether the block object exists at the block position; if the block object does not exist at the block position, store the object with a reference count of one in the reference count information of the block object; if the block object exists at the block position, add the reference count information to the current block object.

[0030] Optionally, the data deduplication method further includes:

[0031] The metadata extended attribute field xattr of the underlying storage is expanded, and the data identifier is recorded in the virtual fingerprint table.

[0032] In a second aspect, the present application discloses a data deduplication device, which is applied to a distributed storage system and includes:

[0033] A data block module is used to block the written data to obtain a number of data blocks when a data write request is detected;

[0034] an object partitioning module, configured to divide the data block into a metadata object and a block object according to storage information, and store the metadata object in a metadata pool and the block object in a block pool; wherein the metadata object is used to record the block identifier of the data block and metadata information of the written data; the block object is used to record the data identifier of the data block and reference count information of the data identifier; the data identifier is an identifier of the data content corresponding to the block identifier determined by a hash algorithm based on the block identifier;

[0035] The data deduplication module is used to filter out target data identifiers whose reference counts in the reference count information reach a preset threshold from a pre-built virtual fingerprint table based on the block mapping between the metadata pool and the block pool, so as to submit the target data identifiers to the actual fingerprint table and determine the corresponding data block positions according to the target data identifiers for data deduplication.

[0036] In a third aspect, the present application discloses an electronic device, comprising a processor and a memory; wherein the memory is used to store a computer program, and the computer program is loaded and executed by the processor to implement the data deduplication method as described above.

[0037] In a fourth aspect, the present application discloses a computer-readable storage medium for storing a computer program; wherein the computer program implements the data deduplication method as described above when executed by a processor.

[0038] In this application, it is applied to a distributed storage system. When a data write request is detected, the written data is divided into blocks to obtain a number of data blocks; the data blocks are divided into metadata objects and block objects according to the storage information, and the metadata objects are stored in the metadata pool and the block objects are stored in the block pool; wherein, the metadata object is used to record the block identifier of the data block and the metadata information of the written data; the block object is used to record the data identifier of the data block and the reference count information of the data identifier; the data identifier is an identifier of the data content corresponding to the block identifier determined by a hash algorithm based on the block identifier; based on the block mapping between the metadata pool and the block pool, the target data identifier whose reference count in the reference count information reaches a preset threshold is filtered out from the pre-constructed virtual fingerprint table, so that the target data identifier is submitted to the actual fingerprint table, and the corresponding data block position is determined according to the target data identifier for data deduplication. It can be seen that based on the distributed storage system, the data blocks are divided into metadata objects and block objects according to the different storage information, and the metadata objects are stored in the metadata pool and the block objects are stored in the block pool for hierarchical storage management. Since the two pools are divided according to their usage, each pool can be managed more effectively according to its different characteristics. Secondly, an improved hash algorithm is used to remap the ordinary block identifier based on the data block to a new data identifier based on the data content, removing the limitation that the traditional fingerprint index table is incompatible with the underlying storage system. In addition, through the layered idea, the data fingerprint is first saved in the virtual fingerprint table. The virtual fingerprint table counts the reference counts of the repeatedly written data blocks. When the reference count reaches the preset threshold, the virtual fingerprint table is submitted to the actual fingerprint table for deduplication. Therefore, the actual fingerprint table occupies less memory. This method reduces the memory resource occupation caused by the excessive size of the fingerprint table when massive data is written. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0040] Figure 1 This is a flow chart of a data deduplication method disclosed in this application;

[0041] Figure 2 A schematic diagram of the corresponding relationship between a block identifier and data content disclosed in this application;

[0042] Figure 3 A schematic diagram of a distributed hash mapping method based on block identification disclosed in this application;

[0043] Figure 4 A schematic diagram of a distributed hash mapping method based on data identification disclosed in this application;

[0044] Figure 5 A schematic diagram of a metadata pool and block pool design disclosed in this application;

[0045] Figure 6 A schematic diagram of a virtual fingerprint table process disclosed in this application;

[0046] Figure 7 This is a structural diagram of a data deduplication device disclosed in this application;

[0047] Figure 8 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0048] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0049] Currently, designing data deduplication in distributed storage systems faces the following challenges: First, using traditional data deduplication methods can compromise the integrity of redundant data copies in distributed storage systems. Second, managing fingerprint index tables based on data block identifiers in distributed storage systems is limited by the inability to scale horizontally. Finally, the processing required by the data deduplication process, including CPU and memory resource usage and additional I / O traffic, can degrade storage system performance.

[0050] To this end, this application provides a data deduplication solution that can improve the efficiency of deduplication, solve the limitation that fingerprint tables cannot be horizontally expanded, reduce the occupation of system resources and I / O traffic amplification, and be compatible with the existing functions of distributed storage systems.

[0051] The embodiment of the present invention discloses a data deduplication method, see Figure 1 As shown, applied to a distributed storage system, embodiment 1 includes:

[0052] Step S11: when a data write request is detected, the write data is divided into blocks to obtain a number of data blocks.

[0053] In the embodiment of the present application, when a data write request is detected, the written data is divided into several data blocks to more effectively detect redundancy. When deduplication is performed, if the data blocks are determined to be deduplication targets, then their current locations in the storage system are the same, and duplicate data will be automatically deleted.

[0054] Step S12: Divide the data block into a metadata object and a block object according to the storage information, and store the metadata object in a metadata pool and the block object in a block pool; wherein the metadata object is used to record the block identifier of the data block and the metadata information of the written data; the block object is used to record the data identifier of the data block and the reference count information of the data identifier; the data identifier is an identifier of the data content corresponding to the block identifier determined by a hash algorithm based on the block identifier.

[0055] The high availability, data recovery, and various data management operations of distributed storage systems are based on each object. Therefore, in the embodiments of this application, all the deduplication information involved is defined in a single object, so that the underlying distributed storage system can handle complex storage functions without additional modification.

[0056] In an embodiment of the present application, since the written data is judged whether to be deleted, the written data is divided into a number of data blocks, so the number of data blocks is an object including metadata objects (metadata) and block objects (chunk). Furthermore, an embodiment of the present application defines two pools based on the stored objects, the metadata pool stores metadata objects, and the block pool stores block objects. Since the two pools are divided according to their use, each pool can be managed more effectively according to its different characteristics. The metadata pool and the block pool can respectively select redundancy schemes between replication and erasure coding according to their usage, and each pool can be placed in different storage locations according to the required performance.

[0057] It is understandable that the hash value of the data block and the location of the block are mapped to the fingerprint table. Therefore, the data location information can be retrieved when the hash value of the new block is queried through the fingerprint index. The distributed storage system also determines the storage location of the object based on the hash algorithm. The difference lies in the different input values of the hash algorithm. The chunk value after the data is written for slice storage is the input of the deduplication method, that is, input = hash (chunk), and the data chunk identifier (ID) is the input value of the distributed storage system, that is, input = hash (Object ID). Therefore, in traditional deduplication technology, there is a problem of limited horizontal expansion of the fingerprint table index, and the deduplication method is difficult to be compatible with the distributed storage system function because the distributed storage system function cannot recognize the data structure of the deduplication method in traditional technology.

[0058] In an embodiment of the present application, the metadata object includes metadata for deduplication, specifically, for recording the block identifier of the data block and the metadata information of the written data. The block identifier of the data block is a common object ID visible to the user and provided by the underlying distributed storage system. The block object includes block data and its reference count information, specifically, for recording the data identifier of the data block and the reference count information of the data identifier. The data identifier of the data block is determined by the data content of the block, and is an identifier determined by calculating the content of the current block identifier using a hash algorithm. For example, the block identifier is 1 (object 1), Hash (1's data) = K, so the data identifier is K (object K).

[0059] Since there is no direct relationship between the data content of an object and its block identifier, the same data content may exist in multiple storage nodes. In this case, to find the same data content for deduplication, it is necessary to traverse all storage nodes or maintain a fingerprint index table of the same proportion. The embodiment of the present application uses an improved hash algorithm to remap the ordinary block identifier to a new data identifier based on data content. In this way, by combining two different input values and based on the relationship between the block identifier of the data and its data content, by adopting this improved hash algorithm mechanism, the problem of limited horizontal expansion of the fingerprint table index can be solved, and redundant duplicate data blocks can be retrieved more quickly.

[0060] Step S13: Based on the block mapping between the metadata pool and the block pool, the target data identifiers whose reference counts in the reference count information reach a preset threshold are filtered out from the pre-built virtual fingerprint table, so that the target data identifiers are submitted to the actual fingerprint table, and the corresponding data block positions are determined according to the target data identifiers to perform data deduplication.

[0061] In an embodiment of the present application, the metadata object is mapped to the chunk and stored based on its offset, and the chunk mapping can reflect the relationship between the metadata object and the chunk object. The chunk mapping includes the offset range (offset) of the data chunk, the data identifier (chunk ID), the cache bit (cache), and the dirty bit (dirty). The offset range and the data identifier represent the mapping information between the metadata object and the chunk object, and the cache bit and the dirty bit describe the status of the chunk. If the status information of the cache bit is true, the chunk is stored in the metadata object, otherwise, it is stored in the chunk object of the chunk pool. If the cache bit of all chunk mapping entries is false, there is no cached data in the data part of the metadata information of the data written in the metadata object. Secondly, if the dirty bit is true, the chunk needs to be deduplicated, and the data chunk is determined to be a deduplication target.

[0062] In an embodiment of the present application, data deduplication can be performed without introducing any external components. In this way, it can be compatible with the distributed storage system function. To this end, the embodiment of the present application extends the metadata extended attribute field (xattr) of the underlying storage to include metadata information of the deduplicated data. When deduplicating data, the background thread runs to perform data deduplication. Specifically, the dirty metadata object in the metadata object is obtained according to the metadata information, and the dirty block object corresponding to the dirty metadata object is determined according to the block mapping; when the dirty block object is the deduplication target, it is determined whether the block entry corresponding to the dirty block object already has the block identifier; if the block entry corresponding to the dirty block object already has the block identifier, the reference to the block object is canceled, and after the block object is dereferenced, the generated new block object is sent to the block pool so that a new block identifier is generated according to the new block object; if the block entry corresponding to the dirty block object does not have the block identifier, the generated new block object corresponding to the dirty block object is directly sent to the block pool so that a new block identifier is generated according to the new block object.

[0063] It is understood that the metadata information for writing data to a metadata object includes a dirty metadata object. Accordingly, the block object also includes a dirty block data identifier corresponding to the dirty metadata. All modifications or new write requests to the metadata object are recorded in the dirty metadata object. The dirty block data identifier is found in the dirty metadata object's block map. The dirty block is cached within the dirty object, i.e., the dirty block object corresponding to the dirty data identifier in the dirty metadata object and the block object. When the current data block is determined to be a deduplication target based on the dirty bit, a check is performed to see if the block entry corresponding to the dirty block object already has a block identifier. If it does, it has been referenced earlier by some block object. Therefore, a dereference message is sent to the current old block object and waited for its completion. Then, a new block object is generated and sent to the block pool, and the new block identifier is re-evaluated based on its content. If the dirty block object does not have a block identifier, a new block object is directly generated based on the dirty block object and sent to the block pool.

[0064] Furthermore, the generated new block object will be placed in the underlying distributed storage system through a hash algorithm. Among them, a CRUSH mapping calculation is performed on the new block identifier to determine the block location of the new block object. CRUSH (Controlled Replication Under Scalable Hashing) is a hash-based data distribution algorithm. It uses the data unique identifier, the topology of the current storage cluster, and the data backup strategy as CRUSH input. It can obtain the underlying storage device location of the data at any time and communicate with it directly, thereby avoiding table lookup operations and achieving decentralization and high concurrency. Find the data for the block identifier. Using the CRUSH algorithm, the client can know the location of the object in the storage bottom layer (i.e., the disk persistent storage device medium) and generate a new data identifier by hashing the content of the object in the storage bottom layer.

[0065] In an embodiment of the present application, if there is no object at the block location determined after performing a hash algorithm on the data content of the block identifier and performing a crush mapping calculation on the obtained data identifier, the object with a reference count of 1 is stored; if an object is already stored at that location, reference count information is added to the object. The reference count information may include the block pool (Pool ID), the block identifier (source object ID), and the offset. Furthermore, when the block pool information is written, the block map of the metadata object is updated.

[0066] In this embodiment, a layered approach is used to first store data fingerprints in a virtual fingerprint table (VFP). The VFP table counts references to repeatedly written data blocks. When the reference count reaches a preset value, the VFP table is submitted to the actual fingerprint table (FP) for deduplication. This reduces the memory usage of the actual fingerprint table FP, reducing the memory usage that would be caused by the excessive size of the fingerprint table FP when writing large amounts of data.

[0067] In this application, it is applied to a distributed storage system. When a data write request is detected, the written data is divided into blocks to obtain a number of data blocks; the data blocks are divided into metadata objects and block objects according to the storage information, and the metadata objects are stored in the metadata pool and the block objects are stored in the block pool; wherein, the metadata object is used to record the block identifier of the data block and the metadata information of the written data; the block object is used to record the data identifier of the data block and the reference count information of the data identifier; the data identifier is an identifier of the data content corresponding to the block identifier determined by a hash algorithm based on the block identifier; based on the block mapping between the metadata pool and the block pool, the target data identifier whose reference count in the reference count information reaches a preset threshold is filtered out from the pre-constructed virtual fingerprint table, so that the target data identifier is submitted to the actual fingerprint table, and the corresponding data block position is determined according to the target data identifier for data deduplication. It can be seen that based on the distributed storage system, the data blocks are divided into metadata objects and block objects according to the different storage information, and the metadata objects are stored in the metadata pool and the block objects are stored in the block pool for hierarchical storage management. Since the two pools are divided according to their usage, each pool can be managed more effectively according to its different characteristics. Secondly, an improved hash algorithm is used to remap the ordinary block identifier based on the data block to a new data identifier based on the data content, removing the limitation that the traditional fingerprint index table is incompatible with the underlying storage system. In addition, through the layered idea, the data fingerprint is first saved in the virtual fingerprint table. The virtual fingerprint table counts the reference counts of the repeatedly written data blocks. When the reference count reaches the preset threshold, the virtual fingerprint table is submitted to the actual fingerprint table for deduplication. Therefore, the actual fingerprint table occupies less memory. This method reduces the memory resource occupation caused by the excessive size of the fingerprint table when massive data is written.

[0068] The following describes in detail Example 2 of a data deduplication method provided by this application. Building on Example 1, Example 2 provides a detailed description of a data deduplication system using practical applications as an example. It should be understood that the specific embodiments described herein are merely illustrative of the present invention and are not intended to limit it. Furthermore, the technical features involved in the various embodiments of the present invention described below may be combined as long as they do not conflict with each other.

[0069] See also Figures 2 to 6 The second embodiment specifically includes three aspects: the comparison of hash algorithm and improved hash algorithm in distributed storage system, the design of metadata pool and block pool system in data deduplication system, and the virtual fingerprint table process of distributed storage system. Each aspect is explained in detail below.

[0070] (1) Based on the distributed storage system, in order to solve the current problem of limited horizontal scalability of fingerprint indexes, the embodiment of the present application can manage the fingerprint table FP, enable fast search, and evenly distribute the fingerprint table in the distributed storage cluster system after expansion. Therefore, an improved hash algorithm based on the content of distributed storage objects is proposed. Since the key mechanism of the fingerprint table index is to retrieve redundant duplicate data blocks more quickly, the traditional deduplication technology is based on the hash algorithm for calculation. The difference lies in the different input values of the hash algorithm. When deduplicating data, the chunk value after the data is written for slicing and storage is the input of the deduplication method, that is, input = hash (chunk), and the object ID is the input value of the distributed storage system, that is, input = hash (Object ID). It can be seen that the distributed storage system function cannot recognize the data structure of the deduplication method in the traditional technology, and therefore it is difficult to be compatible with the distributed storage system function.

[0071] In the embodiment of the present application, the hash algorithm is improved by combining two different input values, such as Figure 2 The figure shows the relationship between the block identifier of the data and its data content (data). It can be seen that in this example, block identifiers 1, 2, and 3 have the same content. Figure 3 This paper shows a traditional hash mapping method for a distributed storage system addressed by block identifiers. Each client can use a predefined hash algorithm to find the object data corresponding to the block identifier. Since there is no relationship between the data content of the object and the block identifier, the same data content may exist in multiple storage nodes. In this case, to find the same data content for deduplication, it is necessary to traverse all storage nodes or maintain a fingerprint index table of the same proportion. Therefore, this embodiment remaps the ordinary block identifier-based data identifier to a new data content-based data identifier by using an improved hash algorithm. Figure 4 As shown, taking client 1 as an example, the corresponding block ID for client 1 is 1, Client 1->OID=1. The data content of the object with OID=1 is hashed, Hash(1's data)=K. Furthermore, the K value is CRUSH mapped in the distributed storage system. Each client can use the distributed storage system's CRUSH algorithm to find the corresponding data content (CRUSH(K)) for the block ID, ultimately determining the location of the chunk requiring deduplication. Therefore, by adopting this improved hashing algorithm, the client can determine the location of the object in the underlying storage layer (i.e., the disk persistent storage device medium). By generating data IDs by hashing the object's content in the underlying storage layer, the traditional fingerprint index table can be eliminated, maintaining compatibility with the underlying storage system's scalability and reducing the complexity of duplicate block detection in existing hashing mechanisms.

[0072] (2) The high availability, data recovery, and various data management operations of the distributed storage system are based on each object. Therefore, the embodiment of the present application saves the distributed storage system and the data that needs to be deduplicated into one object. The objects are divided into two categories according to the stored information: metadata objects and chunk objects, forming a global metadata structure. The underlying distributed storage system can handle complex storage functions without additional modification. In addition, by extending the metadata extended attribute field (xattr) of the underlying storage to include metadata information of the deduplicated data, it can be compatible with the distributed storage system functions without introducing any external components.

[0073] like Figure 5 The figure shows the design of the metadata pool and block pool in a data deduplication system. Metadata objects are stored in the metadata pool in distributed storage and contain metadata used for deduplication. In a data deduplication system, data is divided into multiple blocks based on its block partitioning algorithm to more effectively detect redundancy. The ID of the metadata object is a user-visible normal object ID, also known as the block ID, provided by the underlying distributed storage system. The mapping of metadata objects to chunks is based on their storage offset. A chunk mapping consists of an offset range, a data identifier (chunk ID), a cache bit, and a dirty bit.

[0074] The offset range and data identifier represent the mapping information between the metadata object and the block object, and the cache bit and dirty bit describe the state of the block. If the cache bit is true, the block is stored in the metadata object, otherwise, it is stored in the block object of the block pool. If the dirty bit is true, the block needs to be deduplicated. If the cache bit of all block mapping entries is false, there is no cached data in the data part of the object. Figure 5 In the example, the block identifier 2 is metadata, and its block map represents all the blocks that make up the object. That is, data identifiers B and C are not cached. Therefore, object 2 does not contain data, but only metadata. On the contrary, if the cache bit is true, the block is stored inside the object. Figure 5 This is the case for objects 1 and 3 in .

[0075] In this embodiment, the block object is stored in the block pool, and the block object contains the block data and its reference count information. Figure 5 In the figure, B, C, and D represent data identifiers, which belong to the block object. The data is stored in the data part of the object, and the reference counting information (Pool ID, source object ID, offset) is stored in the metadata part of the object. The data identifier is determined by the data content of the block.

[0076] As can be seen, this embodiment defines two pools based on the objects being stored: the metadata pool stores metadata objects, and the block pool stores block objects. Because these two pools are divided based on usage, each pool can be more efficiently managed based on its distinct characteristics. The metadata pool and the block pool can each choose between replication and erasure coding as a redundancy scheme based on their usage, and each pool can be placed in a different storage location based on the required performance.

[0077] (3) Figure 6 The figure shows the fingerprint reference counting method for the data to be deleted in this embodiment, which shows the virtual fingerprint table process of the distributed storage system. First, the distributed storage client writes the data and performs fixed block slicing into multiple chunks 1 to chunkn; according to the above-mentioned improved HASH calculation method, the fingerprint is recorded in the designed extensible metadata attribute XATTR; secondly, the virtual fingerprint table Virtual Fingerprint (VFP) is designed, and the data fingerprint is first saved in the virtual fingerprint table through the layered idea. The VFP table performs statistical screening on the reference count of the repeatedly written data blocks. When the reference count reaches a preset value, such as the preset threshold repetition number is 3 (configurable), the VFP table will be submitted to the actual fingerprint table FP for deduplication operation. Based on the above steps, the actual fingerprint table FP occupies less memory, which reduces the memory resource occupation caused by the excessive size of the fingerprint table FP when massive data is written.

[0078] In addition, since the distributed storage system and the data to be deduplicated are stored in one object, during the data deduplication process, the cache module manages (cache) and evaluates whether a block needs to be cached. If a block is cached, it is stored in the data part of the metadata object, such as Figure 5 For the object with block ID 1, its cached block is the object with data ID A. Caching data can eliminate the overhead of data deduplication. However, in practice, caching an object from the block pool to the metadata pool requires a strategy, as storing an object incurs additional I / O requests and consumes storage capacity. Objects in the metadata pool can have duplicate blocks, and cache updates and evictions are managed using the LRU cache eviction algorithm.

[0079] Because blocks are the basic unit for detecting redundancy in given data, when a data write request reaches the deduplication system, the data is divided into several blocks, and the data identifier is used as the input key of the hash algorithm of the underlying distributed storage system (i.e., chunk pool). If the contents of two blocks are identical, their locations in the storage system are identical, and the duplicate data will naturally be deleted. The deduplication module is designed to be run by a background thread. During the deduplication operation, the background deduplication module first searches the dirty object ID list for the dirty metadata object containing the dirty block. All modifications or new write requests to the metadata object are recorded in the dirty object ID list; secondly, the data representation of the dirty block is found from the block map of the dirty metadata object. If the cache management module determines that the dirty block is a deduplication target, the deduplication module checks whether the block entry corresponding to the dirty block already has a block object ID. If it has a block object ID, it was previously referenced by some block objects. Therefore, the data deduplication module sends a dereference message to the old block object and waits for its completion. It then generates a new block object and sends it to the block storage pool, while re-evaluating the new block's data identifier based on its content. If the dirty block does not have a block object ID, a new block corresponding to the dirty block is directly generated and sent to the block pool. It is then placed in the underlying distributed storage system using a hash algorithm. If there is no object at the location determined by the hash algorithm, the object with a reference count of 1 is stored. If an object is already stored at that location, reference count information is added to the object. When the chunk pool's chunk write completes, the metadata object's block map is updated. In this way, the virtual fingerprint table performs filtering and statistics based on the reference count information in the block map.

[0080] As can be seen, the data deduplication method provided in this embodiment can improve deduplication efficiency, reduce system resource usage and I / O traffic amplification, and be compatible with existing distributed storage system functions. It also reduces the amount of data actually stored in the distributed storage system, saving the operating costs of storage facility construction. This overcomes the limitation of the existing technology that the fingerprint table FP cannot be horizontally expanded. In addition, the metadata information of the distributed storage system and deduplicated data is stored in a single object to form a global metadata structure, which optimizes and controls deduplication efficiency. This makes the present invention compatible with existing distributed storage functions, data fault recovery, snapshots, horizontal expansion, and other features, while not affecting the integrity of the distributed storage redundant data.

[0081] In summary, this embodiment has at least the following advantages:

[0082] 1. Performance: The present invention can improve the efficiency of data deduplication, reduce the occupation of system resources and the problem of I / O traffic amplification, and achieve certain performance improvements.

[0083] 2. Stability. The deduplication process is completely transparent to upper-layer business applications. The deduplication hash algorithm and data extension attributes are compatible with the distributed storage system service functions, making the distributed storage system stable.

[0084] 3. Security: The present invention adopts a content-based hash algorithm with a low data collision rate, thus ensuring data consistency in the distributed storage system.

[0085] 4. Low cost: The main purpose of the present invention is to reduce the amount of data actually stored in the distributed storage system and save the operating costs of storage facility construction.

[0086] 5. Compatibility: The present invention is compatible with existing distributed storage functions, data failure recovery, snapshots, horizontal expansion and other features, and does not affect the integrity of distributed storage redundant data.

[0087] Correspondingly, the embodiment of the present application also discloses a data deduplication device, which is applied to a distributed storage system. Figure 7 As shown, the device includes:

[0088] The data block module 11 is used to block the written data to obtain a number of data blocks when a data write request is detected;

[0089] The object partitioning module 12 is configured to divide the data block into a metadata object and a block object according to the storage information, and store the metadata object in a metadata pool and the block object in a block pool; wherein the metadata object is used to record the block identifier of the data block and the metadata information of the written data; the block object is used to record the data identifier of the data block and the reference count information of the data identifier; the data identifier is an identifier of the data content corresponding to the block identifier determined by a hash algorithm based on the block identifier;

[0090] The data deduplication module 13 is used to filter out target data identifiers whose reference counts in the reference count information reach a preset threshold from a pre-built virtual fingerprint table based on the block mapping between the metadata pool and the block pool, so as to submit the target data identifiers to the actual fingerprint table and determine the corresponding data block positions according to the target data identifiers for data deduplication.

[0091] Among them, for more specific working processes of the above modules, please refer to the corresponding contents disclosed in the aforementioned embodiments, which will not be repeated here.

[0092] Thus, it can be seen that, through the above scheme of this embodiment, applied to a distributed storage system, when a data write request is detected, the written data is divided into blocks to obtain a number of data blocks; the data blocks are divided into metadata objects and block objects according to the storage information, and the metadata objects are stored in the metadata pool and the block objects are stored in the block pool; wherein, the metadata object is used to record the block identifier of the data block and the metadata information of the written data; the block object is used to record the data identifier of the data block and the reference count information of the data identifier; the data identifier is an identifier of the data content corresponding to the block identifier determined by a hash algorithm based on the block identifier; based on the block mapping between the metadata pool and the block pool, the target data identifier whose reference count in the reference count information reaches a preset threshold is filtered out from the pre-built virtual fingerprint table, so that the target data identifier is submitted to the actual fingerprint table, and the corresponding data block position is determined according to the target data identifier for data deduplication. It can be seen that based on the distributed storage system, the data blocks are divided into metadata objects and block objects according to the different storage information, and the metadata objects are stored in the metadata pool and the block objects are stored in the block pool for hierarchical storage management. Since the two pools are divided according to their usage, each pool can be managed more effectively according to its different characteristics. Secondly, an improved hash algorithm is used to remap the ordinary block identifier based on the data block to a new data identifier based on the data content, removing the limitation that the traditional fingerprint index table is incompatible with the underlying storage system. In addition, through the layered idea, the data fingerprint is first saved in the virtual fingerprint table. The virtual fingerprint table counts the reference counts of the repeatedly written data blocks. When the reference count reaches the preset threshold, the virtual fingerprint table is submitted to the actual fingerprint table for deduplication. Therefore, the actual fingerprint table occupies less memory. This method reduces the memory resource occupation caused by the excessive size of the fingerprint table when massive data is written.

[0093] Furthermore, the embodiment of the present application also discloses an electronic device, Figure 8 This is a structural diagram of an electronic device 20 according to an exemplary embodiment, and the content in the diagram cannot be considered as any limitation to the scope of use of the present application.

[0094] Figure 8 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present application. The electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps of the data deduplication method disclosed in any of the aforementioned embodiments.

[0095] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device. The communication protocol it follows is any communication protocol that can be applied to the technical solution of this application and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world. Its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0096] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, a magnetic disk, or an optical disk. The resources stored thereon may include an operating system 221, a computer program 222, and data 223. The data 223 may include various data. The storage method can be temporary storage or permanent storage.

[0097] The operating system 221 is used to manage and control the hardware devices and computer program 222 on the electronic device 20. The operating system 221 can be Windows Server, NetWare, Unix, Linux, etc. In addition to including a computer program capable of implementing the data deduplication method performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include computer programs capable of performing other specific tasks.

[0098] Furthermore, the embodiments of the present application also disclose a computer-readable storage medium, where the computer-readable storage medium includes a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a magnetic disk or an optical disk, or any other form of storage medium known in the technical field. Wherein, when the computer program is executed by the processor, the aforementioned data deduplication method is implemented. For the specific steps of the method, please refer to the corresponding content disclosed in the aforementioned embodiment, and no further details will be given here.

[0099] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. Reference can be made to the descriptions of the identical or similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and the relevant parts can be referred to the descriptions of the methods.

[0100] The data deduplication or algorithm steps described in conjunction with the embodiments disclosed herein can be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module can be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0101] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0102] The above is a detailed introduction to the data deduplication method, device, equipment and storage medium provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas. At the same time, for general technical personnel in this field, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A data deduplication method, characterized in that: Applied to distributed storage systems, including: When a data write request is detected, the write data is divided into blocks to obtain a number of data blocks; The data block is divided into a metadata object and a block object according to the storage information, and the metadata object is stored in a metadata pool, and the block object is stored in a block pool; wherein the metadata object is used to record the block identifier of the data block and the metadata information of the written data; the block object is used to record the data identifier of the data block and the reference count information of the data identifier; the data identifier is an identifier of the data content corresponding to the block identifier determined by using a hash algorithm based on the block identifier; Based on the block mapping between the metadata pool and the block pool, the target data identifiers whose reference counts in the reference count information reach a preset threshold are filtered out from the pre-built virtual fingerprint table, so that the target data identifiers are submitted to the actual fingerprint table, and the corresponding data block positions are determined according to the target data identifiers for data deduplication.

2. The data deduplication method according to claim 1, wherein: The block map includes the offset range of the data block, the data identifier, a cache bit, and a dirty bit.

3. The data deduplication method according to claim 2, wherein: The block mapping based on the metadata pool and the block pool includes: Determining status information of the cache bit; If the status information of the cache bit is TRUE, the data block is cached in the metadata object; If the status information of the cache bit is FALSE, the data block is cached in the block object.

4. The data deduplication method according to claim 2, wherein: The block mapping based on the metadata pool and the block pool includes: Determining status information of the dirty bit; If the status information of the dirty bit is TRUE, the data block is determined to be a deduplication target; If the status information of the dirty bit is FALSE, it is determined that the data block is not the deduplication target.

5. The data deduplication method according to claim 4, wherein: The process of determining the corresponding data block position according to the target data identifier and performing data deduplication includes: Acquire a dirty metadata object in the metadata object according to the metadata information, and determine a dirty block object corresponding to the dirty metadata object according to the block mapping; When the dirty block object is the deduplication target, determining whether the block entry corresponding to the dirty block object already has the block identifier; If the block identifier already exists in the block entry corresponding to the dirty block object, dereference the block object, and after dereferencing the block object, send the generated new block object to the block pool so as to generate a new block identifier according to the new block object; If the block entry corresponding to the dirty block object does not have the block identifier, the generated new block object corresponding to the dirty block object is directly sent to the block pool so as to generate a new block identifier according to the new block object.

6. The data deduplication method according to claim 5, wherein: The process of determining the corresponding data block position according to the target data identifier and performing data deduplication includes: Performing a CRUSH mapping calculation on the new block identifier to determine a block position of the new block object; Determine whether the block object exists at the block position; if the block object does not exist at the block position, store the object with a reference count of one in the reference count information of the block object; if the block object exists at the block position, add the reference count information to the current block object.

7. The data deduplication method according to any one of claims 1 to 6, characterized in that: Also includes: The metadata extended attribute field xattr of the underlying storage is expanded, and the data identifier is recorded in the virtual fingerprint table.

8. A data deduplication device, characterized in that: Applied to distributed storage systems, including: A data block module is used to block the written data to obtain a number of data blocks when a data write request is detected; an object partitioning module, configured to divide the data block into a metadata object and a block object according to storage information, and store the metadata object in a metadata pool and the block object in a block pool; wherein the metadata object is used to record the block identifier of the data block and metadata information of the written data; the block object is used to record the data identifier of the data block and reference count information of the data identifier; the data identifier is an identifier of the data content corresponding to the block identifier determined by a hash algorithm based on the block identifier; The data deduplication module is used to filter out target data identifiers whose reference counts in the reference count information reach a preset threshold from a pre-built virtual fingerprint table based on the block mapping between the metadata pool and the block pool, so as to submit the target data identifiers to the actual fingerprint table and determine the corresponding data block positions according to the target data identifiers for data deduplication.

9. An electronic device, characterized in that: The electronic device includes a processor and a memory; wherein the memory is used to store a computer program, and the computer program is loaded and executed by the processor to implement the data deduplication method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that Used to store a computer program; wherein when the computer program is executed by a processor, the data deduplication method according to any one of claims 1 to 7 is implemented.

Images