Data deduplication method

Abstract

The invention discloses a data deduplication method, which is characterized in that a hash calculation task is unloaded to a hardware accelerator in a DPU (Data Processing Unit), so that the hash calculation speed is greatly improved, and the request response time is reduced. As the CPU of the host does not participate in Hash calculation, part of calculation resources are released, other services on the host can also use the part of calculation resources, and the utilization rate of the CPU is increased. And a CPU utilization rate perceived block task unloading method is adopted. And when the object deduplication task arrives, the utilization rate of the ARM-CPU in the DPU is checked, if the utilization rate is too high, the host CPU is used for partitioning, and then the partitioned data blocks are sent to the DPU. Otherwise, the object is integrally sent to the DPU, namely, the partitioning task and the Hash task are unloaded to the DPU together. Therefore, the resource consumption of the host CPU can be reduced, and the situation that the load of the ARM-CPU in the DPU is too high, and consequently the network task of the DPU is affected can be avoided.

Description

Technical Field

[0001] The present invention relates to the field of computer storage, and in particular to a data deduplication method. Background Art

[0002] Distributed object storage has a wide range of application scenarios due to its high scalability, elasticity and cost-effectiveness. Distributed object storage can efficiently manage massive amounts of data and is suitable for enterprise data backup and long-term archiving, especially when regulations require long-term data preservation. Because object storage can provide high-throughput and low-latency storage services, it is suitable for the storage and distribution of media files such as video and audio, including streaming platforms and content delivery networks (CDNs). Distributed object storage can support the storage of unstructured data, such as logs, sensor data, machine learning training data, etc., which are huge in volume and highly diverse. Object storage can be integrated with big data platforms such as Hadoop and Spark to achieve efficient data storage and processing.

[0003] Many modern cloud-native applications rely on object storage to store application status, configuration files, user-uploaded data, etc. Cloud providers' object storage services (such as AWS S3 and Azure Blob Storage) are widely used to support serverless architecture and containerized applications. Object storage is suitable for storing and accessing large files and file version control. Enterprises can use it to build file sharing and collaboration platforms. Distributed object storage supports massive amounts of data while making data storage and access more convenient and efficient through flexible management and expansion methods.

[0004] As the amount of data continues to grow, storage overhead also increases. To reduce storage overhead, many companies and researchers have adopted a variety of strategies and technologies: Data compression: Reduce the amount of data storage through compression algorithms. For example, compressing data with LZ77, LZ4, etc. before transmission and storage can greatly reduce the size of data that needs to be stored and reduce storage costs. Data deduplication: Data deduplication is the identification and deletion of duplicate data, retaining only the only copy of the data. This technology is particularly effective in backup systems, file storage, and virtualization environments, and can significantly reduce storage space requirements. Tiered storage: Put different types of data on different storage media to reduce overall costs. For example, put cold data on low-cost hard drives or tapes, while hot data is stored on high-performance but more expensive SSDs.

[0005] Block-level data deduplication technology is usually used to reduce storage overhead. Data deduplication technology reduces the actual amount of data that needs to be stored by removing duplicate data blocks or files and retaining only unique copies. Block-level deduplication: Divide the file into several fixed or variable-sized data blocks, generate a unique hash fingerprint for each block, and store the data of the unique block. If the same hash value is detected, the block is considered to be duplicate data, and only the reference information needs to be stored, not the duplicate data block. This method further refines the deduplication granularity based on file-level deduplication, saving more space.

[0006] However, hash calculations will increase the computational overhead during the data deduplication process, especially when processing a large number of data blocks or files, the time consumption of hash calculations will be more obvious. In short, some optimization measures can indeed be used to improve hash efficiency and reduce the impact on system performance. For example, choose a lightweight hash algorithm, and try to choose a hash algorithm with a faster calculation speed and an acceptable conflict rate. For example, MD5 and MurmurHash have a faster calculation speed. However, for large-scale data deduplication applications, these hash algorithms may cause hash conflicts and lead to data loss. Therefore, using the SHA algorithm is still the safest approach.

[0007] In a traditional Ethernet environment, the bottleneck of a distributed storage system is often concentrated on network transmission rather than CPU computing power. Even though hash calculations require a certain amount of CPU resources, this part of the calculation time is usually not prominent compared to the delay and bandwidth limitations of network transmission. Traditional Ethernet (such as 1Gbps or 10Gbps) is subject to bandwidth limitations during large-scale data transmission, resulting in a low transmission rate. The computing power of the CPU usually far exceeds the network transmission rate, so the time taken for hash calculations is relatively small and will not become a bottleneck. Ethernet is also affected by latency, especially in distributed storage across data centers. Network latency causes communication waits between storage nodes, further amplifying the network bottleneck and masking the CPU hash calculation time.

[0008] The intelligent network card (Data Process Unit, DPU) supports high-speed remote direct memory access (Remote Direct Memory Access, RDMA) based on InfiniBand (IB). When the distributed object storage system adopts DPU, the network transmission bottleneck is greatly alleviated, and data transmission can reach a level close to the memory speed. In this way, the bottleneck will be transferred from the network to the CPU, because the CPU needs to undertake a large number of data processing tasks, such as hash calculation, compression, etc. However, when the CPU is used to process data processing tasks, there is sometimes a problem of slow response time. Summary of the invention

[0009] The purpose of the present invention is to overcome the above-mentioned defects or problems existing in the background technology and provide a data deduplication method.

[0010] To achieve the above objectives, the present invention and its preferred embodiments adopt the following technical solutions, but the embodiments are not limited to the following solutions:

[0011] Solution 1: A data deduplication method.

[0012] The data area of ​​the storage system is divided into a cache area and a chunk area. The cache area stores the object's dedup tag, cache tag, data part, and hash value list. The chunk area stores chunk data and the hash value corresponding to the chunk data. A cached value of 0 means that the object's data cannot be found in the cache area, and a dedup value of 1 means that it has been deleted.

[0013] Based on the storage system, the data deduplication method includes the following steps:

[0014] When the first storage node in the cache area obtains the object deduplication request, it checks the utilization rate of the ARM-CPU of the DPU. If it is higher than or equal to the threshold, the host CPU divides the object into blocks and submits all data blocks to the smart network card daemon. Otherwise, the object is submitted to the smart network card daemon, and the smart network card daemon divides the object into blocks.

[0015] The smart network card daemon process calculates a hash value of each data block, and the smart network card daemon process returns the hash value to the first storage node;

[0016] The first storage node combines the data portion of each data block and the hash value corresponding to the data block into a chunk object, sends the chunk object to the second storage node in the chunk area, and deletes the data portion of the object in the cache area;

[0017] The second storage node searches the local hash value table, and if the hash value exists, does not store it; if the hash value does not exist, stores it; and sets the dedup mark of the object in the cache area to 1, and sets the cache mark to 0.

[0018] Solution 2: Based on Solution 1, when the first storage node in the cache area obtains the object deduplication request, it checks the utilization rate of the ARM-CPU of the DPU. If it is higher than or equal to the threshold, the host CPU blocks the object and submits all data blocks to the smart network card daemon through DMA; otherwise, the object is submitted to the smart network card daemon through DMA, and the smart network card daemon blocks the object.

[0019] The smart network card daemon calculates a hash value for each data block and returns the hash value to the first storage node using RDMA.

[0020] Solution three is based on solution two, wherein the host CPU divides the object into blocks by using a CDC algorithm, and the smart network card daemon divides the object into blocks by using an ARM-CPU.

[0021] Solution 4: an object writing method.

[0022] The data area of ​​the storage system is divided into a cache area and a chunk area. The cache area stores the object's dedup tag, cache tag, data part, and hash value list. The chunk area stores chunk data and the hash value corresponding to the chunk data. A cached value of 0 means that the object's data cannot be found in the cache area, and a dedup value of 1 means that it has been deleted.

[0023] Based on the storage system, the object writing method includes the following steps:

[0024] When the first storage node in the cache area obtains an object write request;

[0025] The first storage node distributes the copy of the object to multiple redundant storage nodes in the cache area;

[0026] The cached flag of the object is set to 1 and the dedup flag is set to 0.

[0027] Solution 5: Based on Solution 4, the client calculates the hash value of the object. Based on the hash value of the object, the client uses the CRUSH algorithm to calculate the object to select the first storage node to write.

[0028] Solution 6: An object reading method.

[0029] The data area of ​​the storage system is divided into a cache area and a chunk area. The cache area stores the object's dedup tag, cache tag, data part, and hash value list. The chunk area stores chunk data and the hash value corresponding to the chunk data. A cached value of 0 means that the object's data cannot be found in the cache area, and a dedup value of 1 means that it has been deleted.

[0030] Based on the storage system, the object reading method includes the following steps:

[0031] When the first storage node in the cache area receives an object read request,

[0032] When the dedup mark of the object is 0 or 1, and the cached mark is 1, the first storage node returns the data of the object;

[0033] Otherwise, the first storage node traverses the recipe of the object, sends a chunk object read request to the second storage node of the chunk area, caches all chunk objects of the object to the first storage node, and the first storage node returns the data of the object and sets the cache mark of the object to 1.

[0034] Solution 7: Based on Solution 6, when the first storage node in the cache area receives an object read request,

[0035] Check the dedup flag of the object, if it is 0, the first storage node returns the data of the object; otherwise, check the cached flag of the object, if it is 1, the first storage node returns the data of the object;

[0036] Otherwise, the first storage node traverses the recipe of the object, sends a chunk object read request to the second storage node of the chunk area, caches all chunk objects of the object to the first storage node, and the first storage node returns the data of the object and sets the cache mark of the object to 1.

[0037] Solution 8, an object eviction method,

[0038] The data area of ​​the storage system is divided into a cache area and a chunk area. The cache area stores the object's dedup tag, cache tag, data part, and hash value list. The chunk area stores chunk data and the hash value corresponding to the chunk data. A cached value of 0 means that the object's data cannot be found in the cache area, and a dedup value of 1 means that it has been deleted.

[0039] Based on the storage system, the object eviction method comprises the following steps:

[0040] Scan all objects in the cache area regularly. If the dedup flag of the object is 0, skip the object.

[0041] Otherwise, the cached tag of the object is checked. If it is 0, the object is skipped. Otherwise, it is determined whether the object is evicted according to the cache replacement algorithm. If it is evicted, the cached tag of the object is set to 0, and then the data cache of the object is deleted.

[0042] Solution 9: A data processing system comprising:

[0043] A storage system, which is suitable for dividing a data area into a cache area and a chunk area; wherein the cache area stores a dedup tag, a cache tag, a data part, and a hash value of an object; and the chunk area stores chunk data and a hash value corresponding to the chunk data; cached being 0 means that the data of the object cannot be found in the cache area, and dedup being 1 means that the data has been deleted;

[0044] A data deduplication module, which is suitable for executing a data deduplication method as described in any one of Schemes 1 to 3;

[0045] Solution 10, based on solution 9, further includes an object writing module and / or an object reading module and / or an object eviction module;

[0046] The object writing module is suitable for executing an object writing method as described in scheme 5 or scheme 6;

[0047] The object reading module is suitable for executing an object reading method as described in Solution 7 or Solution 8;

[0048] The object eviction module is suitable for executing an object eviction method as described in Solution 9.

[0049] From the above description of the present invention and its preferred embodiments, it can be seen that, compared with the prior art, the technical solution of the present invention and its preferred embodiments have the following beneficial effects due to the adoption of the following technical means:

[0050] 1. The deduplication method proposed in the present invention offloads the hash calculation task to the hardware accelerator inside the DPU (called by the smart network card daemon process), which greatly improves the hash calculation speed and reduces the request response time. Since the host CPU does not participate in the hash calculation, some computing resources are released, and other services on the host can also use these resources, improving CPU utilization.

[0051] The deduplication method proposed in the present invention adopts a CPU utilization-aware block task unloading method. Whenever an object deduplication task arrives, the utilization of the ARM-CPU inside the DPU is checked. If the utilization is too high, the host CPU is used for block division, and then the divided data blocks are sent to the DPU. Otherwise, the object is sent to the DPU as a whole, that is, the block division task and the hash task are unloaded to the DPU together. This can not only reduce the resource consumption of the host CPU, but also avoid the excessive load of the ARM-CPU inside the DPU affecting the network tasks of the DPU itself.

[0052] 2. The deduplication method proposed in the present invention uses a combined path computing task offloading method. DMA is used to send data blocks from the host to the DPU, and then the block results and hash calculation results are sent back from the DPU to the host via RDMA. This is done by taking advantage of the lower latency of DMA data transmission and the higher throughput of RDMA when transmitting small data blocks. This combined path method can maximize the overall performance of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0054] Figure 1 is a system structure diagram of the present invention;

[0055] Figure 2 Reading the flow chart for the object of the present invention;

[0056] Figure 3 Write a flow chart for the object of the present invention;

[0057] Figure 4 De-duplication flow chart for the object of the present invention;

[0058] Figure 5 It is a schematic diagram of combined path calculation unloading of the present invention; DETAILED DESCRIPTION

[0059] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are preferred embodiments of the present invention and should not be regarded as excluding other embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0060] In the claims, description and the above-mentioned drawings of the present invention, unless otherwise clearly defined, the use of terms such as "first", "second" or "third" etc. are for distinguishing different objects rather than for describing a specific order.

[0061] In the claims, specification and the above-mentioned drawings of the present invention, unless otherwise explicitly defined, directional words, such as the terms "center", "lateral", "longitudinal", "horizontal", "vertical", "top", "bottom", "inside", "outside", "up", "down", "front", "back", "left", "right", "clockwise", "counterclockwise", etc., indicating directions or positional relationships are based on the directions and positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific direction or be constructed and operated in a specific direction, and therefore cannot be understood as limiting the specific protection scope of the present invention.

[0062] In the claims, specification and the above drawings of the present invention, unless otherwise clearly defined, if the term "fixed connection" or "fixed connection" is used, it should be understood in a broad sense, that is, any connection method without a displacement relationship and relative rotation relationship between the two, that is to say, including non-detachable fixed connection, detachable fixed connection, integrated connection and fixed connection through other devices or elements.

[0063] In the claims, description and drawings of the present invention, if the terms "include", "have" and their variations are used, they are intended to mean "including but not limited to".

[0064] refer to Figure 1 , a data processing system includes a storage system, a smart network card resource initialization module, an object writing module, an object reading module, a data deduplication module and an object eviction module.

[0065] The storage system is suitable for dividing the data area: The storage system is suitable for dividing the data area into the cache area and the chunk area. Each storage node can be located in either the cache area or the chunk area. The client only perceives the cache area and is unaware of the chunk area.

[0066] Among them, the cache area has multiple first storage nodes (the first storage node is the storage node of the cache area), the object located in the cache area is a cache object, the first storage node can store cache objects, the cache object includes the dedup tag, cache tag, data part, and hash value list (recipe) of the object. The hash value list (recipe) is empty when the object is first written. When the deduplication operation is performed on this object, the hash value will be written to the list. cached is 0, which means that the data of the object cannot be found in the cache area, and cached is 1, which means that the data of the object can be found in the cache area. dedup is 1, which means that it has been deleted, and dedup is 0, which means that the object written for the first time has not been deleted and is complete. The chunk area stores chunk data and the hash value corresponding to the chunk data.

[0067] The smart network card resource initialization module is suitable for executing the smart network card resource initialization method;

[0068] The specific steps include:

[0069] (1) Use the DOCA library to open doca_dev. The DOCA library is a C language library provided by Nvidia. Programmers do not need to directly manage DPU hardware resources, but only need to access the software interface provided by the DOCA library.

[0070] (2) Allocate object data memory space and hash value memory space;

[0071] (3) Bind the object data memory space and hash value memory space to doca_mmap;

[0072] (4) Get multiple source doca_buf and target doca_buf from doca_mmap;

[0073] (5) Bind the source doca_buf and the target doca_buf to doca_sha_task_hash.

[0074] The object write module is adapted to implement an object write method:

[0075] refer to Figure 3 , specifically including the following steps:

[0076] (1) The client calculates the hash value of the object. Based on the hash value of the object, the client uses the CRUSH algorithm to calculate the object to select the first storage node to be written. That is, the client uses the CRUSH algorithm to calculate which first storage node (the node is limited to the cache area) the object should be written to. The CRUSH algorithm is a data distribution algorithm used by Ceph, which is a popular open source distributed storage system. The hash value of the object is used as the input value of the algorithm, and the output value is the ID of a first storage node. Then, the object write request is submitted to the first storage node, which is called the master node of this write request.

[0077] (2) When the first storage node in the cache area receives an object write request, it adopts a replication redundancy strategy based on the distributed system. The first storage node uses the RADOS library to distribute copies of the object to multiple redundant storage nodes in the cache area.

[0078] (3) Set the object's cached flag to 1 and its dedup flag to 0.

[0079] The object read module is adapted to execute an object read method.

[0080] refer to Figure 2 , specifically including the following steps:

[0081] (1) The client submits a data read request to a storage node in the cache area. When the first storage node in the cache area receives the object read request, it checks the dedup flag of the object. If it is 0, indicating that the object has not been deleted, the first storage node directly returns the object data to the client. Otherwise, execute (2);

[0082] (2) Check the cached flag of the object. If it is 1, it indicates that the object has been cached in the first storage node, and the first storage node directly returns the object data to the client. Otherwise, execute (3).

[0083] It can be considered that when the dedup flag of the object is 0 or 1, and the cached flag is 1, the first storage node can directly return the data of the object to the client;

[0084] (3) The first storage node traverses the recipe of the object, sends a chunk object read request to the second storage node corresponding to the chunk area, caches all chunk objects of the object to the first storage node, and then the first storage node returns the object data to the client and sets the cache mark of the object to 1, indicating that the object has been cached to the first storage node.

[0085] The data synchronization operation between the first storage node and the other second storage nodes is completed by the RADOS library, which here refers to librados.

[0086] The data deduplication module is suitable for executing a data deduplication method;

[0087] refer to Figure 4 , Figure 5 , specifically including the following steps:

[0088] (1) The client uses the CRUSH algorithm to find the first storage node where the object is stored. The client then submits an object deduplication request to the first storage node in the cache area;

[0089] (2) When the first storage node of the cache area obtains the object deduplication request, the first storage node checks the utilization rate of the ARM-CPU of the DPU. If it is higher than or equal to the threshold, step (3) is executed; otherwise, step (4) is executed. The threshold can be set by itself. In this embodiment, 20% is selected as the threshold.

[0090] (3) The host CPU uses the CDC algorithm to block the object in the first storage node and executes step (5); if the utilization rate is too high, it means that it is not suitable to offload the data block task now, otherwise it may affect other network tasks being executed by the ARM-CPU, such as TCP protocol parsing tasks.

[0091] A variety of different CDC block algorithms can be used, which can be configured by the user. Including RabinCDC, GearCDC, AE, FastCDC, RapidCDC, JumpCDC, TTTD. Compared with Fixed Size Chunking (FSC), CDC has a higher deduplication rate. The average block length can be configured by the user. The common average block length is 4KB to 64KB. The smaller the average block length, the higher the data deduplication rate can be achieved by the system, but it will consume more CPU resources. When the average block length is larger, the block speed is faster, but the data deduplication rate is lower.

[0092] (4) Submit the object to the smart network card daemon through DMA and execute step (6);

[0093] (5) Submit all data blocks to the smart network card daemon through DMA and execute step (7); the DMA library here can be the doca DMA library, which encapsulates the DMA hardware of the DPU.

[0094] (6) The smart NIC daemon uses the ARM-CPU to divide the object into blocks and executes step (7);

[0095] (7) The smart NIC daemon uses the DOCA library to calculate the hash value of each data block and executes step (8); a variety of different hash algorithms can be used, and users can configure them. Such as SHA1, SHA256, SHA512. The DOCA library includes a variety of hardware acceleration libraries, including but not limited to hash acceleration, DMA, and RDMA.

[0096] (8) The smart NIC daemon uses RDMA to return the hash value to the first storage node and executes step (9); the RDMA library used can be the common libvert library or the doca rdma library. If the previous step has passed step (6), the block result also needs to be sent back. For example, an object is divided into 1,000 data blocks. We need to send back the starting offset position of each data block, and this offset is the block result; for example, an object is 64MB and the average block size is 64KB, then there are 1K data blocks. If each data block uses the SHA1 hash algorithm, the DPU needs to send 20KB of data to the host. For the block result, if a 4-byte integer is used to save the offset of each data block, 4KB of data needs to be sent to the host. In general, if both the block task and the hash calculation task are offloaded to the DPU, 24KB of data needs to be sent back.

[0097] (9) The first storage node combines the data portion of each data block and the hash value corresponding to the data block into a chunk object, and sends the chunk object to the second storage node in the chunk area, and deletes the data portion of the object in the cache area (retaining the dedup tag, cache tag, and hash value list of the object) to save space in the cache area; and executes step (10);

[0098] (10) After receiving the chunk object, the second storage node searches the local hash value table. If the hash value table exists, it indicates that the chunk object is a duplicate and the chunk object is not stored. If the hash value table does not exist, it indicates that the chunk object is not a duplicate and the chunk object is stored. Then, execute step (11).

[0099] (11) Set the dedup flag of the object in the cache area to 1 and the cache flag to 0.

[0100] The object eviction module is adapted to perform an object eviction method.

[0101] The specific steps include:

[0102] (1) The object eviction daemon periodically scans all objects in the cache area. If the dedup flag of the object is 0, the object is skipped, otherwise step (2) is executed;

[0103] (2) Check the cached flag of the object. If it is 0, skip the object. Otherwise, execute step (3).

[0104] (3) Determine whether the object is evicted based on the cache replacement algorithm. If evicted, set the cached flag of the object to 0 and delete the data cache of the object. The cache replacement strategy can use multiple algorithms such as LRU and LFU.

[0105] Compared with the prior art, this embodiment has the following beneficial effects:

[0106] 1. The deduplication method proposed in the present invention offloads the hash calculation task to the hardware accelerator inside the DPU, which greatly improves the hash calculation speed and reduces the request response time. Since the host CPU does not participate in the hash calculation, some computing resources are released, and other services on the host can also use these resources, thereby improving CPU utilization.

[0107] The deduplication method proposed in the present invention adopts a CPU utilization-aware block task unloading method. Whenever an object deduplication task arrives, the utilization of the ARM-CPU inside the DPU is checked. If the utilization is too high, the host CPU is used for block division, and then the divided data blocks are sent to the DPU. Otherwise, the object is sent to the DPU as a whole, that is, the block division task and the hash task are unloaded to the DPU together. This can not only reduce the resource consumption of the host CPU, but also avoid the excessive load of the ARM-CPU inside the DPU affecting the network tasks of the DPU itself.

[0108] 2. The deduplication method proposed in the present invention uses a combined path computing task offloading method. DMA is used to send data blocks from the host to the DPU, and then the block division and hash calculation results are sent back from the DPU to the host via RDMA. This is done by taking advantage of the lower latency of DMA data transmission and the higher throughput of RDMA when transmitting small data blocks. This combined path method can maximize the overall performance of the system.

[0109] The data processing system using the above method consumes only a small amount of CPU and memory resources when processing data, greatly reducing disk overhead.

[0110] The description of the above specification and embodiments is used to explain the protection scope of the present invention, but does not constitute a limitation on the protection scope of the present invention. Through the enlightenment of the present invention or the above embodiments, ordinary technicians in this field can obtain modifications, equivalent substitutions or other improvements to the embodiments of the present invention or part of the technical features thereof through logical analysis, reasoning or limited experiments, which should be included in the protection scope of the present invention.

Claims

1. A data deduplication method, characterized in that: The data area of ​​the storage system is divided into a cache area and a chunk area. The cache area stores the object's dedup tag, cache tag, data part, and hash value list. The chunk area stores chunk data and the hash value corresponding to the chunk data. A cached value of 0 means that the object's data cannot be found in the cache area, and a dedup value of 1 means that it has been deleted. Based on the storage system, the data deduplication method includes the following steps: When the first storage node in the cache area obtains the object deduplication request, it checks the utilization rate of the ARM-CPU of the DPU. If it is higher than or equal to the threshold, the host CPU divides the object into blocks and submits all data blocks to the smart network card daemon. Otherwise, the object is submitted to the smart network card daemon, and the smart network card daemon divides the object into blocks. The smart network card daemon process calculates a hash value of each data block, and the smart network card daemon process returns the hash value to the first storage node; The first storage node combines the data portion of each data block and the hash value corresponding to the data block into a chunk object, sends the chunk object to the second storage node in the chunk area, and deletes the data portion of the object in the cache area; The second storage node searches the local hash value table, and if the hash value exists, does not store it; if the hash value does not exist, stores it; and sets the dedup mark of the object in the cache area to 1, and sets the cache mark to 0.

2. A data deduplication method as claimed in claim 1, characterized in that: When the first storage node in the cache area obtains the object deduplication request, it checks the utilization rate of the ARM-CPU of the DPU. If it is higher than or equal to the threshold, the host CPU divides the object into blocks and submits all data blocks to the smart network card daemon through DMA; Otherwise, the object is submitted to the smart network card daemon process through DMA, and the smart network card daemon process divides the object into blocks; The smart network card daemon calculates a hash value for each data block and returns the hash value to the first storage node using RDMA.

3. A data deduplication method as claimed in claim 2, characterized in that: in, The host CPU divides the object into blocks by using a CDC algorithm, and the smart network card daemon process divides the object into blocks by using an ARM-CPU.

4. An object writing method, characterized in that: The data area of ​​the storage system is divided into a cache area and a chunk area. The cache area stores the object's dedup tag, cache tag, data part, and hash value list. The chunk area stores chunk data and the hash value corresponding to the chunk data. A cached value of 0 means that the object's data cannot be found in the cache area, and a dedup value of 1 means that it has been deleted. Based on the storage system, the object writing method includes the following steps: When the first storage node in the cache area obtains an object write request; The first storage node distributes the copy of the object to multiple redundant storage nodes in the cache area; The cached flag of the object is set to 1 and the dedup flag is set to 0.

5. An object writing method as claimed in claim 4, characterized in that: The client calculates a hash value of the object, and based on the hash value of the object, the client uses a CRUSH algorithm to calculate the object to select a first storage node to write.

6. An object reading method, characterized in that: The data area of ​​the storage system is divided into a cache area and a chunk area. The cache area stores the object's dedup tag, cache tag, data part, and hash value list. The chunk area stores chunk data and the hash value corresponding to the chunk data. A cached value of 0 means that the object's data cannot be found in the cache area, and a dedup value of 1 means that it has been deleted. Based on the storage system, the object reading method includes the following steps: When the first storage node in the cache area receives an object read request, When the dedup mark of the object is 0 or 1, and the cached mark is 1, the first storage node returns the data of the object; Otherwise, the first storage node traverses the recipe of the object, sends a chunk object read request to the second storage node of the chunk area, caches all chunk objects of the object to the first storage node, and the first storage node returns the data of the object and sets the cache mark of the object to 1.

7. An object reading method as claimed in claim 6, characterized in that: When the first storage node in the cache area receives an object read request, Check the dedup flag of the object, if it is 0, the first storage node returns the data of the object; otherwise, check the cached flag of the object, if it is 1, the first storage node returns the data of the object; Otherwise, the first storage node traverses the recipe of the object, sends a chunk object read request to the second storage node of the chunk area, caches all chunk objects of the object to the first storage node, and the first storage node returns the data of the object and sets the cache mark of the object to 1.

8. An object eviction method, characterized in that: The data area of ​​the storage system is divided into a cache area and a chunk area. The cache area stores the object's dedup tag, cache tag, data part, and hash value list. The chunk area stores chunk data and the hash value corresponding to the chunk data. A cached value of 0 means that the object's data cannot be found in the cache area, and a dedup value of 1 means that it has been deleted. Based on the storage system, the object eviction method comprises the following steps: Scan all objects in the cache area regularly. If the dedup flag of the object is 0, skip the object. Otherwise, the cached tag of the object is checked. If it is 0, the object is skipped. Otherwise, it is determined whether the object is evicted according to the cache replacement algorithm. If it is evicted, the cached tag of the object is set to 0, and then the data cache of the object is deleted.

9. A data processing system, characterized in that: include A storage system, which is suitable for dividing a data area into a cache area and a chunk area; wherein the cache area stores a dedup tag, a cache tag, a data part, and a hash value of an object; and the chunk area stores chunk data and a hash value corresponding to the chunk data; cached being 0 means that the data of the object cannot be found in the cache area, and dedup being 1 means that the data has been deleted; A data deduplication module, which is suitable for executing a data deduplication method as described in any one of claims 1-3.

10. A data processing system according to claim 9, characterized in that: Also includes an object writing module and / or an object reading module and / or an object eviction module; The object writing module is suitable for executing an object writing method as claimed in claim 5 or 6; The object reading module is suitable for executing an object reading method as claimed in claim 7 or 8; The object eviction module is adapted to execute an object eviction method as claimed in claim 9.

Images