Data deduplication method and device, medium and electronic equipment

Abstract

The invention provides a data deduplication method and device, a medium and electronic equipment, and relates to the technical field of computers. According to the data deduplication method, a first type of processing resources determines a target data fingerprint corresponding to a to-be-processed data block based on a preset fingerprint algorithm; the first type processing resource generates a fingerprint query request corresponding to the to-be-processed data block according to the target data fingerprint and preset query result callback information; the first type processing resource obtains initial metadata corresponding to the to-be-processed data block according to the to-be-processed data block; according to a fingerprint query result and initial metadata generated based on the fingerprint query request by the second type of processing resources, executing a non-physical storage request and a physical storage operation on the to-be-processed data block by the first type of processing resources, and executing a non-physical storage operation on the to-be-processed data block by the second type of processing resources; the non-physical storage operation comprises data comparison; the physical storage operation comprises write-in processing, deletion processing and metadata processing.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data deduplication method, apparatus, medium and electronic device. Background Technology

[0002] Data deduplication and compression are important means to improve the space utilization of storage systems and the lifespan of storage media. With the widespread use of high-performance storage media—NVME flash memory—in high-performance storage systems, the number of write cycles of flash memory media has become a significant factor limiting the lifespan of storage systems. In addition to scheduling algorithms that balance write data, searching for duplicate data from the written data can also directly improve the space utilization and lifespan of storage systems.

[0003] The advantage of online deduplication is that it ensures duplicate data is not written to the disk medium, eliminating wear and tear on the disk medium from the source. This is the main advantage of online deduplication compared to offline deduplication. However, the disadvantage of online deduplication is also obvious: it increases the latency of write I / O, which can have a significant impact on latency-sensitive applications. Compared to online deduplication, offline deduplication is more flexible. It can calculate the data fingerprint in the user I / O context or have the data fingerprint calculated by the background service before performing the deduplication process. It can control the number of tasks and resource consumption of the deduplication process according to the user I / O resource requirements, minimizing interference with user I / O. However, as mentioned earlier, offline deduplication requires that duplicate data has already been written to the storage medium and wear and tear on the storage medium has already occurred.

[0004] Existing data deduplication methods are time-consuming to implement, resulting in low efficiency. Therefore, providing a data deduplication method that solves this efficiency problem is of significant practical importance. Summary of the Invention

[0005] This application provides a data deduplication method, apparatus, medium, and electronic device that can improve the efficiency of data deduplication.

[0006] To achieve the above objectives, the technical solution of this application embodiment is implemented as follows: In a first aspect, embodiments of this application provide a data deduplication method applied to a data deduplication system; the data deduplication system includes multiple nodes; each node includes a first type of processing resource and a second type of processing resource; the first type of processing resource is used for data I / O processing; the second type of processing resource is used for deduplicated data processing; the method includes: The first type of processing resource determines the target data fingerprint corresponding to the data block to be processed based on a preset fingerprint algorithm; The first type of processing resource generates a fingerprint query request corresponding to the data block to be processed based on the target data fingerprint and preset query result callback information; the fingerprint query request is used to enable the second type of processing resource to perform a fingerprint query on the data block to be processed. The first type of processing resource obtains the initial metadata corresponding to the data block to be processed based on the data block to be processed; Based on the fingerprint query result generated by the second type of processing resource based on the fingerprint query request and the initial metadata, the first type of processing resource performs non-physical storage requests and physical storage operations on the data block to be processed, and the second type of processing resource performs non-physical storage operations on the data block to be processed; the non-physical storage operations include data comparison; the physical storage operations include write processing, deletion processing, and metadata processing.

[0007] The data deduplication method provided in this application is applied to a data deduplication system during the data deduplication process. The data deduplication system includes multiple nodes; each node includes a first type of processing resource and a second type of processing resource. The first type of processing resource is used for data I / O processing; the second type of processing resource is used for deduplicated data processing. The method includes: the first type of processing resource determining a target data fingerprint corresponding to the data block to be processed based on a preset fingerprint algorithm; the first type of processing resource generating a fingerprint query request corresponding to the data block to be processed based on the target data fingerprint and preset query result callback information; the fingerprint query request is used to enable the second type of processing resource to process the data block. The data deduplication mechanism involves: fingerprint query of a data block to be processed; obtaining initial metadata corresponding to the data block based on the data block to be processed by the first type of processing resource; and performing non-physical storage requests and physical storage operations on the data block to be processed by the first type of processing resource based on the fingerprint query request and the initial metadata, according to the fingerprint query result generated by the second type of processing resource based on the fingerprint query request and the initial metadata. The non-physical storage operations include data comparison, and the physical storage operations include write processing, deletion processing, and metadata processing. This provides a parallel data deduplication mechanism that can reduce the time consumption of data deduplication and effectively improve the efficiency of data deduplication.

[0008] In one optional embodiment, obtaining the initial metadata corresponding to the data block to be processed, based on the data block to be processed, includes: If the metadata corresponding to the data block to be processed is found, the found metadata will be used as the initial metadata corresponding to the data block to be processed. If the metadata corresponding to the data block to be processed is not found, metadata is generated for the data block to be processed, and the generated metadata is used as the initial metadata corresponding to the data block to be processed.

[0009] The method in this embodiment, by using the metadata corresponding to the data block to be processed as the initial metadata corresponding to the data block to be processed if the metadata corresponding to the data block to be processed is found, and by generating metadata for the data block to be processed if the metadata corresponding to the data block to be processed is not found, and using the generated metadata as the initial metadata corresponding to the data block to be processed, can efficiently configure the initial metadata corresponding to the data block to be processed during data deduplication, and can more effectively improve the efficiency of data deduplication.

[0010] In an optional embodiment, the first type of processing resource performs non-physical storage requests and physical storage operations on the data block to be processed based on the fingerprint query result generated by the second type of processing resource based on the fingerprint query request and the initial metadata, and the second type of processing resource performs non-physical storage operations on the data block to be processed, including: If the fingerprint query result generated by the second type of processing resource based on the fingerprint query request indicates that the fingerprint is duplicated, then the first type of processing resource makes a non-physical storage request for the data block to be processed and modifies the initial metadata to obtain the target metadata, and the second type of processing resource performs a non-physical storage operation on the data block to be processed. If the fingerprint query result indicates that the fingerprint is not repeated, the first type of processing resource will use the initial metadata as the target metadata of the data block to be processed and perform physical storage operations.

[0011] In this embodiment, if the fingerprint query result generated by the second type of processing resource based on the fingerprint query request indicates that the fingerprint is duplicated, then the first type of processing resource makes a non-physical storage request for the data block to be processed and modifies the initial metadata to obtain target metadata, while the second type of processing resource performs a non-physical storage operation on the data block to be processed; if the fingerprint query result indicates that the fingerprint is not duplicated, the first type of processing resource uses the initial metadata as the target metadata of the data block to be processed and performs a physical storage operation. By not performing a physical storage operation when the fingerprint is duplicated and performing a physical storage operation when the fingerprint is not duplicated, the data deduplication process of the parallel structure can be efficiently realized, and the efficiency of data deduplication can be improved more effectively.

[0012] In an optional embodiment, if the fingerprint query result generated by the second type of processing resource based on the fingerprint query request indicates fingerprint duplication, then the first type of processing resource performs a non-physical storage request on the data block to be processed and modifies the initial metadata to obtain target metadata; the second type of processing resource performs a non-physical storage operation on the data block to be processed, including: If the fingerprint query result generated by the second type of processing resource based on the fingerprint query request indicates that the fingerprint is duplicated, then the first type of processing resource makes a non-physical storage request for the data block to be processed. The second type of processing resource responds to the non-physical storage request and performs non-physical storage operations on the data block to be processed, while the first type of processing resource modifies the initial metadata to obtain the target metadata.

[0013] In this embodiment, if the fingerprint query result generated by the second type of processing resource based on the fingerprint query request indicates fingerprint duplication, then the first type of processing resource makes a non-physical storage request for the data block to be processed; the second type of processing resource responds to the non-physical storage request and performs a non-physical storage operation on the data block to be processed; the first type of processing resource modifies the initial metadata to obtain the target metadata, which can realize data deduplication in a parallel structure where the first type of processing resource and the second type of processing resource cooperate with each other, further effectively improving the efficiency of data deduplication.

[0014] In an optional embodiment, the non-physical storage operation further includes fingerprint locking; the non-physical storage request includes a data comparison request and a fingerprint locking request; the second type of processing resource responds to the non-physical storage request and performs a non-physical storage operation on the data block to be processed, and the first type of processing resource modifies the initial metadata to obtain the target metadata, including: The second type of processing resource responds to the data comparison request sent by the first type of processing resource and performs data comparison on the data block to be processed; The second type of processing resource returns the data comparison results to the first type of processing resource; If the first type of processing resource determines, based on the received data comparison result, that the data block to be processed has the same stored data, then it modifies the initial metadata to obtain the target metadata and generates the fingerprint locking request; The second type of processing resource responds to the fingerprint locking request sent by the first type of processing resource and performs fingerprint locking on the data block to be processed.

[0015] The method of this embodiment further includes fingerprint locking through the non-physical storage operation; the non-physical storage request includes a data comparison request and a fingerprint locking request; the second type of processing resource responds to the non-physical storage request and performs a non-physical storage operation on the data block to be processed, and the first type of processing resource modifies the initial metadata to obtain target metadata, including: the second type of processing resource responds to the data comparison request sent by the first type of processing resource and performs data comparison on the data block to be processed; the second type of processing resource returns the data comparison result to the first type of processing resource; if the first type of processing resource determines that the data block to be processed has the same stored data based on the received data comparison result, it modifies the initial metadata to obtain target metadata and generates the fingerprint locking request; the second type of processing resource responds to the fingerprint locking request sent by the first type of processing resource and performs fingerprint locking on the data block to be processed. Fingerprint locking can provide a fingerprint protection mechanism for the data block to be processed, improve the accuracy of the data deduplication process, and improve the efficiency of data deduplication.

[0016] In an optional embodiment, the first type of processing resource performs non-physical storage requests and physical storage operations on the data block to be processed based on the fingerprint query result generated by the second type of processing resource based on the fingerprint query request and the initial metadata, and the second type of processing resource performs non-physical storage operations on the data block to be processed, including: If the fingerprint query result generated by the second type of processing resource based on the fingerprint query request indicates that the fingerprint is duplicated, then the data block to be processed is written into the power-saving memory. The first type of processing resource makes a non-physical storage request for the data block to be processed; The second type of processing resource responds to the non-physical storage request and performs non-physical storage operations on the data block to be processed; If the result of a non-physical storage operation indicates that the data block to be processed contains the same stored data, the first type of processing resource modifies the initial metadata to obtain the target metadata and deletes the data in the power-saving memory; If the result of the non-physical storage operation indicates that the data block to be processed does not have the same stored data, the first type of processing resource will use the initial metadata as the target metadata of the data block to be processed and perform physical storage operation.

[0017] The method of this embodiment, wherein the first type of processing resource performs non-physical storage request and physical storage operation on the data block to be processed based on the fingerprint query result generated by the second type of processing resource based on the fingerprint query request and the initial metadata, and the second type of processing resource performs non-physical storage operation on the data block to be processed, includes: if the fingerprint query result generated by the second type of processing resource based on the fingerprint query request indicates fingerprint duplication, then the data block to be processed is written to power-saving memory; the first type of processing resource makes a non-physical storage request on the data block to be processed; the second type of processing resource responds to the non-physical storage request and performs non-physical storage operation on the data block to be processed; if the result of the non-physical storage operation indicates that the data block to be processed has the same stored data, the first type of processing resource modifies the initial metadata to obtain target metadata and deletes the data in the power-saving memory; if the result of the non-physical storage operation indicates that the data block to be processed does not have the same stored data, the first type of processing resource uses the initial metadata as the target metadata of the data block to be processed and performs physical storage operation. By storing the data block to be processed in power-saving memory, a better data comparison process can be achieved, the wear and tear of the storage medium can be reduced, and the efficiency of data deduplication can be further improved.

[0018] In an optional embodiment, the first type of processing resource performs non-physical storage requests and physical storage operations on the data block to be processed based on the fingerprint query result generated by the second type of processing resource based on the fingerprint query request and the initial metadata, and the second type of processing resource performs non-physical storage operations on the data block to be processed, including: If the fingerprint query result generated by the second type of processing resource based on the fingerprint query request indicates that the fingerprint is duplicated, then the first type of processing resource makes a non-physical storage request for the data block to be processed, and the first type of processing resource performs a physical storage operation on the data block to be processed. The second type of processing resources performs non-physical storage operations on the data block to be processed based on the initial metadata; the non-physical storage operations also include offline deduplication.

[0019] In this embodiment, if the fingerprint query result generated by the second type of processing resource based on the fingerprint query request indicates fingerprint duplication, then the first type of processing resource makes a non-physical storage request for the data block to be processed, and performs physical storage operation on the data block to be processed; the second type of processing resource performs non-physical storage operation on the data block to be processed according to the initial metadata; the non-physical storage operation also includes offline deduplication, whereby after the first type of processing resource detects the same fingerprint, it sends a data comparison request, performs physical storage operation on the data block to be processed, and performs offline data comparison and deduplication through the second type of processing resource, thereby providing a parallel structured offline data deduplication mechanism, which can reduce the time consumption of data deduplication and effectively improve the efficiency of data deduplication.

[0020] In an optional embodiment, the first type of processing resource performs non-physical storage requests and physical storage operations on the data block to be processed based on the fingerprint query result generated by the second type of processing resource based on the fingerprint query request and the initial metadata, and the second type of processing resource performs non-physical storage operations on the data block to be processed, including: The first type of processing resources performs physical storage operations on the data block to be processed; If the fingerprint query result generated by the second type of processing resource based on the fingerprint query request indicates that the fingerprint is duplicated, then the first type of processing resource makes a non-physical storage request for the data block to be processed. The second type of processing resource responds to the non-physical storage request and performs non-physical storage operations on the data block to be processed; the non-physical storage operations also include offline deduplication.

[0021] In this embodiment, the first type of processing resource performs physical storage operations on the data block to be processed. If the fingerprint query result generated by the second type of processing resource based on the fingerprint query request indicates fingerprint duplication, the first type of processing resource makes a non-physical storage request for the data block to be processed. The second type of processing resource responds to the non-physical storage request and performs non-physical storage operations on the data block to be processed. The non-physical storage operation also includes offline deduplication. After completing normal user IO processing, the first type of processing resource checks for identical fingerprints, sends a data comparison request, and forwards it to the duplicate data processing service for offline data comparison and offline deduplication. This method provides another parallel structure for offline data deduplication, which can reduce the time consumption of data deduplication, reduce the latency of data processing, and effectively improve the efficiency of data deduplication.

[0022] Secondly, embodiments of this application also provide a data deduplication device applied to a data deduplication system; the data deduplication system includes multiple nodes; each node includes a first type of processing resource and a second type of processing resource; the first type of processing resource is used for data I / O processing; the second type of processing resource is used for deduplicated data processing; the device includes:

[0023] The fingerprint calculation and management module is used to determine the target data fingerprint corresponding to the data block to be processed based on a preset fingerprint algorithm for the first type of processing resources; The fingerprint query control module is used to generate a fingerprint query request corresponding to the data block to be processed based on the target data fingerprint and preset query result callback information of the first type of processing resources; the fingerprint query request is used to enable the second type of processing resources to perform fingerprint query on the data block to be processed. An initial metadata calculation module is used by the first type of processing resource to obtain the initial metadata corresponding to the data block to be processed based on the data block to be processed. The parallel deduplication module is used to perform non-physical storage requests and physical storage operations on the data block to be processed by the first type of processing resources based on the fingerprint query result generated by the fingerprint query request and the initial metadata, and the second type of processing resources perform non-physical storage operations on the data block to be processed; the non-physical storage operations include data comparison; the physical storage operations include write processing, deletion processing, and metadata processing.

[0024] In an optional embodiment, the initial meta-calculation module is specifically used for:

[0025] If the metadata corresponding to the data block to be processed is found, the found metadata will be used as the initial metadata corresponding to the data block to be processed. If the metadata corresponding to the data block to be processed is not found, metadata is generated for the data block to be processed, and the generated metadata is used as the initial metadata corresponding to the data block to be processed.

[0026] In one optional embodiment, the parallel deduplication module is specifically used for: If the fingerprint query result generated by the second type of processing resource based on the fingerprint query request indicates that the fingerprint is duplicated, then the first type of processing resource makes a non-physical storage request for the data block to be processed and modifies the initial metadata to obtain the target metadata, and the second type of processing resource performs a non-physical storage operation on the data block to be processed. If the fingerprint query result indicates that the fingerprint is not repeated, the first type of processing resource will use the initial metadata as the target metadata of the data block to be processed and perform physical storage operations.

[0027] In one optional embodiment, the parallel deduplication module is specifically used for: If the fingerprint query result generated by the second type of processing resource based on the fingerprint query request indicates that the fingerprint is duplicated, then the first type of processing resource makes a non-physical storage request for the data block to be processed. The second type of processing resource responds to the non-physical storage request and performs non-physical storage operations on the data block to be processed, while the first type of processing resource modifies the initial metadata to obtain the target metadata.

[0028] In an optional embodiment, the non-physical storage operation further includes fingerprint locking; the non-physical storage request includes a data comparison request and a fingerprint locking request; the parallel deduplication module is specifically used for: The second type of processing resource responds to the data comparison request sent by the first type of processing resource and performs data comparison on the data block to be processed; The second type of processing resource returns the data comparison results to the first type of processing resource; If the first type of processing resource determines, based on the received data comparison result, that the data block to be processed has the same stored data, then it modifies the initial metadata to obtain the target metadata and generates the fingerprint locking request; The second type of processing resource responds to the fingerprint locking request sent by the first type of processing resource and performs fingerprint locking on the data block to be processed.

[0029] In one optional embodiment, the parallel deduplication module is specifically used for: If the fingerprint query result generated by the second type of processing resource based on the fingerprint query request indicates that the fingerprint is duplicated, then the data block to be processed is written into the power-saving memory. The first type of processing resource makes a non-physical storage request for the data block to be processed; The second type of processing resource responds to the non-physical storage request and performs non-physical storage operations on the data block to be processed; If the result of a non-physical storage operation indicates that the data block to be processed contains the same stored data, the first type of processing resource modifies the initial metadata to obtain the target metadata and deletes the data in the power-saving memory; If the result of the non-physical storage operation indicates that the data block to be processed does not have the same stored data, the first type of processing resource will use the initial metadata as the target metadata of the data block to be processed and perform physical storage operation.

[0030] In one optional embodiment, the parallel deduplication module is specifically used for: If the fingerprint query result generated by the second type of processing resource based on the fingerprint query request indicates that the fingerprint is duplicated, then the first type of processing resource makes a non-physical storage request for the data block to be processed, and the first type of processing resource performs a physical storage operation on the data block to be processed. The second type of processing resources performs non-physical storage operations on the data block to be processed based on the initial metadata; the non-physical storage operations also include offline deduplication.

[0031] In one optional embodiment, the parallel deduplication module is specifically used for: The first type of processing resources performs physical storage operations on the data block to be processed; If the fingerprint query result generated by the second type of processing resource based on the fingerprint query request indicates that the fingerprint is duplicated, then the first type of processing resource makes a non-physical storage request for the data block to be processed. The second type of processing resource responds to the non-physical storage request and performs non-physical storage operations on the data block to be processed; the non-physical storage operations also include offline deduplication.

[0032] Thirdly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the data deduplication method of the first aspect.

[0033] Fourthly, embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and when the computer program is executed by the processor, the processor implements the data deduplication method of the first aspect.

[0034] The technical effects of any of the implementation methods in the second to fourth aspects can be found in the technical effects of the corresponding implementation methods in the first aspect, and will not be repeated here. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 A structural block diagram of a data deduplication system provided in this application embodiment; Figure 2 A flowchart illustrating a data deduplication method provided in an embodiment of this application; Figure 3 This is a flowchart illustrating the process of obtaining the initial metadata corresponding to the data block to be processed in a data deduplication method provided in this application embodiment. Figure 4 A schematic diagram illustrating the parallel deduplication process of a data deduplication method provided in this application embodiment; Figure 5 This is a schematic diagram illustrating the process of parallel deduplication when fingerprints are duplicated, as provided in an embodiment of this application. Figure 6 A schematic diagram illustrating the response to a non-physical storage request when the fingerprint of a data deduplication method provided in this application is duplicated; Figure 7 This is a second schematic diagram illustrating the parallel deduplication process of a data deduplication method provided in an embodiment of this application. Figure 8 The third schematic diagram of the parallel deduplication process of a data deduplication method provided in this application embodiment; Figure 9 The fourth schematic diagram of the parallel deduplication process of a data deduplication method provided in this application embodiment; Figure 10 This is a schematic diagram of the structure of a data deduplication device provided in an embodiment of this application; Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0038] It should be noted that the terms "comprising" and "having" and their variations used in this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such process, method, product, or device.

[0039] The following are explanations of some of the words that appear in the text: (1) Fingerprint: A fingerprint is a summary of user data that can determine the data with a high probability of being equal. In some embodiments of this application, the fingerprint is calculated using algorithms such as MD5 or SHA-1.

[0040] (2) Metadata: Metadata is system data used to directly or indirectly describe user data.

[0041] (3) Fingerprint database: The fingerprint database is a functional module and service used to store user data fingerprints in the system.

[0042] (4) Power-saving memory: Power-saving memory is a physical component that maintains the power supply to the memory in the event of a power failure by means of an independent battery or supercapacitor.

[0043] (5) KV database: KV database is a functional component that stores data in the form of key and value tuples. Common KV databases include Rocks DB.

[0044] Data deduplication and compression are important means to improve the space utilization of storage systems and the lifespan of storage media. With the widespread use of high-performance storage media—NVME flash memory—in high-performance storage systems, the number of write cycles of flash memory media has become a significant factor limiting the lifespan of storage systems. In addition to scheduling algorithms that balance write data, searching for duplicate data from the written data can also directly improve the space utilization and lifespan of storage systems.

[0045] The advantage of online deduplication is that it ensures duplicate data is not written to the disk medium, eliminating wear and tear on the disk medium from the source. This is the main advantage of online deduplication compared to offline deduplication. However, the disadvantage of online deduplication is also obvious: it increases the latency of write I / O, which can have a significant impact on latency-sensitive applications. Compared to online deduplication, offline deduplication is more flexible. It can calculate the data fingerprint in the user I / O context or have the data fingerprint calculated by the background service before performing the deduplication process. It can control the number of tasks and resource consumption of the deduplication process according to the user I / O resource requirements, minimizing interference with user I / O. However, as mentioned earlier, offline deduplication requires that duplicate data has already been written to the storage medium and wear and tear on the storage medium has already occurred.

[0046] Existing data deduplication methods are time-consuming to implement, resulting in low efficiency. Therefore, providing a data deduplication method that solves this efficiency problem is of significant practical importance.

[0047] To address existing technical problems, this application provides a data deduplication method, apparatus, medium, and electronic device, applied in a data deduplication system during data deduplication. The data deduplication system includes multiple nodes; each node includes a first type of processing resource and a second type of processing resource; the first type of processing resource is used for data I / O processing; the second type of processing resource is used for deduplicated data processing; the method includes: the first type of processing resource determining a target data fingerprint corresponding to a data block to be processed based on a preset fingerprint algorithm; the first type of processing resource generating a fingerprint query request corresponding to the data block to be processed based on the target data fingerprint and preset query result callback information; the fingerprint query request... This mechanism enables the second type of processing resources to perform fingerprint queries on the data blocks to be processed; the first type of processing resources obtain the initial metadata corresponding to the data blocks to be processed; based on the fingerprint query results and initial metadata generated by the second type of processing resources based on the fingerprint query request, the first type of processing resources perform non-physical storage requests and physical storage operations on the data blocks to be processed, and the second type of processing resources perform non-physical storage operations on the data blocks to be processed; the non-physical storage operations include data comparison; the physical storage operations include write processing, deletion processing, and metadata processing, providing a parallel data deduplication mechanism that can reduce the time consumption of data deduplication and effectively improve the efficiency of data deduplication.

[0048] To make the inventive objectives, technical solutions, and advantages of the embodiments of this application clearer, the present application will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0049] The technical solutions provided in the embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0050] The data deduplication method provided in this application can be applied to a server or a terminal device. The following embodiments of this application illustrate the application of the data deduplication method to a terminal device. This terminal device can be a node in the data deduplication system.

[0051] Figure 1 This paper illustrates a structural block diagram of a data deduplication system according to an embodiment of the present application, which consists of, as shown in the diagram... Figure 1 As shown in the structural diagram, the data deduplication system 10 includes multiple nodes, such as Node_1 and Node_2. In the embodiments of this application, each node of the data deduplication system includes a first type of processing resource and a second type of processing resource. The first type of processing resource is used for data I / O processing; the second type of processing resource is used for deduplicated data processing. For example, the first type of processing resource may be... Figure 1 The data I / O processing service shown is used for data I / O processing; the second type of processing resource can be... Figure 1 The deduplication service shown is used for deduplication.

[0052] In some embodiments of this application, each node of the data deduplication system contains one or more CPUs, and each CPU contains multiple cores; each node may also contain one or more flash drives and may install power-saving memory. Data I / O processing services and deduplication services can be started on each node, wherein the data I / O processing services and deduplication services are bound to different cores.

[0053] In some embodiments of this application, the data I / O processing service and the recurring data processing service can be bound to different CPU cores, and the data I / O processing service and the recurring data processing service can access each other across nodes. For example, the data I / O processing service of Node_1 can access the recurring data processing service of Node_1, and can also access the recurring data processing service of Node_2.

[0054] This application provides a data deduplication method applied to a data deduplication system, which includes multiple nodes; each node includes a first type of processing resource and a second type of processing resource; the first type of processing resource is used for data I / O processing; the second type of processing resource is used for deduplication processing; such as Figure 2 As shown, it includes the following steps: Step S201: The first type of processing resource determines the target data fingerprint corresponding to the data block to be processed based on a preset fingerprint algorithm.

[0055] In practice, the data deduplication system uses the first type of processing resources to determine the target data fingerprint corresponding to the data block to be processed based on a preset fingerprint algorithm.

[0056] In some optional embodiments, specifically the target node of the data deduplication system, the target data fingerprint corresponding to the data block to be processed is determined by a preset fingerprint algorithm using a first type of processing resource. The target node can be any node in the data deduplication system.

[0057] In some alternative embodiments, the data deduplication system may also have only one node.

[0058] In some alternative embodiments, the fingerprint algorithm is a fingerprint extraction algorithm, such as MD5 or SHA-1.

[0059] For example, the data I / O processing service of node_1 in the data deduplication system 10 determines the target data fingerprint F_P_Targ corresponding to the data block Block_i to be processed based on a preset fingerprint algorithm.

[0060] In step S202, the first type of processing resource generates a fingerprint query request corresponding to the data block to be processed based on the target data fingerprint and the preset query result callback information.

[0061] The fingerprint query request is used to enable the second type of processing resources to perform fingerprint queries on the data blocks to be processed.

[0062] In practice, the data deduplication system uses the first type of processing resources to generate a fingerprint query request corresponding to the data block to be processed, based on the target data fingerprint and the preset query result callback information.

[0063] For example, the data IO processing service of node_1 of the data deduplication system 10 generates a fingerprint query request F_P_Requ corresponding to the data block Block_i to be processed, based on the target data fingerprint F_P_Targ and the preset query result callback information.

[0064] In some embodiments of this application, the query result callback information is specifically a callback context structure that stores the query results.

[0065] Step S203: The first type of processing resource obtains the initial metadata corresponding to the data block to be processed based on the data block to be processed.

[0066] In practice, the data deduplication system obtains the initial metadata corresponding to the data block to be processed based on the data block to be processed through the first type of processing resources.

[0067] In some embodiments of this application, the first type of processing resource of the data deduplication system queries or generates metadata information of the data block to be processed, which serves as the initial metadata corresponding to the data block to be processed.

[0068] For example, the data IO processing service of node_1 of the data deduplication system 10 obtains the initial metadata Start_metadata corresponding to the data block Block_i to be processed based on the data block Block_i to be processed.

[0069] In some optional embodiments, the process in step S203 above, where the first type of processing resource obtains the initial metadata corresponding to the data block to be processed based on the data block to be processed, is as follows: Figure 3 As shown, this can be achieved through the following steps: Step S301: If the metadata corresponding to the data block to be processed is found, the found metadata is used as the initial metadata corresponding to the data block to be processed.

[0070] Step S302: If the metadata corresponding to the data block to be processed is not found, then the metadata corresponding to the data block to be processed is generated, and the generated metadata is used as the initial metadata corresponding to the data block to be processed.

[0071] In the method of this embodiment, the initial metadata corresponding to the data block to be processed can be configured efficiently during data deduplication, which can more effectively improve the efficiency of data deduplication.

[0072] Step S204: Based on the fingerprint query result and initial metadata generated by the second type of processing resource based on the fingerprint query request, the first type of processing resource performs non-physical storage request and physical storage operation on the data block to be processed, and the second type of processing resource performs non-physical storage operation on the data block to be processed; the non-physical storage operation includes data comparison.

[0073] Physical storage operations include write processing, delete processing, and metadata processing.

[0074] In some embodiments of this application, the second type of processing resource records the fingerprint query result in the context visible to the first type of processing resource, so that the first type of processing resource can efficiently obtain the fingerprint query result.

[0075] In some embodiments of this application, the deletion process includes offline re-deletion.

[0076] In some embodiments of this application, metadata processing is used to determine the target metadata of the data block to be processed based on the initial metadata.

[0077] In the following embodiments, the first type of processing resource is... Figure 1 The data I / O processing services shown, the second type of processing resource is Figure 1 The illustrated deduplication service is used as an example. This data I / O processing service is used for data I / O processing; this deduplication service is used for deduplication.

[0078] In some optional embodiments, the process in step S204 above, where the first type of processing resource performs non-physical storage requests and physical storage operations on the data block to be processed based on the fingerprint query results and initial metadata generated by the second type of processing resource based on the fingerprint query request, and the second type of processing resource performs non-physical storage operations on the data block to be processed, is as follows: Figure 4 As shown, this can be achieved through the following steps: In step S401, if the fingerprint query result generated by the second type of processing resource based on the fingerprint query request indicates that the fingerprint is duplicated, then the first type of processing resource makes a non-physical storage request for the data block to be processed and modifies the initial metadata to obtain the target metadata, and the second type of processing resource performs a non-physical storage operation on the data block to be processed.

[0079] In specific implementation, if the fingerprint query result generated by the second type of processing resource based on the fingerprint query request indicates that the fingerprint is duplicated, then the first type of processing resource generates non-physical storage request information and sends the non-physical storage request information to the second type of processing resource, and modifies the initial metadata to obtain the target metadata; the second type of processing resource performs non-physical storage operation on the data block to be processed; wherein, the non-physical storage request information is used to enable the second type of processing resource to perform non-physical storage operation on the data block to be processed.

[0080] In some embodiments of this application, when the first type of processing resource sends non-physical storage request information to the second type of processing resource, the context of the current data IO processing service is suspended, and a timeout is set to asynchronously wait for the result information of the non-physical storage operation, such as the data comparison result.

[0081] In step S402, if the fingerprint query result indicates that the fingerprint is not repeated, the first type of processing resource will use the initial metadata as the target metadata of the data block to be processed and perform physical storage operations.

[0082] The method in this embodiment, by not performing physical storage operations when fingerprints are repeated and performing physical storage operations when fingerprints are not repeated, can efficiently realize the data deduplication process in a parallel structure and more effectively improve the efficiency of data deduplication.

[0083] In some optional embodiments, in step S401 above, if the fingerprint query result generated by the second type of processing resource based on the fingerprint query request indicates fingerprint duplication, then the first type of processing resource performs a non-physical storage request on the data block to be processed and modifies the initial metadata to obtain the target metadata, and the second type of processing resource performs a non-physical storage operation on the data block to be processed, as follows: Figure 5 As shown, this can be achieved through the following steps: Step S501: If the fingerprint query result generated by the second type of processing resource based on the fingerprint query request indicates that the fingerprint is duplicated, then the first type of processing resource makes a non-physical storage request for the data block to be processed.

[0084] In step S502, the second type of processing resource responds to the non-physical storage request and performs non-physical storage operations on the data block to be processed, while the first type of processing resource modifies the initial metadata to obtain the target metadata.

[0085] The method in this embodiment can realize data deduplication in a parallel structure where the first type of processing resources and the second type of processing resources cooperate with each other, thereby further improving the efficiency of data deduplication.

[0086] In some optional embodiments, the non-physical storage operation further includes fingerprint locking; the non-physical storage request includes a data comparison request and a fingerprint locking request; in step S502 above, the second type of processing resource responds to the non-physical storage request and performs a non-physical storage operation on the data block to be processed, and the first type of processing resource modifies the initial metadata to obtain the target metadata, as follows: Figure 6 As shown, this can be achieved through the following steps: Step S601: The second type of processing resource responds to the data comparison request sent by the first type of processing resource and performs data comparison on the data block to be processed.

[0087] Step S602: The second type of processing resource returns the data comparison result to the first type of processing resource.

[0088] In step S603, if the first type of processing resource determines that the data block to be processed has the same stored data based on the received data comparison result, it modifies the initial metadata to obtain the target metadata and generates a fingerprint locking request.

[0089] In step S604, the second type of processing resource responds to the fingerprint locking request sent by the first type of processing resource and performs fingerprint locking on the data block to be processed.

[0090] In some embodiments of this application, when the fingerprint query result indicates that the fingerprint is duplicated, the first type of processing resources perform a temporary fingerprint persistence operation.

[0091] In some embodiments of this application, the fingerprint temporary persistence operation is a log that records fingerprint references.

[0092] In some embodiments, the log for recording fingerprint references includes: recording the query record in the log: the queryer and the index information of the retrieved record.

[0093] In some embodiments of this application, fingerprint locking specifically involves persistent fingerprint reference information.

[0094] In some embodiments, persistent fingerprint reference information includes: if the same fingerprint already exists, adding a reference record to the fingerprint to prevent the fingerprint record from being deleted.

[0095] In some embodiments, persisting fingerprint reference information further includes: reclaiming logs previously used to lock the fingerprint record.

[0096] The method in this embodiment can provide a fingerprint protection mechanism for the data block to be processed through fingerprint locking, thereby improving the accuracy of the data deduplication process and increasing the efficiency of data deduplication.

[0097] In some optional embodiments, the process in step S204 above, where the first type of processing resource performs non-physical storage requests and physical storage operations on the data block to be processed based on the fingerprint query results and initial metadata generated by the second type of processing resource based on the fingerprint query request, and the second type of processing resource performs non-physical storage operations on the data block to be processed, is as follows: Figure 7 As shown, this can be achieved through the following steps: Step S701: If the fingerprint query result generated by the second type of processing resource based on the fingerprint query request indicates that the fingerprint is duplicated, then the data block to be processed is written to the power-saving memory.

[0098] For example, after the data IO processing service detects a matching fingerprint, it writes the user data to the power-saving memory.

[0099] Step S702: The first type of processing resource makes a non-physical storage request for the data block to be processed.

[0100] For example, the data IO processing service sends a data comparison request to the duplicate data processing service.

[0101] Step S703: The second type of processing resource responds to the non-physical storage request and performs non-physical storage operations on the data block to be processed.

[0102] For example, after receiving a data comparison request, the duplicate data processing service performs a data comparison and sends the comparison result back to the data IO processing service.

[0103] Step S704: If the result of the non-physical storage operation indicates that the data block to be processed has the same stored data, the first type of processing resource modifies the initial metadata to obtain the target metadata and deletes the data in the power-saving memory.

[0104] For example, if the data IO processing service detects that the data is the same, it modifies the metadata and cleans up the power-saving data.

[0105] Step S705: If the result of the non-physical storage operation indicates that the data block to be processed does not have the same storage data, the first type of processing resource will use the initial metadata as the target metadata of the data block to be processed and perform physical storage operation.

[0106] For example, if the data IO processing service detects that the data is different, it will write the user data normally and then clean up the power-saving data.

[0107] The method in this embodiment, by storing the data blocks to be processed in a power-saving memory, can achieve a better data comparison process, reduce the wear and tear of the storage medium, and further improve the efficiency of data deduplication.

[0108] In some optional embodiments, the process in step S204 above, where the first type of processing resource performs non-physical storage requests and physical storage operations on the data block to be processed based on the fingerprint query results and initial metadata generated by the second type of processing resource based on the fingerprint query request, and the second type of processing resource performs non-physical storage operations on the data block to be processed, is as follows: Figure 8 As shown, this can be achieved through the following steps: Step S801: If the fingerprint query result generated by the second type of processing resource based on the fingerprint query request indicates that the fingerprint is duplicated, then the first type of processing resource makes a non-physical storage request for the data block to be processed and performs a physical storage operation on the data block to be processed.

[0109] For example, after the data IO processing service completes the query / generation of metadata, it checks the completion result of the fingerprint query request. If the fingerprint query result is complete and the fingerprints are the same, it sends a comparison data request. In parallel with the data IO processing service sending the comparison data request, the data IO processing service performs normal user data writing.

[0110] In step S802, the second type of processing resources performs non-physical storage operations on the data blocks to be processed based on the initial metadata; the non-physical storage operations also include offline deduplication.

[0111] For example, after receiving a data comparison request, the deduplication service compares and processes the data, and performs offline deduplication.

[0112] The method in this embodiment sends a data comparison request after the first type of processing resources detect the same fingerprint. The first type of processing resources perform physical storage operations on the data block to be processed, and the second type of processing resources perform offline data comparison and deduplication. This provides a parallel offline data deduplication mechanism, which can reduce the time consumption of data deduplication and effectively improve the efficiency of data deduplication.

[0113] In some optional embodiments, the process in step S204 above, where the first type of processing resource performs non-physical storage requests and physical storage operations on the data block to be processed based on the fingerprint query results and initial metadata generated by the second type of processing resource based on the fingerprint query request, and the second type of processing resource performs non-physical storage operations on the data block to be processed, is as follows: Figure 9 As shown, this can be achieved through the following steps: Step S901: The first type of processing resources performs physical storage operations on the data blocks to be processed.

[0114] For example, after the data IO processing service completes the query / generation of metadata, it checks the completion result of the fingerprint query request. If the fingerprint query result is not yet complete, it performs normal user write processing.

[0115] Step S902: If the fingerprint query result generated by the second type of processing resource based on the fingerprint query request indicates that the fingerprint is duplicated, then the first type of processing resource makes a non-physical storage request for the data block to be processed.

[0116] For example, after the data IO processing service completes the normal user write processing, it checks the fingerprint query result again. If the fingerprints are the same, it sends a data comparison request.

[0117] In step S903, the second type of processing resource responds to the non-physical storage request and performs non-physical storage operations on the data block to be processed; the non-physical storage operations also include offline deduplication.

[0118] For example, the data deduplication service compares / processes data, and performs offline deduplication by the data deduplication service.

[0119] This method provides an alternative parallel structure for offline data deduplication, which can reduce the time spent on data deduplication, lower the latency of data processing, and effectively improve the efficiency of data deduplication.

[0120] In some optional embodiments, the process in step S204 above, where the first type of processing resource performs non-physical storage requests and physical storage operations on the data block to be processed based on the fingerprint query results and initial metadata generated by the second type of processing resource based on the fingerprint query request, and the second type of processing resource performs non-physical storage operations on the data block to be processed, can be implemented through the following steps: Step A01: The first type of processing resource checks the fingerprint query result. If the fingerprint query result is not yet complete, normal user write processing is performed.

[0121] Step A02: The second type of processing resource sends the fingerprint query result back to the first type of processing resource; Step A03: After receiving the fingerprint query result returned by the second type of processing resource, if the first type of processing resource determines that the fingerprint query result has been checked before this moment, and if the received fingerprint query result indicates that there is a matching fingerprint, then a non-physical storage request is sent to the second type of processing resource; the non-physical storage request includes a request to cancel fingerprint locking. Step A04: The second type of processing resource performs fingerprint lock cancellation processing.

[0122] This method can further reduce data processing latency and effectively improve the efficiency of data deduplication.

[0123] The data deduplication method provided in this application is applied to a data deduplication system during the data deduplication process. The data deduplication system includes multiple nodes; each node includes a first type of processing resource and a second type of processing resource; the first type of processing resource is used for data I / O processing; the second type of processing resource is used for deduplication; the method includes: the first type of processing resource determines a target data fingerprint corresponding to the data block to be processed based on a preset fingerprint algorithm; the first type of processing resource generates a fingerprint query request corresponding to the data block to be processed based on the target data fingerprint and preset query result callback information; the fingerprint query request is used to enable the second type of processing resource to perform deduplication. The data deduplication mechanism involves several steps: fingerprint query of the data block to be processed; initial metadata corresponding to the data block to be processed obtained by the first type of processing resource; fingerprint query results and initial metadata generated by the second type of processing resource based on the fingerprint query request; non-physical storage requests and physical storage operations performed by the first type of processing resource on the data block to be processed; and non-physical storage operations performed by the second type of processing resource on the data block to be processed. Non-physical storage operations include data comparison; physical storage operations include write processing, deletion processing, and metadata processing. This provides a parallel data deduplication mechanism that can reduce the time consumption of data deduplication and effectively improve the efficiency of data deduplication.

[0124] Based on the same inventive concept, this application also provides a data deduplication device. Since this device corresponds to the data deduplication method provided in this application, and the principle of solving the problem by this device is similar to that of the method, the implementation of this device can refer to the implementation of the above method, and the repeated parts will not be described again.

[0125] Figure 10 This illustration shows a schematic diagram of a data deduplication device provided in an embodiment of this application, applied to a data deduplication system. The data deduplication system includes multiple nodes; each node includes a first type of processing resource and a second type of processing resource; the first type of processing resource is used for data I / O processing; the second type of processing resource is used for deduplicated data processing; as shown... Figure 10 As shown, the data deduplication device includes a fingerprint calculation and management module 1001, a fingerprint query and control module 1002, an initial element calculation module 1003, and a parallel deduplication module 1004.

[0126] Among them, the fingerprint calculation management module 1001 is used to determine the target data fingerprint corresponding to the data block to be processed based on the preset fingerprint algorithm of the first type of processing resources; The fingerprint query control module 1002 is used by the first type of processing resources to generate a fingerprint query request corresponding to the data block to be processed based on the target data fingerprint and the preset query result callback information; the fingerprint query request is used to enable the second type of processing resources to perform fingerprint query on the data block to be processed. The initial metadata calculation module 1003 is used by the first type of processing resources to obtain the initial metadata corresponding to the data block to be processed based on the data block to be processed. The parallel deduplication module 1004 is used to perform non-physical storage requests and physical storage operations on the data block to be processed by the first type of processing resources based on the fingerprint query results and initial metadata generated by the fingerprint query request by the second type of processing resources, and to perform non-physical storage operations on the data block to be processed by the second type of processing resources; the non-physical storage operations include data comparison; the physical storage operations include write processing, deletion processing, and metadata processing.

[0127] In an optional embodiment, the initial meta-calculation module 1003 is specifically used for:

[0128] If the metadata corresponding to the data block to be processed is found, the found metadata will be used as the initial metadata corresponding to the data block to be processed. If no metadata corresponding to the data block to be processed is found, metadata is generated for the data block to be processed, and the generated metadata is used as the initial metadata corresponding to the data block to be processed.

[0129] In an optional embodiment, the parallel deduplication module 1004 is specifically used for: If the fingerprint query result generated by the second type of processing resource based on the fingerprint query request indicates that the fingerprint is duplicated, then the first type of processing resource makes a non-physical storage request for the data block to be processed and modifies the initial metadata to obtain the target metadata, while the second type of processing resource performs a non-physical storage operation on the data block to be processed. If the fingerprint query result indicates that the fingerprint is not repeated, the first type of processing resource will use the initial metadata as the target metadata of the data block to be processed and perform physical storage operations.

[0130] In an optional embodiment, the parallel deduplication module 1004 is specifically used for: If the fingerprint query result generated by the second type of processing resources based on the fingerprint query request indicates that the fingerprint is duplicated, then the first type of processing resources will make a non-physical storage request for the data block to be processed. The second type of processing resource responds to non-physical storage requests and performs non-physical storage operations on the data blocks to be processed. The first type of processing resource modifies the initial metadata to obtain the target metadata.

[0131] In an optional embodiment, the non-physical storage operation further includes fingerprint locking; the non-physical storage request includes a data comparison request and a fingerprint locking request; the parallel deduplication module 1004 is specifically used for: The second type of processing resource responds to the data comparison request sent by the first type of processing resource and compares the data blocks to be processed. The second type of processing resource returns the data comparison results to the first type of processing resource; If the first type of processing resource determines that the data block to be processed has the same stored data based on the received data comparison results, it modifies the initial metadata to obtain the target metadata and generates a fingerprint locking request. The second type of processing resource responds to the fingerprint locking request sent by the first type of processing resource and performs fingerprint locking on the data block to be processed.

[0132] In an optional embodiment, the parallel deduplication module 1004 is specifically used for: If the fingerprint query result generated by the second type of processing resource based on the fingerprint query request indicates that the fingerprint is duplicated, then the data block to be processed is written to the power-saving memory. The first type of processing resource makes non-physical storage requests for the data blocks to be processed; The second type of processing resource responds to non-physical storage requests and performs non-physical storage operations on the data blocks to be processed. If the result of a non-physical storage operation indicates that the data block to be processed contains the same stored data, the first type of processing resource modifies the initial metadata to obtain the target metadata and deletes the data in the power-saving memory; If the result of the non-physical storage operation indicates that the data block to be processed does not have the same stored data, the first type of processing resource will use the initial metadata as the target metadata of the data block to be processed and perform physical storage operations.

[0133] In an optional embodiment, the parallel deduplication module 1004 is specifically used for: If the fingerprint query result generated by the second type of processing resource based on the fingerprint query request indicates that the fingerprint is duplicated, then the first type of processing resource makes a non-physical storage request for the data block to be processed, and the first type of processing resource performs a physical storage operation on the data block to be processed. The second type of processing resources performs non-physical storage operations on the data blocks to be processed based on the initial metadata; non-physical storage operations also include offline deduplication.

[0134] In an optional embodiment, the parallel deduplication module 1004 is specifically used for: The first type of processing resources performs physical storage operations on the data blocks to be processed; If the fingerprint query result generated by the second type of processing resources based on the fingerprint query request indicates that the fingerprint is duplicated, then the first type of processing resources will make a non-physical storage request for the data block to be processed. The second type of processing resource response is to non-physical storage requests, and to perform non-physical storage operations on the data blocks to be processed; non-physical storage operations also include offline deduplication.

[0135] Based on the same inventive concept, this application also provides a data deduplication system. Since this system includes the node provided by the method in the foregoing embodiments of this application; the node includes a first type of processing resources and a second type of processing resources; the first type of processing resources are used for data I / O processing; the second type of processing resources are used for deduplication processing; and the principle of this system in solving the problem is similar to that of the method, the implementation of this system can refer to the implementation of the above method, and repeated details will not be elaborated further.

[0136] Based on the same inventive concept as the above-described method embodiments, this application also provides an electronic device. This electronic device can be used for data deduplication. In this application embodiment, the electronic device can be a server or a terminal device. In one embodiment, the electronic device can be a terminal device. The terminal device can be a node included in the data deduplication system. In this embodiment, the structure of the electronic device can be as follows: Figure 11 As shown, it includes a memory 1101, a communication module 1103, and one or more processors 1102.

[0137] The memory 1101 is used to store computer programs executed by the processor 1102. The memory 1101 may mainly include a program storage area and a data storage area. The program storage area may store the operating system and programs required to run instant messaging functions, etc.; the data storage area may store various instant messaging information and operation instruction sets, etc.

[0138] Memory 1101 may be volatile memory, such as random-access memory (RAM); memory 1101 may also be non-volatile memory, such as read-only memory, flash memory, hard disk drive (HDD), or solid-state drive (SSD); or memory 1101 may be any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto. Memory 1101 may be a combination of the above-described memories.

[0139] Processor 1102 may include one or more central processing units (CPUs) or digital processing units, etc. Processor 1102 is used to implement the above-described data deduplication method when calling computer programs stored in memory 1101.

[0140] The communication module 1103 is used to communicate with terminal devices and other servers.

[0141] This application embodiment does not limit the specific connection medium between the memory 1101, communication module 1103, and processor 1102. This application embodiment... Figure 11 The memory 1101 and the processor 1102 are connected via a bus 1104, and the bus 1104 is in Figure 11 The connections between other components are shown in thick lines only and are not intended to be limiting. Bus 1104 can be divided into address bus, data bus, control bus, etc. For ease of illustration, Figure 11 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0142] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the data deduplication method described in the above embodiments. The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0143] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A data deduplication method, characterized in that, This is applied to a data deduplication system; the data deduplication system includes multiple nodes; each node includes a first type of processing resource and a second type of processing resource; the first type of processing resource is used for data I / O processing. The second type of processing resources is used for repetitive data processing; the method includes: The first type of processing resource determines the target data fingerprint corresponding to the data block to be processed based on a preset fingerprint algorithm; The first type of processing resource generates a fingerprint query request corresponding to the data block to be processed based on the target data fingerprint and preset query result callback information; the fingerprint query request is used to enable the second type of processing resource to perform a fingerprint query on the data block to be processed. The first type of processing resource obtains the initial metadata corresponding to the data block to be processed based on the data block to be processed; Based on the fingerprint query result generated by the second type of processing resource based on the fingerprint query request and the initial metadata, the first type of processing resource performs non-physical storage requests and physical storage operations on the data block to be processed, and the second type of processing resource performs non-physical storage operations on the data block to be processed; the non-physical storage operations include data comparison; the physical storage operations include write processing, deletion processing, and metadata processing.

2. The method according to claim 1, characterized in that, The process of performing non-physical storage requests and physical storage operations on the data block to be processed by the first type of processing resource based on the fingerprint query result generated by the second type of processing resource based on the fingerprint query request and the initial metadata, and performing non-physical storage operations on the data block to be processed by the second type of processing resource, includes: If the fingerprint query result generated by the second type of processing resource based on the fingerprint query request indicates that the fingerprint is duplicated, then the first type of processing resource makes a non-physical storage request for the data block to be processed and modifies the initial metadata to obtain the target metadata, and the second type of processing resource performs a non-physical storage operation on the data block to be processed. If the fingerprint query result indicates that the fingerprint is not repeated, the first type of processing resource will use the initial metadata as the target metadata of the data block to be processed and perform physical storage operations.

3. The method according to claim 2, characterized in that, If the fingerprint query result generated by the second type of processing resource based on the fingerprint query request indicates fingerprint duplication, then the first type of processing resource performs a non-physical storage request on the data block to be processed and modifies the initial metadata to obtain target metadata. The second type of processing resource performs a non-physical storage operation on the data block to be processed, including: If the fingerprint query result generated by the second type of processing resource based on the fingerprint query request indicates that the fingerprint is duplicated, then the first type of processing resource makes a non-physical storage request for the data block to be processed. The second type of processing resource responds to the non-physical storage request and performs non-physical storage operations on the data block to be processed, while the first type of processing resource modifies the initial metadata to obtain the target metadata.

4. The method according to claim 3, characterized in that, The non-physical storage operation further includes fingerprint locking; the non-physical storage request includes a data comparison request and a fingerprint locking request; the second type of processing resource responds to the non-physical storage request and performs non-physical storage operations on the data block to be processed, and the first type of processing resource modifies the initial metadata to obtain the target metadata, including: The second type of processing resource responds to the data comparison request sent by the first type of processing resource and performs data comparison on the data block to be processed; The second type of processing resource returns the data comparison results to the first type of processing resource; If the first type of processing resource determines, based on the received data comparison result, that the data block to be processed has the same stored data, then it modifies the initial metadata to obtain the target metadata and generates the fingerprint locking request; The second type of processing resource responds to the fingerprint locking request sent by the first type of processing resource and performs fingerprint locking on the data block to be processed.

5. The method according to claim 1, characterized in that, The process of performing non-physical storage requests and physical storage operations on the data block to be processed by the first type of processing resource based on the fingerprint query result generated by the second type of processing resource based on the fingerprint query request and the initial metadata, and performing non-physical storage operations on the data block to be processed by the second type of processing resource, includes: If the fingerprint query result generated by the second type of processing resource based on the fingerprint query request indicates that the fingerprint is duplicated, then the data block to be processed is written into the power-saving memory. The first type of processing resource makes a non-physical storage request for the data block to be processed; The second type of processing resource responds to the non-physical storage request and performs non-physical storage operations on the data block to be processed; If the result of a non-physical storage operation indicates that the data block to be processed contains the same stored data, the first type of processing resource modifies the initial metadata to obtain the target metadata and deletes the data in the power-saving memory; If the result of the non-physical storage operation indicates that the data block to be processed does not have the same stored data, the first type of processing resource will use the initial metadata as the target metadata of the data block to be processed and perform physical storage operation.

6. The method according to claim 1, characterized in that, The process of performing non-physical storage requests and physical storage operations on the data block to be processed by the first type of processing resource based on the fingerprint query result generated by the second type of processing resource based on the fingerprint query request and the initial metadata, and performing non-physical storage operations on the data block to be processed by the second type of processing resource, includes: If the fingerprint query result generated by the second type of processing resource based on the fingerprint query request indicates that the fingerprint is duplicated, then the first type of processing resource makes a non-physical storage request for the data block to be processed, and the first type of processing resource performs a physical storage operation on the data block to be processed. The second type of processing resources performs non-physical storage operations on the data block to be processed based on the initial metadata; the non-physical storage operations also include offline deduplication.

7. The method according to claim 1, characterized in that, The process of performing non-physical storage requests and physical storage operations on the data block to be processed by the first type of processing resource based on the fingerprint query result generated by the second type of processing resource based on the fingerprint query request and the initial metadata, and performing non-physical storage operations on the data block to be processed by the second type of processing resource, includes: The first type of processing resources performs physical storage operations on the data block to be processed; If the fingerprint query result generated by the second type of processing resource based on the fingerprint query request indicates that the fingerprint is duplicated, then the first type of processing resource makes a non-physical storage request for the data block to be processed. The second type of processing resource responds to the non-physical storage request and performs non-physical storage operations on the data block to be processed; the non-physical storage operations also include offline deduplication.

8. A data deduplication device, characterized in that, This is applied to a data deduplication system; the data deduplication system includes multiple nodes; each node includes a first type of processing resource and a second type of processing resource; the first type of processing resource is used for data I / O processing. The second type of processing resources is used for repetitive data processing; the apparatus includes: The fingerprint calculation and management module is used to determine the target data fingerprint corresponding to the data block to be processed based on a preset fingerprint algorithm for the first type of processing resources; The fingerprint query control module is used to generate a fingerprint query request corresponding to the data block to be processed based on the target data fingerprint and preset query result callback information of the first type of processing resources; the fingerprint query request is used to enable the second type of processing resources to perform fingerprint query on the data block to be processed. An initial metadata calculation module is used by the first type of processing resource to obtain the initial metadata corresponding to the data block to be processed based on the data block to be processed. The parallel deduplication module is used to perform non-physical storage requests and physical storage operations on the data block to be processed by the first type of processing resources based on the fingerprint query result generated by the fingerprint query request and the initial metadata, and the second type of processing resources perform non-physical storage operations on the data block to be processed; the non-physical storage operations include data comparison; the physical storage operations include write processing, deletion processing, and metadata processing.

9. The apparatus according to claim 8, characterized in that, The parallel deduplication module is specifically used for: If the fingerprint query result generated by the second type of processing resource based on the fingerprint query request indicates that the fingerprint is duplicated, then the data block to be processed is written into the power-saving memory. The first type of processing resource makes a non-physical storage request for the data block to be processed; The second type of processing resource responds to the non-physical storage request and performs non-physical storage operations on the data block to be processed; If the result of a non-physical storage operation indicates that the data block to be processed contains the same stored data, the first type of processing resource modifies the initial metadata to obtain the target metadata and deletes the data in the power-saving memory; If the result of the non-physical storage operation indicates that the data block to be processed does not have the same stored data, the first type of processing resource will use the initial metadata as the target metadata of the data block to be processed and perform physical storage operation.

10. The apparatus according to claim 8, characterized in that, The parallel deduplication module is specifically used for: If the fingerprint query result generated by the second type of processing resource based on the fingerprint query request indicates that the fingerprint is duplicated, then the first type of processing resource makes a non-physical storage request for the data block to be processed, and the first type of processing resource performs a physical storage operation on the data block to be processed. The second type of processing resources performs non-physical storage operations on the data block to be processed based on the initial metadata; the non-physical storage operations also include offline deduplication.

11. The apparatus according to claim 8, characterized in that, The parallel deduplication module is specifically used for: The first type of processing resources performs physical storage operations on the data block to be processed; If the fingerprint query result generated by the second type of processing resource based on the fingerprint query request indicates that the fingerprint is duplicated, then the first type of processing resource makes a non-physical storage request for the data block to be processed. The second type of processing resource responds to the non-physical storage request and performs non-physical storage operations on the data block to be processed; the non-physical storage operations also include offline deduplication.

12. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

13. An electronic device, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program that can run on the processor, and when the computer program is executed by the processor, it implements the method of any one of claims 1 to 7.

Images