Data deduplication method, device and equipment

Abstract

The invention discloses a data deduplication method, device and equipment, and the method comprises the steps: sampling a plurality of data fragments of first data to obtain a first data fragment, and obtaining a fingerprint of the first data fragment; and determining a deduplication domain of the first data fragment by using a fingerprint index and the fingerprint of the first data fragment, wherein the fingerprint index is an index constructed based on the fingerprint of the data fragment in the data set. Determining a deduplication domain of the first data fragment by using the fingerprint index and the fingerprint of the first data fragment; and determining a deduplication domain of a second data fragment according to the deduplication domain of the first data fragment, wherein the second data fragment is other data fragments in the first data. And writing the first data fragment in the deduplication domain of the first data fragment. And writing the second data fragment in the deduplication domain of the second data fragment. The deduplication domain of the second data fragment determined on the basis of the deduplication domain of the first data fragment is more suitable for being written into the second data fragment, so that the probability of existence of duplicated data is higher, and the deduplication rate of the data is effectively improved.

Description

Technical Field

[0001] This application relates to the field of storage technology, and more particularly to a data deduplication method, apparatus, and device. Background Technology

[0002] As data volumes grow exponentially, storage systems increasingly require data reduction capabilities. Data deduplication is one way to achieve this. The common implementation process for data deduplication is as follows:

[0003] The data to be written is split into multiple data shards. One shard is selected as a representative, and the deduplication field allocated based on this shard becomes the deduplication field for the entire data to be written. The deduplication field is essentially the set of system storage space or node storage space where the data to be written needs to be written. The data to be written is then written into this deduplication field. Within this deduplication field, deduplication is performed on the data; that is, data that is identical to data already stored within the deduplication field is deleted, and the deduplicated data is stored.

[0004] In the aforementioned data deduplication process, the deduplication rate (which indicates the ratio of deletable data to the total data volume) is highly dependent on the selected data shard and the allocated deduplication field. If the deduplication field allocated to the data shard does not contain any data identical to or partially identical to the data to be written, the deduplication rate of the data to be written will be low. Because the process of selecting a data shard from multiple data shards is random, this data deduplication method cannot reliably guarantee the deduplication rate. Summary of the Invention

[0005] This application provides a data deduplication method, apparatus, and device to improve the data deduplication rate.

[0006] In a first aspect, embodiments of this application also provide a data deduplication method, which is executed by a deduplication device. In this method, the deduplication device samples multiple first data fragments from multiple data fragments of first data and obtains fingerprints of the multiple first data fragments. The multiple first data fragments are some or all of the multiple data fragments.

[0007] The deduplication device can allocate deduplication fields to each data fragment (that is, determine the deduplication fields of each data fragment). For the first data fragment obtained by sampling and the remaining data fragments, the deduplication device can allocate deduplication fields in different ways:

[0008] The deduplication device uses a fingerprint index and the fingerprints of multiple first data fragments to determine the deduplication domains of multiple first data fragments. The fingerprint index is an index constructed based on the fingerprints of data fragments in the dataset. For any first data fragment, the deduplication device uses the fingerprint index and the fingerprint of that first data fragment to determine the deduplication domain of that first data fragment.

[0009] The deduplication device determines the deduplication field of the second data fragment based on the deduplication field of the first data fragment. The second data fragment is the data fragment in the first data that is not sampled.

[0010] For any one of the multiple first data fragments, the deduplication device writes the first data fragment into the deduplication field of the first data fragment. The deduplication device writes the second data fragment into the deduplication field of the second data fragment.

[0011] Using the above method, the deduplication device accurately determines the deduplication field of the first data fragment using a fingerprint index. Based on this field, the deduplication field of the second data fragment is more suitable for writing to that second data fragment, meaning there is a higher probability of duplicate data existing within its deduplication field, effectively improving the data deduplication rate. Furthermore, the deduplication fields of the first and second data fragments are identical, meaning they are written to the same area. This ensures that data stored in the first data fragment can be stored within the same area, guaranteeing data locality.

[0012] In one possible implementation, the fingerprint index records multiple key-value pairs, where the key in any key-value pair is the fingerprint of a data fragment in the dataset, and the value indicates the region where the data fragment is located. When the deduplication device uses the fingerprint index and the fingerprint of the first data fragment to determine the deduplication field of the first data fragment, for any first data fragment, it uses the fingerprint of the first data fragment to query the fingerprint index and determines the deduplication field of the first data fragment based on the query result.

[0013] By using the above method, since the fingerprint index records the area where the data fragment is located, when determining the deduplication field of the first data fragment, the fingerprint index can be used to determine whether there is a data fragment in the data set that is the same as the first data fragment, and then the deduplication field of the first data fragment can be determined, ensuring that the first data fragment can be written into the area where the same data fragment is located.

[0014] In one possible implementation, the query result indicates that the fingerprint of the first data shard was found in the fingerprint index, the deduplication field of the first data shard is the region indicated by the value in the target key-value pair, and the target key-value pair is the key-value pair in the fingerprint index whose key is the fingerprint of the first data shard.

[0015] Using the above method, the deduplication domain of the first data fragment contains a data fragment identical to the first data fragment. Writing the first data fragment into this deduplication domain can guarantee the deduplication rate of the first data fragment.

[0016] In one possible implementation, when the deduplication device determines the deduplication field of the second data segment based on the deduplication fields of multiple first data segments, it can determine the deduplication field of the second data segment based on the deduplication fields of K first data segments that are adjacent to the second data segment in the first data segment, where K is a positive integer.

[0017] Using the above method, adjacent data fragments in the first data are written into the same deduplication field, which can improve the deduplication rate and ensure data locality.

[0018] In one possible implementation, the deduplication field of the second data shard is the deduplication field that appears most frequently among the deduplication fields of the K neighboring first data shards.

[0019] Using the above method, the fact that the deduplication field appears most frequently indicates that there are many data fragments in the deduplication field that are the same as the first data. Furthermore, the probability that there are data fragments in the deduplication field that are the same as the second data fragment is also relatively high, so as to effectively ensure the deduplication rate of the first data.

[0020] In one possible implementation, when the deduplication device determines the deduplication field of the second data segment based on the deduplication fields of the multiple first data segments, it determines the deduplication field of the second data segment based on the number of occurrences of the deduplication fields of the multiple first data segments.

[0021] The above method can ensure that the second data fragment is written to a more suitable area, and can also ensure that the deduplication field is written to more data fragments in the first data, thus ensuring the data locality of the first data.

[0022] In one possible implementation, there are many ways to sample multiple first data fragments from multiple data fragments of the first data. One such method is listed here:

[0023] The deduplication device generates fingerprints of multiple data segments in the first data, and selects multiple first data segments whose fingerprints have preset features from the fingerprints of the multiple data segments.

[0024] Using the above method, the first data shard can be obtained simply and quickly from the multiple data shards.

[0025] In one possible implementation, the data set includes data stored in multiple nodes in the storage system, and the node to which the deduplication field of the first data shard belongs or the node to which the deduplication field of the second data shard belongs is one of the multiple nodes.

[0026] Using the above method, the allocation of deduplication domains can be realized within the storage system, and the deduplication of the first data can be realized within the storage system.

[0027] Secondly, embodiments of this application also provide a deduplication device that performs the behaviors described in the method examples of the first aspect. The beneficial effects are described in the first aspect description and will not be repeated here. The functions can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the aforementioned functions. In one possible design, the device structure includes a sampling module, an allocation module, and a data storage module, which can perform the corresponding functions in the method examples of the third aspect.

[0028] The sampling module is used to sample multiple first data segments from multiple data segments of the first data and obtain the fingerprints of the multiple first data segments.

[0029] The allocation module is used to determine the deduplication domain of multiple first data shards using a fingerprint index and the fingerprints of multiple first data shards. The fingerprint index is an index constructed based on the fingerprints of data shards in the data set. The module also determines the deduplication domain of a second data shard based on the deduplication domain of the multiple first data shards. The second data shard is a data shard in the first data set other than the multiple first data shards.

[0030] The data storage module is used to write a first data fragment into the deduplication field of any one of the multiple first data fragments; and to write a second data fragment into the deduplication field of any one of the multiple second data fragments.

[0031] In one possible implementation, the fingerprint index includes multiple key-value pairs, where the key in any key-value pair is the fingerprint of a data shard in the data set, and the value indicates the region where the data shard is located. When the allocation module allocates a deduplication field for the first data shard, it uses the fingerprint of the first data shard to query the fingerprint index and determines the deduplication field of the first data shard based on the query result.

[0032] In one possible implementation, the query result indicates that the fingerprint of the first data shard was found in the fingerprint index, the deduplication field of the first data shard is the region indicated by the value in the target key-value pair, and the target key-value pair is the key-value pair in the fingerprint index whose key is the fingerprint of the first data shard.

[0033] In one possible implementation, when determining the deduplication field of the second data fragment, the allocation module determines the deduplication field of the second data fragment based on the deduplication fields of the K first data fragments adjacent to the second data fragment in the first data, where K is a positive integer.

[0034] In one possible implementation, the deduplication field of the second data shard is the deduplication field that appears most frequently among the deduplication fields of the K neighboring first data shards.

[0035] In one possible implementation, when determining the deduplication field of the second data fragment, the allocation module determines the deduplication field of the second data fragment based on the number of occurrences of the deduplication fields of multiple first data fragments.

[0036] In one possible implementation, the sampling module generates fingerprints of multiple data segments in the first data, and selects multiple first data segments whose fingerprints conform to preset characteristics from the fingerprints of the multiple data segments.

[0037] In one possible implementation, the data set includes data stored in multiple nodes in the storage system, and the node to which the deduplication field of the first data shard belongs or the node to which the deduplication field of the second data shard belongs is one of the multiple nodes.

[0038] Thirdly, this application also provides a computing device, which includes a processor and a memory, and may further include a communication interface. The processor executes computer program instructions stored in the memory to perform the method provided in the first aspect or any possible implementation thereof. The memory is coupled to the processor and stores computer program instructions and data necessary for determining the data deduplication process. The communication interface is used for communicating with other devices, such as transmitting a first data fragment or a second data fragment.

[0039] Fourthly, this application provides a computing device system including at least one computing device. Each computing device includes a memory and a processor. The processor of at least one computing device is used to access computer program instructions in the memory to execute the methods provided in the first aspect or any possible implementation thereof.

[0040] Fifthly, this application provides a computer-readable storage medium that, when executed by a computing device, allows the computing device to perform the methods provided in the first aspect or any possible implementation thereof. The storage medium includes, but is not limited to, volatile memory, such as random access memory, and non-volatile memory, such as flash memory, hard disk drive (HDD), and solid-state drive (SSD).

[0041] Sixthly, this application provides a computing device program product, which includes computer program instructions. When executed by a computing device, the computing device performs the methods provided in the first aspect or any possible implementation thereof. The computer program product can be a software installation package, and when it is necessary to use the methods provided in the first aspect or any possible implementation thereof, the computer program product can be downloaded and executed on the computing device.

[0042] In a seventh aspect, this application also provides a computer chip connected to a memory, the chip being used to read and execute computer program instructions stored in the memory, and to execute the methods described in the first aspect and various possible implementations of the first aspect.

[0043] For the technical effects that can be achieved in the second to seventh aspects mentioned above, please refer to the description of the technical effects that can be achieved by the corresponding design scheme in the first aspect mentioned above. This application will not repeat them here. Attached Figure Description

[0044] Figure 1 This application provides a schematic diagram of the structure of a data deduplication system;

[0045] Figure 2 This is a schematic diagram of the structure of a deduplication device provided in an embodiment of this application;

[0046] Figure 3 This application provides a schematic diagram of a data deduplication method.

[0047] Figure 4 A fingerprint sampling diagram provided for this application;

[0048] Figure 5 A schematic diagram illustrating the allocation of deduplication fields for a second data fragment, as provided in this application;

[0049] Figures 6-9 This application provides a schematic diagram of a data deduplication scenario.

[0050] Figures 10-11 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application. Detailed Implementation

[0051] Before introducing the data deduplication method, system, apparatus, and device provided in the embodiments of this application, some concepts involved in the embodiments of this application will be explained first:

[0052] (1) Metadata.

[0053] Metadata, also known as intermediary data or relay data, is data about data. It mainly describes information about data properties, such as the storage address of the data, the data modification history, the size of the data, and the creation date of the data.

[0054] (2) Data fingerprint (or fingerprint for short).

[0055] A data fingerprint is information formed based on the data itself and can be used to represent the data. A data fingerprint can be understood as a "summary" extracted from the data. Data fingerprints can be used to compare data; when two sets of data have the same data fingerprint, it means that the two sets of data are very likely the same.

[0056] This application does not limit the method of generating data fingerprints. Generally, data fingerprints can be generated by calling a data fingerprint algorithm. Data fingerprint algorithms include, but are not limited to: MD5, xxhash, sha1, sha256, sha3, and blake3. This application does not limit the specific type of data fingerprint algorithm.

[0057] When generating a fingerprint of data, the data can first be segmented into multiple parts; for ease of explanation, one of these multiple parts is called a data slice. After obtaining multiple data slices, a fingerprint for each data slice is generated. The fingerprint of the data includes the fingerprints of all the data slices.

[0058] The granularity of data segmentation can be fixed, meaning each resulting data slice is the same size. Alternatively, the granularity can be variable, meaning each resulting data slice is not necessarily the same size. In this type of segmentation, a range of values ​​for the granularity can be set, thus limiting the range of sizes for the resulting data slices. Each data slice formed during data segmentation must fall within this range.

[0059] (2) Deleting domains.

[0060] For any data shard, the deduplication field refers to the storage area (also simply called a region) allocated for that data shard and required to be written to it. Typically, during data deduplication, deduplication fields are allocated to each data shard included in the data. For any given data shard, it needs to be written to its allocated deduplication field. If a data shard identical to the original already exists within the deduplication field, then when writing to the deduplication field, no write operation is required. That is, the data shard does not need to occupy storage space within the deduplication field; only the storage address of the data shard already stored within the deduplication field needs to be recorded. This storage address is the storage address of the data shard to be written to the deduplication field. If no identical data fragment is stored in the deduplication field of a data fragment, then when writing to the deduplication field, a write operation needs to be performed on that data fragment. This requires the data fragment to occupy storage space within the deduplication field to record its storage address. Therefore, the deduplication field of a data fragment limits its deduplication scope. In other words, when deduplicating a data fragment, the data already stored in its deduplication field is used as the data for comparison.

[0061] The specific form of the deduplication field in a data shard depends on the data writing scenario and the granularity of storage space partitioning within that scenario. For example, when writing data to a node and performing deduplication on that data, deduplication fields can be allocated within that node for each data shard included in the data. These fields can be the node's memory or its storage pool. A storage pool can be understood as a segment of storage space logically partitioned within the node. Similarly, when writing data to a storage system and performing deduplication on that data, deduplication fields can be allocated within the storage system for each data shard included in the data. These fields can be nodes within the storage system or their memory. The specific table format of the deduplication field in a data shard is quite flexible, and this application does not limit the specific form of the deduplication field.

[0062] (3) Fingerprint table, global fingerprint table.

[0063] The fingerprint table contains multiple key-value pairs. For any key-value pair, the key is the fingerprint of the data shard, and the value indicates the metadata of the data shard, such as the data shard's metadata or its storage address. This embodiment uses the example of a value indicating the data shard's metadata. In practice, however, the value can indicate part of the data shard's metadata, such as the data shard's storage address.

[0064] The fingerprint of a data fragment can be used to obtain information such as the storage address of that data fragment from the fingerprint table. From the perspective of the data range corresponding to the fingerprint recorded in the fingerprint table, fingerprint tables can be divided into global fingerprint tables and "local" fingerprint tables.

[0065] In this embodiment, a global fingerprint table is used as an example to illustrate a fingerprint table for a system with multiple nodes. The key-value pairs recorded in the global fingerprint table cover all multiple nodes. In other words, the global fingerprint table records the fingerprints of data shards on the multiple nodes and their metadata. In contrast, the fingerprint table of a specific node in the system is a "local" fingerprint table, which records the fingerprints of the data shards on that node and their metadata. It is important to clarify that in practical applications, the global fingerprint table may only record the fingerprints and metadata of a portion of the data shards on the multiple nodes. This design reduces the size of the global fingerprint table, preventing it from becoming too large. Similarly, a node's fingerprint table may only record the fingerprints and metadata of a portion of the data shards on that node, reducing the storage space required for that node's fingerprint table.

[0066] From the descriptions of "global fingerprint tables" and "local fingerprint tables," it's clear that "global fingerprint tables" and "local fingerprint tables" belong to the concept of "scope." This can be understood as follows: for a dataset containing multiple data sets, the fingerprint table built for that dataset can be called a global fingerprint table; that is, the global fingerprint table records the fingerprints of each data set and its metadata. For a subset of that dataset, the fingerprint table built for that subset can be understood as a "local" fingerprint table. The terms "global fingerprint table" and "local fingerprint table" are related to the settings of the dataset and subset. In the aforementioned explanation, the set of data from all nodes within the system is considered the dataset, and the set of data from a specific node is considered the subset. Therefore, extending this further, the global fingerprint table is not necessarily for a system with multiple nodes; the global fingerprint can be for a dataset. For example, when the dataset consists of data from a specific node, then the fingerprint table for that node is the global fingerprint table. Similarly, when the dataset consists of data from a specific memory location within that node, then the fingerprint table recording the fingerprints of the data fragments in that memory location and their metadata is the local fingerprint table. In this embodiment, the example of using the dataset consisting of data from various nodes within the system is merely used for illustration; in fact, the specific configuration of this dataset is not limited.

[0067] It should be noted that in this embodiment, the example given is that the value in the key-value pair indicates part or all of the metadata of the data shard. Given the flexible construction method of the indicator table (or fingerprint index) and the diverse value settings, this embodiment does not limit the information indicated by the values ​​in the key-value pairs of the indicator table (or fingerprint index); it only needs to ensure that the values ​​in the key-value pairs of the indicator table (or fingerprint index) can indicate the deduplication field of the data shard. For example, the value in the key-value pair can be the identifier of the node, or the identifier of the node's internal memory or storage pool. Alternatively, the value in the key-value pair can also be null. That is, the fingerprint table (or fingerprint index) only records the fingerprints of the data shards. For example, for a node's fingerprint table, when the fingerprint of a data shard is found in the fingerprint table, it indicates that the node stores the data shard, and the node can be used as the deduplication field of the data shard. For a fingerprint table of a certain memory within a node (that is, the fingerprint table records the fingerprints of data fragments in that memory), when the fingerprint of a data fragment is found in the fingerprint table, it means that the memory stores the data fragment, and the memory can be used as the deduplication domain of the data fragment.

[0068] (4) Fingerprint index.

[0069] An index is a data structure that sorts data according to a certain rule. Indexes enable fast data retrieval; common indexes include hash indexes, B-trees, and B+ trees. This application does not limit the specific structure of the fingerprint index.

[0070] A fingerprint index is an index built based on a fingerprint, in which information related to that fingerprint is retrieved by querying the fingerprint. The fingerprint-related information retrieved from a fingerprint index will vary depending on the application scenario.

[0071] In this embodiment, the "information related to the fingerprint" retrieved from the fingerprint index using the fingerprint serves as the metadata for the data shard. The fingerprint index records the data shards and their metadata in key-value pairs. For details regarding the key-value pairs and the information indicated by their values, please refer to the foregoing description; it will not be repeated here. Similar to fingerprint table types, fingerprint indexes can be categorized into global fingerprint indexes and "local" fingerprint indexes based on the data range corresponding to the fingerprints recorded in the fingerprint index. A global fingerprint index can be understood as a fingerprint index constructed based on the fingerprints of data within a dataset, while a "local" fingerprint index is a fingerprint index constructed based on the fingerprints of data within a subset of that dataset.

[0072] In the embodiments of this application, the fingerprint table can be regarded as a type of "fingerprint index".

[0073] like Figure 1The diagram shown is an architectural schematic of a data deduplication system provided in an embodiment of this application. The data deduplication system includes a deduplication device 100 and a management device 200.

[0074] The management device 200 interacts with the deduplication device 100 to perform data deduplication. The management device 200 can trigger the deduplication device 100 to perform deduplication operations on the data. In other words, the deduplication device 100 performs data deduplication at the request of the management device 200.

[0075] This application embodiment does not limit the different ways in which the management device 200 triggers the deduplication device 100 to perform deduplication operations on data. Several possible methods are listed below:

[0076] Method 1: The management device 200 sends a data write request to the deduplication device 100, the data write request carrying the first data to be written. The data write request is used to request the writing of the first data.

[0077] In this approach, the management device 200 has data access functionality and can write first data to the deduplication device 100 by sending a data write request. Upon receiving the data write request, the deduplication device 100 performs a deduplication operation on the first data. The deduplication device 100 deletes some or all of the data in the first data, wherein the deleted data is identical to data in a data set, which is a set of data accessible to the deduplication device 100, such as data already stored by the deduplication device 100. After deleting some or all of the data in the first data, the deduplication device 100 stores the remaining, undeleted data.

[0078] Method 2: The management device 200 sends a data deduplication request to the deduplication device 100. The data deduplication request is used to request the deduplication of the first data, and the data deduplication request carries the first data.

[0079] In this approach, the deduplication device 100 provides data deduplication services to the management device 200, which obtains the data deduplication service from the deduplication device 100 by sending a data deduplication request. Upon receiving the data deduplication request, the deduplication device 100 deletes some or all of the data in the first data set. The deleted data consists of duplicate data in the first data set and a data set that the deduplication device 100 can access, such as data managed by the management device 200. After deleting some or all of the data in the first data set, the deduplication device 100 returns the remaining, undeleted data to the management device 200.

[0080] Method 3: The management device 200 sends a data deduplication request to the deduplication device 100. This data deduplication request is used to determine whether there is data in the data set that is identical to the first data. The data deduplication request carries the first data or the fingerprint of the first data. The first data includes multiple data fragments, and the fingerprint of the first data includes the fingerprints of the multiple data fragments.

[0081] In this approach, the deduplication device 100 provides deduplication services to the management device 200, meaning the deduplication device 100 can inform the management device 200 which duplicate data exists in the first data to be queried. The management device 200 obtains the deduplication service from the deduplication device 100 by sending a data deduplication request. Upon receiving the data deduplication request, the deduplication device 100 obtains the fingerprint of the first data from the data deduplication request, or obtains the first data from the data query device, and generates the fingerprint of the first data. The deduplication device 100 determines whether there is data identical to the first data in the data set based on the fingerprint of the first data. Since the first data includes multiple data fragments, when determining whether there is data identical to the first data in the data set, the deduplication device 100 can determine whether there is data completely identical to the first data in the data set, and also whether there is data identical to data fragments in the first data in the data set. If there is data completely identical to the first data in the data set, the deduplication device 100 notifies the management device 200 that there is data completely identical to the first data in the data set. If the data set does not contain any data completely identical to the first data, but the data set contains data fragments identical to those in the first data, the deduplication device 100 notifies the management device 200 that the data set contains data fragments identical to those in the first data. Furthermore, the deduplication device 100 can also inform the management device 200 which one or more data fragments of the first data already exist in the data set. If the data set does not contain any data completely identical to the first data, and the data set does not contain data fragments identical to those in the first data, the deduplication device 100 notifies the management device 200 that the data set does not contain data identical to the first data or its data fragments.

[0082] This application does not limit the specific form of the management device 200. The management device 200 can be a hardware device, such as a computing device, a computing device cluster, or a chip or processor within a computing device. For example, the management device 200 and the deduplication device 100 can be deployed in a distributed storage system, which includes computing nodes and storage nodes, wherein the management device 200 can be a computing node and the deduplication device 100 can be a storage node. As another example, the management device 200 and the deduplication device 100 can be deployed in a distributed storage system, which includes multiple nodes, each node having both data processing and data storage functions, wherein the management device 200 and the deduplication device 100 are different nodes. The management device 200 can also be a software device, which can be deployed on one or more computing devices as containers or virtual machines, or other computing instances. For example, the computing instance where the management device 200 resides carries data management services. This computing instance needs to determine which node the first data to be written exists on. The computing instance can send a data query request to the deduplication device 100 to request confirmation that data identical to the first data exists in the dataset on that node. The management device 200 can also be an application deployed on a computing device. For example, the management device 200 can be database management software that periodically backs up the first data in the database. This database management software can back up the first data to the deduplication device 100 and can send a data write request to the deduplication device 100 to request writing the first data into the deduplication device 100.

[0083] The deduplication device 100 has a data deduplication function, meaning it can perform a data deduplication operation on the first data, deleting data fragments in the first data that are identical to those in the data set. Unlike conventional data deduplication operations, in this embodiment, the data deduplication operation performed by the deduplication device 100 is more efficient and can guarantee data locality to a certain extent. The advantages of this data deduplication operation performed by the device are mainly reflected in some or all of the following aspects:

[0084] Aspect 1: Fingerprint sampling.

[0085] When performing a data deduplication operation on the first data, the deduplication device 100 first samples the fingerprint of the first data. The first data includes multiple data segments, and the fingerprint of the first data includes the fingerprints of the multiple data segments. When sampling the fingerprint of the first data, the deduplication device 100 obtains the fingerprints of a portion of the data segments from the fingerprints of the multiple data segments included in the first data; the fingerprints of this portion of the data segments are the fingerprints obtained through sampling.

[0086] For ease of explanation, each data segment in this data segment is referred to as the first data segment, and the fingerprint obtained by sampling is the fingerprint of the first data segment. In this embodiment, the number of first data segments is not limited; there can be one or more. Each data segment in the first data segment other than the first data segment is referred to as the second data segment, and the fingerprint of the second data segment is the fingerprint that was not sampled.

[0087] Fingerprint sampling can greatly reduce the workload when querying the ownership of data fragments, reduce the amount of data to be queried, and ensure the efficiency of the data deduplication operation.

[0088] It should be noted that the embodiments in this application are only described as examples where the sampling of the deduplication device 100 is performed on the fingerprint of the first data (i.e., the fingerprint of multiple data fragments). In practical applications, the deduplication device 100 can first sample the multiple data fragments, and use the extracted (also known as acquired) multiple data fragments as multiple first data fragments, thereby generating the fingerprint of the first data fragment.

[0089] Part Two: Inquiry into the attribution of sampled fingerprints.

[0090] After fingerprint sampling, the fingerprint of the first data will be divided into two parts: one part is the sampled fingerprint, which is the fingerprint of the first data segment, and the other part is the unsampled fingerprint, which is the fingerprint of the second data segment.

[0091] For any fingerprint of a first data segment, the deduplication device 100 determines the deduplication field of the first data segment based on the fingerprint index, that is, determines the area that the first data segment needs to be written.

[0092] The deduplication field of the first data fragment determined by the fingerprint index is more accurate, and the probability of storing the same first data fragment in the deduplication field of the first data fragment is higher, which can effectively improve the deduplication rate of the first data.

[0093] Thirdly, the prediction of the attribution of unsampled fingerprints.

[0094] For the fingerprint of the second data segment, the deduplication device 100 can determine the deduplication field of the second data segment based on the deduplication field of the first data segment. The process of determining the deduplication field of the second data segment based on the deduplication field of the first data segment is a prediction process for the deduplication field of the second data segment. In this process, the deduplication fields of the queried first data segment can be analyzed, the possible deduplication fields of the second data segment can be predicted, and then the deduplication field of the second data segment can be determined from the predicted possible deduplication fields of the second data segment.

[0095] Determining the deduplication field of the second data fragment using a more accurate deduplication field of the first data fragment ensures that the second data fragment is allocated to a more suitable region, where the probability of storing the same second data fragment is higher, thus improving the overall deduplication rate of the first data. In addition, there is a certain correlation between the deduplication fields of the first and second data fragments, and the deduplication fields of the second and first data fragments may be the same. The first and second data fragments can be written into the same deduplication field, which ensures that continuous data within the first data (i.e., adjacent first and second data fragments) are located within the same deduplication field, thus guaranteeing data locality.

[0096] This application does not limit the specific form of the deduplication device 100. The deduplication device 100 can be a hardware device, such as a computing device, a computing device cluster, or a chip or processor within a computing device. For example, the deduplication device 100 can be deployed in a storage system used for data backup, such as on a controller in the storage system (e.g., a controller in a centralized storage system), or on a storage node in the storage system that has data storage capabilities. It performs deduplication on any data that needs to be backed up to the storage system, and then stores the deduplicated data after the deduplication is complete. The deduplication device 100 can also be a software device, such as a container or virtual machine instance deployed on one or more computing devices. For example, the computing instance where the deduplication device 100 resides carries the data deduplication service, and the management device 200 can request data deduplication services from the deduplication device 100. The deduplication device 100 can also be an application deployed on a computing device. For example, the deduplication device 100 is software used to implement data deduplication. The software can periodically deduplicat data stored in the computing device where the software is located, or deduplicat data that needs to be written to the computing device.

[0097] The structure of the deduplication device 100 is described below, such as... Figure 2 The diagram shown is a structural schematic of the deduplication device 100. Logically, the deduplication device 100 includes a routing module 110 and a fingerprint index. Optionally, the deduplication device 100 also includes a data storage module 120, which includes a deduplication detection module 121 and a storage module 122.

[0098] The routing module 110 can sample multiple first data fragments from multiple data fragments of the first data, and allocate deduplication fields for the first data fragments and the data fragments other than the first data fragments (i.e., the second data fragments).

[0099] For example, the routing unit includes a sampling module 111 and an allocation module 112.

[0100] The sampling module 111 is used to segment the first data to obtain multiple data fragments of the first data. The sampling module 111 samples from the multiple data fragments to obtain multiple first data fragments. The allocation module 112 allocates deduplication fields to the multiple data fragments.

[0101] For the first data shard, the allocation module 112 uses the fingerprint of the first data shard to query the fingerprint index and determines the deduplication field of the first data shard based on the query result.

[0102] For the second data fragment, the allocation module 112 uses the deduplication field of the first data fragment to determine the deduplication field of the second data fragment. For details regarding the deduplication fields of the first and second data fragments, please refer to [link to relevant documentation]. Figure 3 The relevant descriptions in the illustrated embodiments will not be repeated here.

[0103] After determining the deduplication domain of the first data fragment and the deduplication domain of the second data fragment, the allocation module 112 can distribute the first data fragment and the second data fragment to the corresponding deduplication domain.

[0104] The data storage module 120 is used to write the first data fragment into the deduplication field of the first data fragment and to write the second data fragment into the deduplication field of the second data fragment.

[0105] The deduplication module 121 is deployed in each deduplication domain. For any deduplication domain, the deduplication module 121 checks any data fragment (first data fragment or second data fragment) distributed to that deduplication domain by the routing module 110 (i.e., the allocation module 112) to determine whether there is a data fragment identical to the data fragment in that deduplication domain. If not, the data fragment is transmitted to the storage module 122. If it exists, the metadata of the data fragment is recorded, such as the storage address and write time of the data fragment.

[0106] The storage module 122 receives the data fragments transmitted by the deduplication module 121, allocates a storage address for the data fragment, stores the data fragment at the storage address, and records the metadata of the data fragment.

[0107] It should be noted that the module division in this embodiment is illustrative and represents only one logical functional division; in actual implementation, there may be other division methods. The functional modules in this embodiment can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules can be implemented in hardware or as software functional modules.

[0108] The following is combined Figure 3This application describes a data deduplication method provided in its embodiments. Figure 3 The following example illustrates how management device 200 requests data to be written to deduplication device 100. In other methods where management device 200 triggers deduplication device 100 to perform data deduplication, the deduplication operation performed by deduplication device 100 is similar, differing only in the information sent by management device 200 and whether deduplication device 100 executes only some steps. For instance, when management device 200 requests deduplication device 100 to provide data deduplication service, deduplication device 100 may only execute steps 304-305. After executing steps 304-305, it can inform management device 200 of the deduplication domains for the first and second data fragments. Similarly, when management device 200 requests deduplication device 100 to provide data deduplication service, deduplication device 100 may only execute steps 304-306. After executing steps 304 to 306, the deduplication device 100 can deduplicat the first data fragment in the deduplication field of the first data fragment, and deduplicat the second data fragment in the deduplication field of the second data fragment, and then feed back the remaining data fragments after deduplication to the management device 200.

[0109] Step 301: The management device 200 sends a data write request to the deduplication device 100. The data write request is used to request the writing of first data, and the data write request carries the first data.

[0110] There are many scenarios in which the management device 200 sends a data write request to the deduplication device 100. For example, when the deduplication device 100 is deployed on a storage node in a storage system, and the management device 200 needs to back up the first data, it can send a data write request to the deduplication device 100. Another example is when the deduplication device 100 is a device with data storage capabilities, and the management device 200 is located close to the user. When the user needs to store the first data, or for example, when the user operates the management device 200, it triggers the management device 200 to send a data write request to the deduplication device 100.

[0111] Step 302: After receiving a data write request, the deduplication device 100 divides the first data into multiple data fragments and generates fingerprints for each data fragment.

[0112] The deduplication device 100 receives the data write request, obtains first data from the data write request, and segments the first data to obtain multiple data fragments. This application embodiment does not limit the method by which the deduplication device 100 segments the first data. For example, the deduplication device 100 segments the first data with a fixed data granularity to form multiple data fragments of the same length. Alternatively, the deduplication device 100 segments the first data with a variable data granularity to form multiple data fragments of different lengths, such as using content-defined chunking (CDC) to segment the first data.

[0113] It should be noted that in practical applications, there may be cases where the first data carried in the data write request sent by the management device 200 to the deduplication device 100 is relatively small. In this case, the deduplication device 100 can perform data caching, that is, cache the first data, and continue to receive other data write requests sent by the management device 200, caching the data to be written carried in these other data write requests, until the amount of cached data reaches a threshold. Step 302 is then performed on the cached data (including the first data), that is, the cached data is split to obtain multiple data fragments.

[0114] Step 303: The deduplication device 100 samples the fingerprints of each data segment to obtain the fingerprint of that data segment. Each data segment in that data segment is called a first data segment. That is, the deduplication device 100 samples one or more first data segments from the multiple data segments.

[0115] Sampling refers to selecting a subset of fingerprints from multiple data segments. This application does not limit the sampling method used by the deduplication device 100 for each data segment's fingerprints; several sampling methods are listed below:

[0116] Sampling method 1: Interval sampling, which means sampling the fingerprint of a data segment every N data segments.

[0117] The multiple data fragments are arranged in a sequential order according to their position in the first data. The deduplication device 100 can extract the fingerprint of one data fragment every N fragments based on the sequential arrangement of the multiple data fragments. The fingerprint of the extracted data fragment is the fingerprint of the first data fragment. N is a positive integer and can be preset.

[0118] Sampling method 2, content sampling, involves selecting the fingerprint of the data segment whose fingerprint matches a preset characteristic from the fingerprints of the multiple data segments. The fingerprint of the data segment whose fingerprint matches the preset characteristic is the fingerprint of the first data segment.

[0119] In the content sampling method, fingerprints of data segments with the same fingerprint characteristics are extracted. This preset characteristic can be pre-set, and the specific content of the preset characteristic is not limited in this application embodiment. For example, the preset characteristic can be that the Mth bit in the fingerprint is 1. Another example is that the preset characteristic is FP%(1 / R) = 0, where FP is the fingerprint of the data segment, and R represents the sampling ratio. This preset characteristic means that the remainder of the fingerprint of the data segment divided by the sampling ratio is equal to zero. When sampling fingerprints from multiple data segments according to this preset characteristic, the sampling ratio (the sampling ratio is equal to the ratio of the number of fingerprints of the sampled data segments to the total number of fingerprints of the multiple data segments) can reach R or be close to R.

[0120] Sampling method 3: Random sampling.

[0121] In the random sampling method, only the sampling ratio can be set. The deduplication device 100 randomly selects the fingerprint of the first data segment from the fingerprints of the multiple data segments. The ratio of the number of fingerprints obtained from the selected first data segment to the number of fingerprints of the multiple data segments is equal to the sampling ratio.

[0122] Sampling method 4: Select the fingerprints of the P data segments with the largest fingerprint values ​​from the fingerprints of the multiple data segments.

[0123] The fingerprints of the multiple data shards are sorted from largest to smallest according to their values. After sorting, the first P data shards are the fingerprints of the first data shard. This P can be preset or determined according to the sampling ratio.

[0124] Sampling method 5: Select the Q data segments with the smallest fingerprint values ​​from the fingerprints of these multiple data segments.

[0125] The fingerprints of the multiple data slices are sorted in ascending order of their values. After sorting, the first Q data slices are the fingerprints of the first data slice. This Q can be preset or determined based on the sampling ratio.

[0126] The sampling methods listed above are relatively simple and efficient, and can reduce the size of the fingerprint index.

[0127] Step 304: For any data fragment, the deduplication device 100 determines the deduplication domain of the first data fragment based on the fingerprint index. The fingerprint index is an index constructed based on the fingerprints of the data fragments in the dataset.

[0128] After acquiring the fingerprints of multiple first data segments, the deduplication device 100 uses the fingerprints of the multiple first data segments to query the fingerprint index and allocate deduplication fields to the first data segments.

[0129] For any first data shard, querying the fingerprint index using the fingerprint of that first data shard yields two query results:

[0130] Query result 1: The fingerprint of the first data fragment was found in the fingerprint index.

[0131] The fact that the fingerprint of the first data fragment is found in the fingerprint index indicates that the fingerprint of the first data fragment is recorded in the fingerprint index, and that a data fragment identical to the first data fragment already exists in the dataset. For ease of explanation, the data fragment identical to the first data fragment in the dataset is referred to as the target data fragment.

[0132] When the fingerprint index records a "value" corresponding to the fingerprint of the target data fragment, the deduplication field of the first data fragment is determined based on the value.

[0133] For example, when this value indicates the metadata or storage address of the target data shard, the metadata of the target data shard includes its storage address. Based on this storage address, the region where the target data shard is located can be determined. This region can be a node, a memory, a storage pool, etc. The specific representation of this region depends on how the storage space of the data set is divided. For example, when the data set is distributed among nodes, meaning the data set includes data within nodes, then the region where the target data shard is located is the storage space within that node. This region can be the node's memory or a storage pool, etc. When the data set is distributed in a system including multiple nodes, meaning the data set includes data within that system, then the region where the target data shard is located is the storage space within that system. This region can be the node's memory, a storage pool, etc., or it can be the node itself. The deduplication field of the first data shard can be the region where the target data shard is located.

[0134] For example, when this value indicates the region where the target data shard is located, the deduplication field of the first data shard can be the region where the target data shard is located.

[0135] When the fingerprint index does not record a "value" corresponding to the fingerprint of the target data shard, meaning the fingerprint index only records the fingerprints of each data shard in the dataset, the deduplication field of the first data shard is the region where the dataset is distributed. For example, when the dataset is distributed on a node, the deduplication field of the first data shard is that node. When the dataset is distributed in the memory of a node, meaning the dataset is a data shard stored in that memory, the deduplication field of the first data shard is that memory.

[0136] like Figure 4 The diagram illustrates how the deduplication device 100 samples the fingerprint of the first data segment from the fingerprints of multiple data segments and assigns a deduplication field to the first data segment.

[0137] exist Figure 4 In the process, the deduplication device 100 divides the first data based on variable data granularity to form multiple data fragments, which are respectively identified as data fragment A, data fragment B, data fragment C, data fragment D, data fragment E, data fragment F, data fragment G, data fragment H, data fragment I, data fragment J, data fragment K, data fragment L, data fragment M, and data fragment N.

[0138] The deduplication device 100 samples and selects data fragments C, G, J, and M as the first data fragments. The device 100 then uses each of these fragments to query a fingerprint index to determine if they exist within the already stored data fragments. This determines the regions where the stored fragments are located. For example, if the fingerprint index indicates that stored fragments C, J, and M are located in region 2, and stored fragment G is located in region 1, then the deduplication domains for data fragments C, G, J, and M in the first data fragment are region 2, region 1, region 2, and region 2, respectively.

[0139] Query result 2: No fingerprint of the first data segment was found in the fingerprint index.

[0140] The fact that no fingerprint of the first data segment was found in the fingerprint index indicates that the fingerprint index does not record the fingerprint of the first data segment. It is highly likely that there is no data segment in the dataset that is the same as the first data segment. Since the fingerprint index may only record the fingerprints of some data segments, the fact that no fingerprint of the first data segment was found in the fingerprint index only indicates that there is a high probability that the first data segment does not exist in the dataset.

[0141] In this case, the embodiments of this application do not limit the method by which the deduplication device 100 allocates deduplication fields to the first data fragment. Several methods are listed here.

[0142] Allocation Method 1: The deduplication device 100 allocates a deduplication domain to the first data shard based on a load balancing strategy.

[0143] As can be seen from the foregoing explanations regarding the "target data fragment location region" and the deduplication domain, the specific form of a region (or deduplication domain) is related to the way storage space is divided, such as using nodes as the granularity, or using memory or storage pools as the granularity. For ease of explanation, the storage space that can serve as a deduplication domain is referred to as a candidate region. A candidate region only represents a segment of storage space, such as a node, memory, or storage pool.

[0144] The deduplication device 100 selects a candidate region from multiple candidate regions as the deduplication domain for the first data shard based on a load balancing strategy. The load balancing strategy refers to the fact that the load (such as the idle computing power of the node or the idle storage space) of the multiple candidate regions can reach a relatively balanced state, such as ensuring that the difference between the loads of the multiple candidate regions is within a preset difference range, or that the loads of the multiple candidate regions are all within a preset load range.

[0145] For example, each candidate region is a memory within a node. When the deduplication device 100 selects a candidate region from multiple candidate regions as the deduplication domain for the first data shard based on a load balancing strategy, it refers to the free storage space in each memory and selects the memory with the most free storage space as the deduplication domain for the first data shard.

[0146] Allocation Method Two: The deduplication device 100 allocates deduplication fields to the first data fragment based on the deduplication fields of the first data fragment whose deduplication fields have been determined.

[0147] Since it is highly likely that no data fragment identical to the first data fragment exists in the dataset, regardless of which candidate region is chosen as the deduplication domain for the first data fragment, storage space will most likely need to be allocated within that domain to store the first data fragment. This first data fragment will require storage space within the deduplication domain.

[0148] When allocating deduplication fields for the first data fragment, the deduplication device 100 strives to ensure that the deduplication fields of the first data fragment are the same as those of its neighboring data fragments. Here, a data fragment adjacent to the first data fragment refers to a data fragment in the first data that is geographically close to the first data fragment. A data fragment adjacent to the first data fragment can be a data fragment that is adjacent to the first data fragment, or a data fragment that is geographically separated from the first data fragment by a small interval (e.g., the length of the interval is less than an interval threshold).

[0149] The deduplication device 100 determines data fragments adjacent to the first data fragment from the first data fragment with a determined deduplication domain. For ease of explanation, the determined data fragments adjacent to the first data fragment are referred to as adjacent data fragments. These adjacent data fragments are one or more of the sampled first data fragments, and deduplication domains have been assigned to them. The deduplication device 100 uses the deduplication domain of the adjacent data fragment as the deduplication domain of the first data fragment. If the deduplication device 100 determines multiple adjacent data fragments, and the multiple adjacent data fragments have different deduplication domains, the deduplication device 100 can use the deduplication domain of one of the adjacent data fragments as the deduplication domain of the first data fragment, such as using the deduplication domain of the adjacent data fragment closest to the first data fragment as the deduplication domain of the first data fragment.

[0150] Step 305: The deduplication device 100 determines the deduplication field of the second data fragment based on the deduplication field of the first data fragment.

[0151] In step 304, the deduplication device 100 allocates deduplication fields to each first data fragment. For the unsampled data fragment, that is, the second data fragment, it is also necessary to allocate deduplication fields to it.

[0152] For any second data fragment, the regeneration device determines the deduplication field of the second data fragment based on the deduplication field of the first data fragment. This application does not limit the specific implementation of step 305; two possible implementations are listed below:

[0153] Method 1: Nearest Neighbor Prediction

[0154] The deduplication device 100 predicts the deduplication domain of the second data fragment based on the deduplication domains of the K first data fragments adjacent to the second data fragment. K is a set positive integer. The definition of proximity can be found in the foregoing description and will not be repeated here. K represents the number of first data fragments adjacent to the second data fragment.

[0155] If the deduplication field of the K adjacent first data fragments is the same, then the deduplication field of the second data fragment is the same as the deduplication field of the K first data fragments.

[0156] If the deduplication fields of the K adjacent first data fragments are not the same, the deduplication device 100 uses the deduplication field of one of the K first data fragments as the deduplication field of the second data fragment. For example, the deduplication device 100 determines the number of times different deduplication fields appear in the deduplication fields of the K first data fragments and uses the deduplication field that appears most frequently as the deduplication field of the second data fragment.

[0157] like Figure 5 The diagram shown illustrates how the deduplication device 100 determines the second data allocation deduplication field. Figure 5 In the diagram, the first data segment comprises multiple data shards, with each square representing a data shard. A positive integer value within a square indicates that the data shard is the first data shard, and the positive integer value is the region identifier of the deduplication field of the first data shard. A value of -1 within a square indicates that the data shard is the second data shard.

[0158] Let K equal 4. For the first second data fragment, determine its four neighboring first data fragments. The deduplication fields of these four first data fragments are different. The deduplication field of three of the first data fragments is region 1, and the deduplication field of one of the first data fragments is region 2. Region 1 is used as the deduplication field of the first second data fragment.

[0159] For another second data shard, identify its four neighboring first data shards. The deduplication field of these four first data shards is the same, and the deduplication field of these four first data shards is region 2. Use region 2 as the deduplication field of this second data shard.

[0160] Method 2: Statistical forecasting.

[0161] The deduplication device 100 counts the deduplication fields of each first data segment and uses the deduplication field that appears most frequently as the deduplication field of the second data segment.

[0162] Alternatively, the deduplication device 100 may only perform statistics on the first data fragments corresponding to the fingerprints retrieved from the fingerprint index. The deduplication device 100 counts the deduplication fields of these first data fragments and uses the deduplication field that appears most frequently as the deduplication field of the second data fragment.

[0163] Step 306: The deduplication device 100 writes the first data fragment into the deduplication field of the first data fragment, and writes the second data fragment into the deduplication field of the second data fragment.

[0164] After determining the deduplication fields of the first data fragment and the second data fragment, the deduplication device 100 can write the first data fragment and the second data fragment into their respective deduplication fields.

[0165] Since the deduplication fields of the first data fragment and the second data fragment may share the same deduplication field, the deduplication device 100 can aggregate the first data fragments and / or the second data fragments with the same deduplication field together. Thus, for any deduplication field, a data fragment set containing at least one data fragment can be formed. For ease of explanation, the aggregated data fragment set is referred to as a fragment set. The deduplication device 100 writes each of the formed fragment sets into its corresponding deduplication field.

[0166] This application does not limit the method of writing the fragment set to the corresponding deduplication field; only the writing process is briefly described here:

[0167] The deduplication device 100 performs a deduplication check on the fragment set within the deduplication domain. For any data fragment (such as the first data fragment or the second data fragment) within the fragment set, it determines whether a duplicate data fragment has been stored in the deduplication domain, and whether the duplicate data fragment is the same as the data fragment in the fragment set.

[0168] If duplicate data shards are stored, record the metadata of the data shards in the shard set that are identical to the duplicate data shard, such as the storage address of the data shard (the storage address of the data shard is the same as the storage address of the duplicate data shard), the write time of the data shard, etc.

[0169] If no duplicate data fragments are stored, a storage address is allocated for the data fragment within the deduplication domain, the data fragment is written to the storage address, and the metadata of the data fragment is recorded, such as the storage address of the data fragment and the write time of the data fragment.

[0170] After recording the metadata of the data fragment, the fingerprint index can be updated to update the metadata of the data fragment or to add the fingerprint of the data fragment and its metadata to the fingerprint index.

[0171] The following lists several scenarios in which the data deduplication method provided in the embodiments of this application is applicable.

[0172] Scenario 1: Backup Scenario.

[0173] Backup scenarios are common in systems with multiple nodes, where data can be backed up from one node to another. This node can be understood as a computing device such as a server, or a component with computing capabilities (such as a processor or controller).

[0174] like Figure 6 The diagram shown is a structural schematic of a backup system provided in this embodiment. The backup system includes multiple nodes. Figure 6 Only two nodes are shown as examples: node 1 and node 2.

[0175] Taking the example of Node 1 backing up data to Node 2, Node 1 can send a data write request to Node 2, which carries the first data. After receiving the data write request, Node 2 can execute steps 302 to 306. In this case, the deduplication domain allocated by Node 2 for each data shard is in Node 2. For example, the deduplication domain can be a memory or storage pool in Node 2.

[0176] Scenario 2: A scenario where the storage system provides data backup functionality to external users.

[0177] Storage systems have data storage capabilities; therefore, they are commonly used for backup functions, accepting backups from external sources and performing data backups. During the backup process, data can be deleting and then stored. This explanation will use the example of the first set of data to be backed up.

[0178] The following section uses a distributed storage system as an example to illustrate the data deduplication method in data storage scenarios.

[0179] like Figure 7The diagram shown is a structural schematic of a distributed storage system provided in this embodiment. The distributed storage system includes multiple nodes. It is assumed that the distributed storage system includes n nodes, which are respectively identified as node 1, node 2, ..., node n.

[0180] Suppose node 1 receives a data backup request from an external source, requesting the backup of first data. Node 1 segments the first data to obtain multiple data fragments; node 1 generates fingerprints of these multiple data fragments, and samples these fingerprints to obtain the fingerprint of the first data fragment.

[0181] Node 1 broadcasts a data deduplication request to the other nodes. The data deduplication request carries the fingerprint of the first data fragment. The data query request is used to ask whether the other nodes have already stored the first data fragment.

[0182] For any of the remaining nodes, taking node 2 as an example, after receiving a data deduplication request, node 2 queries its fingerprint index based on the fingerprint of the first data shard to determine whether it stores a data shard identical to the first data shard. After completing the query, node 2 sends a data deduplication response back to node 1. This response indicates whether node 2 stores a data shard identical to the first data shard, and which first data shard is stored.

[0183] Of course, Node 1 will also query the fingerprint index of Node 1 based on the fingerprint of the first data shard to determine whether Node 1 stores the same data shard as the first data shard.

[0184] After receiving the data deduplication responses from each node and completing its query, Node 1 will divide the first data shard into two categories. One category consists of first data shards already stored in nodes within the distributed storage system; the deduplication field for this type of first data shard is the node where the first data shard resides. The other category consists of first data shards not stored in nodes within the distributed storage system. Node 1 can allocate deduplication fields to this type of first data shard. The allocation method can be found in the relevant explanation in step 304, and will not be repeated here.

[0185] After Node 1 has allocated a deduplication field to the first data shard, it can allocate a deduplication field to the second data shard. The allocation method can be found in the relevant description of step 305, which will not be repeated here.

[0186] After Node 1 allocates deduplication fields for the first and second data shards, it can aggregate data allocations with the same deduplication fields to form a shard set, and distribute the shard set to the corresponding nodes. After receiving the shard set, the corresponding node performs deduplication on the shard set and stores it.

[0187] Scenario 3: Data storage scenario.

[0188] Data storage scenarios are common in storage systems. When a storage system receives a data write request from an external source, it will delete the first piece of data to be written and then store it.

[0189] The scenario where a storage system provides data backup functionality to the outside world is similar to the data storage scenario. The scenario where a storage system provides data backup functionality to the outside world can be regarded as a special case of the data storage scenario. In the scenario where a storage system provides data backup functionality to the outside world, the data that needs to be written is the first data that needs to be backed up.

[0190] Ren Yi Figure 7 For example, here is a schematic diagram of a distributed storage system provided in this embodiment. The distributed storage system includes multiple nodes. Assume that the distributed storage system includes n nodes, which are respectively identified as node 1, node 2, ..., node n.

[0191] Suppose node 1 receives a data write request from an external source, requesting the writing of first data. Node 1 segments the first data to obtain multiple data fragments; node 1 generates fingerprints of these multiple data fragments, samples these fingerprints, and obtains the fingerprint of the first data fragment.

[0192] Node 1 broadcasts a data deduplication request to the other nodes. The data deduplication request carries the fingerprint of the first data fragment. The data query request is used to ask whether the other nodes have already stored the first data fragment.

[0193] The operations performed by Node 1 and the other nodes are similar to those described in the scenario where the storage system provides data backup functionality. For details, please refer to the foregoing explanation, which will not be repeated here.

[0194] Scenario 4: Source-side deleting scenario.

[0195] Source-side deduplication refers to the deduplication of the first data at the "source end" of the data (in some scenarios, the "source end" can also be called the client). The source end interacts with the storage side to determine which data fragments of the first data have been stored by the storage side. The source end only sends the data fragments of the first data that have not been stored by the storage side to the storage side for storage. For the data fragments of the first data that have been stored by the storage side, the source end can send the metadata of the data fragments to the storage side.

[0196] like Figure 8The diagram shown is a schematic representation of a source-side deduplication system provided in this embodiment. The source-side deduplication system includes a client and a storage terminal. The client can be understood as the front end of the storage terminal. Both the client and the storage terminal can be hardware devices or software modules. This embodiment does not limit the specific form of the client and the storage terminal.

[0197] When the client needs to store the first data on the storage end, the client splits the first data into multiple data fragments; the client generates fingerprints of the multiple data fragments, samples the fingerprints of the multiple data fragments, and obtains the fingerprint of the first data fragment.

[0198] The client sends a data deduplication request to the storage device. The data deduplication request carries the fingerprint of the first data fragment. The data query request is used to inquire whether the first data fragment has been stored in the storage device.

[0199] After receiving a data deduplication request, the storage terminal queries its fingerprint index based on the fingerprint of the first data shard to determine whether a data shard identical to the first data shard is stored. Upon completion of the query, the storage terminal sends a data deduplication response back to the client. This response indicates whether a data shard identical to the first data shard is stored in the storage terminal, and which first data shard is stored.

[0200] After receiving the data deduplication response, the client will categorize the first data fragment into two types. One type consists of first data fragments already stored on the storage side. The client does not need to send these first data fragments to the storage side; it only needs to send the metadata of these first data fragments to the storage side. The other type consists of first data fragments not stored on the storage side. The client will send these first data fragments to the storage side. Upon receiving the first data fragments, the storage side will store them.

[0201] For the second data fragment, the client can directly transmit the second data fragment to the storage end, and the storage end can delete the second data fragment and store the deleted second data fragment.

[0202] Scenario 4: Cloud-based deleting scenario.

[0203] The cloud-based deduplication scenario refers to a scenario where data deduplication is implemented in the cloud. The difference between the system architecture of scenario 4 and scenario 3 lies in the deployment of storage in the cloud.

[0204] like Figure 9The diagram shown illustrates another source-side deduplication system provided in this embodiment. This system includes a client and a cloud storage terminal. The client can be understood as the front end of the cloud storage terminal, deployed closer to the user, while the cloud storage terminal is deployed in the cloud. Given the flexible deployment of the cloud storage terminal, multiple cloud nodes are stored within it, each capable of carrying some functions of the deduplication device. For example, these multiple cloud nodes include at least one cloud deduplication check node and at least one cloud storage node. Each cloud deduplication check node corresponds to a deduplication domain and stores the fingerprint index of that domain. The deduplication domain can be understood as an area within the cloud storage node where data is stored. The cloud deduplication check node can query the fingerprint index based on the fingerprint of the received data fragment to determine whether the data fragment has been stored. The cloud deduplication check node also manages its data and can perform deduplication on the data fragments allocated to the deduplication domain. The cloud storage node is a storage device deployed in the cloud for storing data.

[0205] When a client needs to store the first data in the cloud, the client splits the first data into multiple data fragments; the client generates fingerprints of these multiple data fragments, samples the fingerprints of these multiple data fragments, and obtains the fingerprint of the first data fragment.

[0206] The client sends a data deduplication request to one or more cloud deduplication nodes in the cloud storage. The data deduplication request carries the fingerprint of the first data fragment. The data query request is used to inquire whether the first data fragment has been stored in the cloud storage.

[0207] Upon receiving a data deduplication request, the cloud-based deduplication node in the cloud storage terminal queries the fingerprint index of the deduplication domain based on the fingerprint of the first data shard to determine whether a data shard identical to the first data shard is stored. After the query is completed, the cloud storage terminal sends a data deduplication response back to the client. This response indicates whether a data shard identical to the first data shard is stored in the cloud storage terminal, which first data shard is stored, and in which deduplication domain the stored first data shard is located.

[0208] After receiving the data deduplication response, the client will divide the first data fragment into two categories. One category consists of first data fragments already found in the fingerprint index, meaning at least one cloud-based deduplication node has confirmed the presence of the fingerprint for this first data fragment in the fingerprint index. The client does not need to send these first data fragments to the cloud storage; it only needs to send the metadata of these first data fragments to the cloud storage (i.e., the cloud storage node). The other category consists of first data fragments not found in the fingerprint index, meaning each cloud-based deduplication node has confirmed the presence of the fingerprint for this first data fragment in the fingerprint index. The client will allocate a deduplication domain for this type of first data fragment and send it to the cloud-based deduplication node managing that domain. Upon receiving the first data fragment, the cloud-based deduplication node will perform deduplication within that domain and return the deduplicated first data fragment to the client. The client will then store the deduplicated first data fragment on the cloud storage node.

[0209] For the second data shard, the client can determine the deduplication domain of the second data shard based on the deduplication domain of the first data shard, and send the second data shard to the cloud deduplication node managing the deduplication domain. Upon receiving the second data shard, the cloud deduplication node performs deduplication on the second data shard within the deduplication domain. At the cloud deduplication node, there are two possible deduplication results for the second data shard: First, the cloud deduplication node determines, through querying the fingerprint index, that an identical data shard exists within the deduplication domain. In this case, the cloud deduplication node informs the client that deduplication was successful, and the client does not need to send the second data shard to the cloud storage; it only needs to send the metadata of this type of first data shard to the cloud storage (i.e., the cloud storage node). Second, the cloud deduplication node determines, through querying the fingerprint index or by comparing the data within the deduplication domain (i.e., performing data deduplication), that no identical data shard exists within the deduplication domain. In this case, the cloud deduplication node informs the client that deduplication failed, and the client stores the second data shard to the cloud storage node.

[0210] This application also provides, for example Figure 10 The computing device 1000 shown includes a bus 1001, a processor 1002, a communication interface 1003, and a memory 1004. The processor 1002, the memory 1004, and the communication interface 1003 communicate with each other via the bus 1001.

[0211] The processor 1002 can be a central processing unit (CPU) or other specific integrated circuits. The processor 132 can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0212] Memory 1004 can be DRAM. Besides DRAM, memory 1004 can also be other random access memory (such as SRAM). Additionally, memory 1002 can also be ROM. For read-only memory, for example, it can be PROM, EPROM, etc. Memory 1004 can also be flash memory, HDD, or SSD.

[0213] The memory 1004 stores computer program instructions, and the processor 1002 executes these computer program instructions to perform the aforementioned tasks. Figure 3 The steps performed by the deduplication device 100 in the described method. The memory 1004 may also include software modules required by other running processes, such as an operating system (e.g., multiple modules in the deduplication device 100). The operating system may be LINUX. TM UNIX TM WINDOWS TM wait.

[0214] This application also provides a computing device system, the computing device system including at least one such as Figure 11 The computing device 1100 shown includes a bus 1101, a processor 1102, a communication interface 1103, and a memory 1104. The processor 1102, the memory 1104, and the communication interface 1103 communicate with each other via the bus 1101. At least one computing device 1100 in the computing device system communicates with each other via a communication path.

[0215] The specific types of processor 1102 and memory 1104 can be found in the relevant descriptions of processor 1002 and memory 1004, and will not be repeated here. Processor 1102 executes the computer program instructions stored in memory 1104 to perform the aforementioned tasks. Figure 3 The deduplication device 100 executes some or all of the steps. The memory may also include software modules required for other running processes, such as an operating system. The operating system may be Linux.TM UNIX TM WINDOWS TM wait.

[0216] At least one computing device 1100 in the computing device system establishes communication with each other through a communication network, and each computing device 1100 runs any one or any multiple modules of the deduplication device 100.

[0217] The descriptions of the processes corresponding to the above-mentioned figures each have their own emphasis. For parts of a process that are not described in detail, please refer to the relevant descriptions of other processes.

[0218] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented, in whole or in part, in the form of a computer program product. A computer program product includes computer program instructions, which, when loaded and executed on a computer, generate, in whole or in part, the product according to the embodiments of the present invention. Figure 3 The process or function described.

[0219] The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., SSD).

[0220] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A data deduplication method, characterized in that, include: Multiple first data segments are obtained by sampling from multiple data segments of the first data, and the fingerprints of the multiple first data segments are obtained; The deduplication domain of the plurality of first data fragments is determined by using a fingerprint index and the fingerprints of the plurality of first data fragments, wherein the fingerprint index is an index constructed based on the fingerprints of the data fragments in the data set; The deduplication field of the second data fragment is determined based on the deduplication field of the plurality of first data fragments, and the second data fragment is the data fragment in the first data other than the plurality of first data fragments; For any one of the plurality of first data fragments, write the first data fragment into the deduplication field of the first data fragment; Write the second data fragment into the deduplication field of the second data fragment.

2. The method as described in claim 1, characterized in that, The fingerprint index includes multiple key-value pairs, where the key in any key-value pair is the fingerprint of a data shard in the data set, and the value indicates the region where the data shard is located. Determining the deduplication domain of the first data shard using the fingerprint index and the fingerprint of the first data shard includes: The fingerprint index is queried using the fingerprint of the first data shard, and the deduplication field of the first data shard is determined based on the query result.

3. The method as described in claim 2, characterized in that, The query result indicates that the fingerprint of the first data segment was found in the fingerprint index, and the deduplication field of the first data segment is the region indicated by the value in the target key-value pair. The target key-value pair is the key-value pair in the fingerprint index where the key is the fingerprint of the first data segment.

4. The method according to any one of claims 1 to 3, characterized in that, Determining the deduplication field of the second data fragment based on the deduplication fields of the plurality of first data fragments includes: The deduplication field of the second data fragment is determined based on the deduplication fields of the K first data fragments adjacent to the second data fragment in the first data fragment, where K is a positive integer.

5. The method as described in claim 4, characterized in that, The deduplication field of the second data fragment is the deduplication field that appears most frequently among the K neighboring deduplication fields of the first data fragment.

6. The method according to any one of claims 1 to 3, characterized in that, Determining the deduplication field of the second data fragment based on the deduplication fields of the plurality of first data fragments includes: The deduplication field of the second data fragment is determined based on the number of occurrences of the deduplication field of the plurality of first data fragments.

7. The method according to any one of claims 1 to 6, characterized in that, The step of sampling multiple first data fragments from multiple data fragments of the first data and obtaining fingerprints of the multiple first data fragments includes: Generate fingerprints of multiple data segments in the first data, and select the multiple first data segments whose fingerprints have preset features from the multiple data segment fingerprints.

8. The method according to any one of claims 1 to 7, characterized in that, The data set includes data stored in multiple nodes in the storage system, and the node to which the deduplication field of the first data shard belongs or the node to which the deduplication field of the second data shard belongs is one of the multiple nodes.

9. A deduplication device, characterized in that, include: The sampling module is used to sample multiple first data segments from multiple data segments of the first data to obtain multiple first data segments, and to obtain the fingerprints of the multiple first data segments; The allocation module is used to determine the deduplication domain of the plurality of first data fragments using a fingerprint index and the fingerprints of the plurality of first data fragments, wherein the fingerprint index is an index constructed based on the fingerprints of the data fragments in the data set; The deduplication field of the second data fragment is determined based on the deduplication field of the plurality of first data fragments, and the second data fragment is the data fragment in the first data other than the plurality of first data fragments; The data storage module is used to write the first data fragment into the deduplication field of any one of the plurality of first data fragments. For any one of the plurality of second data fragments, the second data fragment is written into the deduplication field of the second data fragment.

10. The apparatus as claimed in claim 9, characterized in that, The fingerprint index includes multiple key-value pairs, where the key in any key-value pair is the fingerprint of a data fragment in the data set, and the value indicates the region where the data fragment is located. The allocation module is used for: The fingerprint index is queried using the fingerprint of the first data shard, and the deduplication field of the first data shard is determined based on the query result.

11. The apparatus as claimed in claim 10, characterized in that, The query result indicates that the fingerprint of the first data segment was found in the fingerprint index, and the deduplication field of the first data segment is the region indicated by the value in the target key-value pair. The target key-value pair is the key-value pair in the fingerprint index where the key is the fingerprint of the first data segment.

12. The apparatus according to any one of claims 9 to 11, characterized in that, The allocation module is used for: The deduplication field of the second data fragment is determined based on the deduplication fields of the K first data fragments adjacent to the second data fragment in the first data fragment, where K is a positive integer.

13. The apparatus as claimed in claim 12, characterized in that, The deduplication field of the second data fragment is the deduplication field that appears most frequently among the K neighboring deduplication fields of the first data fragment.

14. The apparatus according to any one of claims 9 to 11, characterized in that, The allocation module is used for: The deduplication field of the second data fragment is determined based on the number of occurrences of the deduplication field of the plurality of first data fragments.

15. The apparatus according to any one of claims 9 to 14, characterized in that, The data set includes data stored in multiple nodes in the storage system, and the node to which the deduplication field of the first data shard belongs or the node to which the deduplication field of the second data shard belongs is one of the multiple nodes.

16. A computing device, characterized in that, The computing device includes a processor and memory; The memory is used to store computer program instructions; The processor executes computer program instructions in the memory to perform the method as described in any one of claims 1 to 8.

17. A computer-readable storage medium, characterized in that, When the computer-readable storage medium is executed by a computing device, the computing device performs the method of any one of claims 1 to 8.

Images