Multi-node storage system and data deduplication method thereof

Abstract

Embodiments of the present disclosure relate to a multi-node storage system and a method for deduplicating data thereof. The method comprises: determining a similarity hash value of a superblock, wherein the similarity hash value indicates the similarity between superblocks; comparing the similarity hash value of the superblock with a characteristic similarity hash value of a node of the multi-node storage system to determine whether the superblock matches the node; in response to determining that the superblock matches the node, allocating the superblock to the node.

Description

Technical Field

[0001] Embodiments of the present disclosure relate generally to storage systems, and more particularly to methods, devices, and computer-readable storage media for data deduplication in a multi-node storage system. Background Art

[0002] Nowadays, data in not only physical data centers but also cloud data centers are growing explosively, and traditional single-node data deduplication storage systems are no longer sufficient, so they face great challenges in providing reasonable throughput and scalable capacity necessary to meet service level agreements (SLAs). The next generation of cluster storage systems solves this problem by providing a cluster with multiple scalable data deduplication storage system nodes.

[0003] The data size processed by the cluster data deduplication storage system may be at the petabyte (PB), exabyte (EB) or even higher level, so one of the key points to reduce such big data is data deduplication.

[0004] Compared with single-node data deduplication storage systems, data deduplication in clustered data deduplication storage systems faces more challenges. For single nodes, all data is stored and deduplicated as fixed or variable-sized blocks in a single node, so it is natural to deduplicate global data. However, for cluster systems, data deduplication is much more complicated because each node in the cluster is independently performing intra-node data deduplication, and the lack of inter-node data deduplication will significantly reduce the global data deduplication rate, thereby wasting storage space. Summary of the invention

[0005] Embodiments of the present disclosure provide a method, device, computer-readable storage medium, and computer program product for data deduplication in a multi-node storage system.

[0006] In a first aspect, a method for deduplication of data in a multi-node storage system is provided. The method comprises: determining a similarity hash value of a superblock, wherein the similarity hash value indicates the similarity between superblocks; comparing the similarity hash value of the superblock with a characteristic similarity hash value of a node of the multi-node storage system to determine whether the superblock matches the node; and in response to determining that the superblock matches the node, assigning the superblock to the node.

[0007] In a second aspect, an electronic device is provided. The device includes: a processing unit; a memory coupled to the processing unit and including instructions stored thereon, wherein the instructions, when executed by the processing unit, cause the device to perform actions, the actions including: determining a similarity hash value of a superblock, wherein the similarity hash value indicates similarity between superblocks; comparing the similarity hash value of the superblock with a feature similarity hash value of a node of the multi-node storage system to determine whether the superblock matches the node; and in response to determining that the superblock matches the node, assigning the superblock to the node.

[0008] In a third aspect, a computer-readable storage medium is provided, on which machine-executable instructions are stored. When the machine-executable instructions are executed by at least one processor, the at least one processor implements the method according to the first aspect.

[0009] In a fourth aspect, a computer program product is provided, the computer program product being stored on a computer readable medium and comprising machine executable instructions which, when executed, cause a machine to perform the method according to the first aspect.

[0010] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the disclosure, nor is it intended to limit the scope of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and other objects, features and advantages of the present disclosure will become more apparent through a more detailed description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, wherein like reference numerals generally represent like components in the exemplary embodiments of the present disclosure.

[0012] Figure 1 A schematic diagram of a single-node storage system according to some embodiments of the present disclosure is shown;

[0013] Figure 2 A schematic diagram of a multi-node storage system according to some embodiments of the present disclosure is shown;

[0014] Figure 3A and Figure 3B A flow chart of a data deduplication method according to some embodiments of the present disclosure is shown;

[0015] Figure 4-Figure 7 A minimum hashing method according to some embodiments of the present disclosure is shown;

[0016] Fig. 8A A schematic diagram showing a general hash according to some embodiments of the present disclosure;

[0017] Figure 8B A schematic diagram of locality sensitive hashing according to some embodiments of the present disclosure is shown;

[0018] Fig. 9 A schematic diagram of a signature matrix according to some embodiments of the present disclosure is shown;

[0019] Fig.10 A schematic diagram showing a mapping of locality sensitive hashing according to some embodiments of the present disclosure; and

[0020] Fig.11 A schematic block diagram of an example device according to some embodiments of the present disclosure that can be used to implement embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0021] The preferred embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the preferred embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0022] As used herein, the term "including" and its variations mean open inclusion, i.e., "including but not limited to". Unless otherwise stated, the term "or" means "and / or". The term "based on" means "based at least in part on". The terms "an example embodiment" and "an embodiment" mean "at least one example embodiment". The term "another embodiment" means "at least one additional embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0023] To solve the global data deduplication problem of cluster storage systems, a larger block size can be used, for example, a 64KB block size. This provides better data locality and less computation and memory overhead, but also affects redundancy detection. As a result, this solution achieves data deduplication between cluster nodes, but within each node, the data deduplication rate is reduced due to the larger block size.

[0024] In some embodiments, superblock-based stateless data routing can be used to distribute similar data among cluster nodes at a coarse-grained superblock level to preserve data locality. On the other hand, fine-grained block-level data deduplication is performed within each cluster node to achieve high data deduplication efficiency. However, since stateless routing only uses the first 64 bytes from a superblock for routing, it suffers from severe load imbalance in large clusters.

[0025] In some embodiments, a global data deduplication scheme based on similarity hashing is proposed for a cluster storage system. In this scheme, the super block information is compressed into a single hash value, called a "similarity hash value", which retains the similarity relationship of the original super block. For example, compression can be performed by the minimum hash (MinHash) and locality sensitive hashing (LSH) algorithms. The calculated similarity hash value can be used to evaluate the similarity between different super blocks, and super blocks with the same or similar hash values ​​can be routed to the same cluster node. In this way, data can be better utilized, resulting in better duplication rate and balance between cluster nodes.

[0026] Figure 1 FIG. 1 is a schematic diagram of a single-node storage system 100 according to some embodiments of the present disclosure. Figure 1 As shown, the incoming data is first cut into blocks of variable size to form a block data stream 101, where the block size is 4-8KB on average. For each block, a hash value is calculated to uniquely identify the data of the block. The hash value is called the "fingerprint" of the block and has an average size of 20 bytes. In addition, a mapping of the fingerprint to the block data is retained on the hard disk, called a fingerprint index 104. Given a specific fingerprint, the fingerprint index is used to find the corresponding block data 105.

[0027] The size of the fingerprint index 104 grows with the size of the protected storage, so it may be too large to be completely saved in the memory 110. Therefore, the fingerprint cache 103 will remain in the memory, retaining only a portion of the complete fingerprint index.

[0028] For each incoming block data, the deduplication engine 102 first checks whether the fingerprint of the block data exists in the fingerprint cache 103 in the memory 110. If so, the block data will be copied and will not occupy additional space.

[0029] If the fingerprint of the block data does not exist in the fingerprint cache 103, then continue to search whether the fingerprint exists in the fingerprint index on the hard disk storage 120. If yes, the block data is copied and will not occupy additional space; if not, this block data is new data and will occupy additional space on the hard disk storage 120.

[0030] Figure 2 FIG. 2 is a schematic diagram of a multi-node storage system 200 according to some embodiments of the present disclosure. Figure 2 As shown, the multi-node storage system 200 includes nodes 202, 204, and 206, each of which may have Figure 1The single-node storage system 100 shown has the same structure and can implement the same intra-node deduplication mechanism. It should be understood that the multi-node storage system 200 can also include more or fewer nodes, and can be expanded to add more nodes as data increases.

[0031] like Figure 2 As shown, the data stream 210 can be divided into blocks 212, and multiple blocks 212 can be merged into a super block 214. The target storage node in the multi-node cluster is dynamically selected before performing data backup. For example, the similarity hash value of each super block 214 can be calculated to determine which node the super block 214 should be routed to.

[0032] Figure 3A A flowchart of a method 300 for data backup of a cluster storage system according to some embodiments of the present disclosure is shown.

[0033] At block 302, a similarity hash value of the super block is determined, wherein the similarity hash value indicates the similarity between the super blocks. In some embodiments, a signature matrix containing a backup of the super block can be determined by MinHash with the block as a feature or element. Figure 3B This is described in detail in conjunction with block 316. For example, the super block may be sampled to obtain a portion of the blocks in the super block, and the signature matrix of the super block may be determined using the portion of the blocks as features. In this way, the computational efficiency may be improved and the computational time may be reduced.

[0034] In some embodiments, determining the signature matrix of the super block includes: determining a binary feature matrix of the backup, the binary feature matrix indicating whether a corresponding super block among the multiple super blocks of the backup includes a corresponding block; and simplifying the binary feature matrix by minimum hashing to obtain the signature matrix.

[0035] In some embodiments, determining the binary feature matrix includes: determining a feature matrix of the multiple super blocks of the backup; based on the feature matrix, determining a collection of all features of the backup; and determining whether the features in the collection are within the corresponding super block to determine the binary feature matrix.

[0036] In some embodiments, the similarity hash value can be calculated based on the signature matrix by local sensitive hashing. Figure 3B This is described in detail in conjunction with box 318.

[0037] At block 304, the similarity hash value of the superblock is compared to the characteristic similarity hash value of a node of the storage system to determine whether the superblock matches the node.

[0038] In some embodiments, the characteristic similarity hash value of the node includes a similarity hash value range, and the similarity hash value of the super block is compared to see whether it is within the similarity hash value range.

[0039] If it is determined at block 304 that the superblock matches a node, the method 300 proceeds to block 306 to assign the superblock to the node.

[0040] Figure 3B A flowchart of a method 310 for data backup of a cluster storage system according to some embodiments of the present disclosure is shown.

[0041] At block 312, superblocks in a backup are divided into multiple subgroups for parallel computation.

[0042] Usually, we can calculate the similarity between two super blocks by counting how many common blocks they have. However, this solution is impractical due to the high computational and memory overhead. For each comparison, we need to use two nested loops to compare all blocks one by one. Overall, the time complexity of this comparison is O(n 3 ), which is unacceptable.

[0043] For each block in a hyperblock, it can be called the feature of the hyperblock. Therefore, each hyperblock can have multi-dimensional features. In order to obtain a better data deduplication rate, similar hyperblocks should be identified and routed to the same cluster node. However, for a large number of hyperblocks with high-dimensional features, similarity calculation will be a time-consuming task. Traditional nearest neighbor search algorithms (such as KNN, KD tree, and other pairwise comparison methods) are not feasible in this big data situation, because comparing the features of all pairs of hyperblocks is quadratic in number, so the "curse of dimensionality" problem will arise. The delay should be controlled within a relatively low boundary to ensure the throughput efficiency of the cluster system.

[0044] Here, it is proposed to use the hash value as the representation of the super block for super block similarity calculation. The following will describe in detail how to generate the representation of the super block.

[0045] A backup super-block feature matrix H m*n It can be expressed as:

[0046] Table 1

[0047]

[0048] Where X ij is the matrix H m*n elements, and i∈m,j∈n.

[0049] The features of the super block have been constructed as a two-dimensional matrix as shown in Table 1. Here, the matrix is ​​named H m*nFor the matrix H m*n , m is the total number of superblocks, and n is the characteristic size of each superblock.

[0050] In block 314, only selected sample blocks in the super block are selected for calculation. This is because the number of features of a single super block may also be large, and to avoid high computational overhead, the feature matrix only contains selected sample blocks.

[0051] The set V containing all superblock features z It will be created by the union of all superblock features. The total number of features is z.

[0052] V=H1∪H2∪H3∪H i

[0053] Each H i Contains all features of a single backup, where i∈m.

[0054] For example, given the feature matrix shown in Table 1, the complete super-block feature set should merge the features of each super-block with a union operation. The set V of feature matrices in Table 1 z Should be:

[0055] Table 2

[0056] 1fcb78999… f5701d1d… 34e3342a0c… 1bc7e83c5a0… 0826b1c32… 9a096b7a5…

[0057] The next step is to transform the feature matrix H m*n Represented as a Boolean matrix, the size of the matrix will be z*m. The rows of size z represent the z All possible features of . The column of size m represents the number of superblocks.

[0058] For M(r,S) = 1, it means that r is represented in row r and column S only if and only if r is a member of S. If the superblock feature set S contains feature r, the corresponding value in the matrix M will be 1. Usually, the matrix M is a sparse matrix.

[0059] For the feature matrix in Table 1, the corresponding Boolean matrix M z*m Should be:

[0060] Table 3

[0061] Block Hash Super Block 1 Hyperblock 2 Super Block m 1fcb78999… 1 0 … f5701d1d… 1 0 … 34e3342a0c… 1 0 … 1bc7e83c5a0… 0 1 … 0826b1c32… 0 1 … 9a096b7a5… 0 1 …

[0062] In block 316, MinHash is used to reduce the multi-dimensional feature matrix of the superblock. The output is a simplified matrix representing the superblock functionality. For example, MinHash can be used to reduce the dimensionality of the matrix M for further similarity calculations.

[0063] Here is a brief description of the steps performed by MinHash:

[0064] a. Select a random permutation of the rows for matrix M

[0065] b. Define a hash function for set S

[0066] i. h(S) = the index of the first row (in permuted order) where column S has a 1.

[0067] ii. Or

[0068] iii. h(S) = the index of the first element of S in permuted order.

[0069] c. Use k (e.g., k = 100) independent random permutations to create signatures.

[0070] Figure 4-Figure 7 An example of the MinHash algorithm is shown. In this example, sets S1 to S4 each contain features with 7 boolean numbers. Let k = 3, that is, three random permutations are used to calculate the MinHash signature.

[0071] As Figure 4 shown, the rows of the boolean matrix M are randomly permuted. The first "1" in S1 appears in the first row, so h(S1) = 1; the first "1" in S2 appears in the second row, so h(S2) = 2; the first "1" in S3 appears in the first row, so h(S3) = 1; and the first "1" in S4 appears in the second row, so h(S4) = 1. Therefore,[[]] Figure 4 the permutation of [] obtains the signature {1, 2, 1, 2}. Figure 5-Figure 6 Two other different permutations are shown. According to a method similar to Figure 4 the corresponding signatures can be obtained.

[0072] In Figure 7 the signatures obtained are merged together to form a signature matrix, which is a compressed representation of the boolean matrix M. Figure 4-Figure 6 Finally, if h < n, where h represents the number of independent random permutations, then a small signature matrix MH

[0073] will be an equivalent compressed representation of the sparse matrix M. In other words, the signature matrix has a good approximation. h*m By converting the superblock feature matrix into a smaller matrix, similarity calculations can be performed on the signature matrix. For the signature matrix MH

[0074] h*m ​, the smaller h is, the higher the probability of MinHash similarity false positive. In order to control the possibility of false positive, it may still be necessary to select many h. If the vector distance calculation method is used to calculate the similarity of the signature matrix, the "dimensionality curse" problem cannot be completely solved. Assuming that as the size of the hyperblock increases, for 10 6 superblocks, which means 5*10 11 If the computation speed is 1 microsecond per comparison, it will take 6 days to complete the comparison.

[0075] At block 318, a similarity hash value is calculated based on the feature matrix or the signature matrix by locality sensitive hashing (LSH) to uniquely represent the superblock.

[0076] If the goal is to calculate the similarity of each pair, although parallel computing can reduce the time consumed, it can do nothing to reduce the workload. However, usually only the most similar pairs or all pairs above the similarity lower limit are desired. Therefore, it is only necessary to focus on pairs that may be similar, rather than investigating every pair. This problem can be solved by locality sensitive hashing (LSH).

[0077] Fig. 8A and Figure 8B Schematic diagrams of general hashing and locality sensitive hashing (LSH) are shown respectively. Fig. 8A As shown, general hashing maps data to corresponding hash values ​​respectively, and these hash values ​​do not retain the similarity of the original data. On the other hand, Locality Sensitive Hashing (LSH) maps similar data to the same bucket and maps dissimilar data to different buckets, thereby maintaining the similarity of the original data as much as possible.

[0078] Locality Sensitive Hashing (LSH) is an algorithm for solving approximate or exact nearest neighbor searches in high-dimensional spaces. LSH "hashes" items several times to make it more likely that similar items will be hashed into the same bucket and thus become candidate pairs. In this way, it is not necessary to calculate the similarity of each pair, but only to check whether the candidate pairs are similar.

[0079] Fig. 9 and Fig.10 FIG. 4 is a schematic diagram of locality sensitive hashing according to some embodiments of the present disclosure. Fig. 9As shown, the signature matrix 900 can be divided into b bands, each of which consists of r rows. For each band, a hash function can be set, the input of which is a vector of r integers in this band (also called a mini signature), and the function hashes the vector into the corresponding bucket. Therefore, for each signature, b hashes can be performed. Pairs with at least one band hashed to the same bucket can be considered as candidate pairs. It should be noted that the same hash function can be used for all bands, but different bands use different bucket arrays, so that different bands will not hash to the same bucket.

[0080] like Fig.10 As shown, hash table 902 includes four buckets, however, in general, hash table 902 may include a large number of buckets. For a band of signature matrix 900, mini signatures 2 and 6 are mapped to the same bucket. Therefore, signatures 2 and 6 can be considered as candidate pairs.

[0081] According to an embodiment of the present disclosure, LSH can generate an integer hash value, and for similar inputs, the hash values ​​are also similar. Therefore, the hash value generated by LSH is also called a similarity hash value. All MinHash values ​​in each band can be calculated one by one into the LSH hash table through LSH. For the incoming super block with MinHash, the LSH hash table can be used to obtain similar backups with a time complexity of O(1). In addition, false negatives can be reduced by adjusting variables b and r according to the similarity requirements.

[0082] Return now Figure 3B , at block 320 , it is determined whether the similarity hash value matches a distributed hash table (DHT).

[0083] A distributed hash table (DHT) is a type of distributed system that provides a lookup service similar to a hash table, except that the mapping from keys to values ​​is distributed across nodes. This allows DHT to be extended to a very large number of nodes, such as a clustered data deduplication storage system. Alternatively, a distributed hash table (DHT) can also be implemented using a centralized hash table.

[0084] DHT is used to record the similarity hash value range and target node ID pair {similarity-hash-range,target-node-ID} to speed up the determination of the target node. Any hyperblock whose similarity hash value falls within the similarity hash value range will be routed to the specified target node ID. This is because hyperblocks with similarity hash values ​​in the same range have the possibility of sharing more common blocks.

[0085] By specifying an appropriate similarity hash range value and using the DHT as a cache, the process of determining the target node can be accelerated.

[0086] If it is determined at block 320 that the similarity hash value matches the DHT, the method 310 proceeds to block 322 to retrieve the matching target node ID from the DHT. Then, at block 330, the superblock is routed to the target node ID specified in the DHT.

[0087] If it is determined at block 320 that no match is found for the similarity hash value in the DHT, the method 310 proceeds to block 324 to calculate the target node ID based on the similarity hash value. For example, the target node ID may be calculated using the following equation, where N represents the number of cluster nodes.

[0088] Target node ID = (similarity hash value) / 1000%N

[0089] First, the similarity hash value is divided by 1000 to filter out the effect of low-order digits. Then, the quotient is modulo N to find the specific target node ID.

[0090] At block 326, for a given similarity hash value, calculate the similarity hash value ±<range_number> Get the neighbor range around it, where range_number is an adjustable parameter that depends on different data patterns and cluster environments. For example, range_number can be set to 10000.

[0091] At block 328, the neighbor range is updated into the DHT along with the target node ID as {similarity-hash-range, target-node-ID}.

[0092] Please note that after the DHT record is updated, there may be some overlap in the similarity hash value ranges, so the similarity hash value ranges can be merged and modified through the background post-processing of DHT.

[0093] Then, at block 330, the superblock is routed to the calculated target node ID.

[0094] For backup, determining the backup target is very critical, and using method 310, the time complexity is O(1). This means that no matter how many backups there are, each backup has multiple super blocks, the query time is basically a relatively small constant.

[0095] Generally speaking, it is impossible to achieve a balance in the effect of data deduplication within a single node and between multiple cluster nodes, and they must compromise between the block granularity and the cluster size. According to an embodiment of the present disclosure, MinHash and LSH can be used to represent and calculate the similarity at the super block level, and the similarity hash value can be routed to the corresponding storage node using DHT. Therefore, a good balance of data deduplication effects is achieved within a single node and between multiple cluster nodes, and the scalability of the storage cluster is also improved.

[0096] Fig.11 1 shows a schematic block diagram of a device 1100 that can be used to implement an embodiment of the present disclosure. Fig.11 As shown, the device 1100 includes a central processing unit (CPU) 1101, which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 1102 or loaded from a storage unit 1108 to a random access memory (RAM) 1103. In the RAM 1103, various programs and data required for the operation of the device 1100 can also be stored. The CPU 1101, the ROM 1102, and the RAM 1103 are connected to each other via a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0097] A number of components in the device 1100 are connected to the I / O interface 1105, including: an input unit 1106, such as a keyboard, a mouse, etc.; an output unit 1107, such as various types of displays, speakers, etc.; a storage unit 1108, such as a disk, an optical disk, etc.; and a communication unit 1109, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1109 allows the device 1100 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0098] The various processes and processing described above, such as methods 300 and 310, may be performed by the processing unit 1101. For example, in some embodiments, methods 300 and 310 may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 1108. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1100 via the ROM 1102 and / or the communication unit 1109. When the computer program is loaded into the RAM 1103 and executed by the CPU 1101, one or more steps of the methods 300 and 310 described above may be performed.

[0099] The present disclosure may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present disclosure.

[0100] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0101] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0102] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0103] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0104] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0105] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0106] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0107] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for deduplicating data in a multi-node storage system, comprising: receiving a backup data stream; dividing the data stream into a plurality of blocks for the backup; Merging a plurality of the blocks into the backup super block; Determine a similarity hash value of the super block, wherein the similarity hash value indicates a similarity between super blocks, wherein determining the similarity hash value of the super block comprises: determining a signature matrix comprising the backup of the superblock by characterizing the blocks by minimum hashing, wherein the signature matrix is ​​determined by sampling the superblock to obtain partial blocks in the superblock, and determining the signature matrix for the superblock by characterizing the partial blocks; and Based on the signature matrix, calculating the similarity hash value by local sensitive hashing; comparing the similarity hash value of the superblock with a similarity hash value range using a distributed hash table (DHT) to determine whether the similarity hash value is within the similarity hash value range, the similarity hash value range being associated with a node of the multi-node storage system, wherein the similarity hash value range is stored in the DHT, and wherein the DHT is used as a cache; Determining that the similarity hash value is outside the similarity hash value range; Based on determining that the similarity hash value is outside the similarity hash value range, calculating an identification of the node based on the similarity hash value, and assigning the super block to the node based on the identification of the node; updating the similarity hash value range by adding a range number to the similarity hash value to set an upper bound of the similarity hash value range and subtracting the range number from the similarity hash value to set a lower bound of the similarity hash value range; and Based on assigning the superblock to the node and in response to updating the similarity hash value range, the superblock is stored in a memory of the node.

2. An electronic device, comprising: Processing unit; a memory coupled to the processing unit and comprising instructions stored thereon, the instructions, when executed by the processing unit, causing the electronic device to perform a method, the method comprising: receiving a backup data stream; dividing the data stream into a plurality of blocks for the backup; Merging a plurality of the blocks into the backup super block; Determine a similarity hash value of the super block, wherein the similarity hash value indicates a similarity between super blocks, wherein determining the similarity hash value of the super block comprises: determining a signature matrix comprising the backup of the superblock by characterizing the blocks by minimum hashing, wherein the signature matrix is ​​determined by sampling the superblock to obtain partial blocks in the superblock, and determining the signature matrix for the superblock by characterizing the partial blocks; and Based on the signature matrix, calculating the similarity hash value by local sensitive hashing; comparing the similarity hash value of the superblock with a similarity hash value range using a distributed hash table (DHT) to determine whether the similarity hash value is within the similarity hash value range, the similarity hash value range being associated with a node of a multi-node storage system, wherein the similarity hash value range is stored in the DHT, and wherein the DHT is used as a cache; Determining that the similarity hash value is outside the similarity hash value range; Based on determining that the similarity hash value is outside the similarity hash value range, calculating an identification of the node based on the similarity hash value, and assigning the super block to the node based on the identification of the node; updating the similarity hash value range by adding a range number to the similarity hash value to set an upper bound of the similarity hash value range and subtracting the range number from the similarity hash value to set a lower bound of the similarity hash value range; and Based on assigning the superblock to the node and in response to updating the similarity hash value range, the superblock is stored in a memory of the node.

3. A computer-readable storage medium having machine-executable instructions stored thereon, which, when executed by at least one processor, causes the at least one processor to implement a method, the method comprising: receiving a backup data stream; dividing the data stream into a plurality of blocks for the backup; Merging a plurality of the blocks into the backup super block; Determine a similarity hash value of the super block, wherein the similarity hash value indicates a similarity between super blocks, wherein determining the similarity hash value of the super block comprises: determining a signature matrix comprising the backup of the superblock by characterizing the blocks by minimum hashing, wherein the signature matrix is ​​determined by sampling the superblock to obtain partial blocks in the superblock, and determining the signature matrix for the superblock by characterizing the partial blocks; and Based on the signature matrix, calculating the similarity hash value by local sensitive hashing; comparing the similarity hash value of the superblock with a similarity hash value range using a distributed hash table (DHT) to determine whether the similarity hash value is within the similarity hash value range, the similarity hash value range being associated with a node of a multi-node storage system, wherein the similarity hash value range is stored in the DHT, and wherein the DHT is used as a cache; Determining that the similarity hash value is outside the similarity hash value range; Based on determining that the similarity hash value is outside the similarity hash value range, calculating an identification of the node based on the similarity hash value, and assigning the super block to the node based on the identification of the node; updating the similarity hash value range by adding a range number to the similarity hash value to set an upper bound of the similarity hash value range and subtracting the range number from the similarity hash value to set a lower bound of the similarity hash value range; and Based on assigning the superblock to the node and in response to updating the similarity hash value range, the superblock is stored in a memory of the node.

Images