Information processing device, duplication removal method, and duplication removal program

Abstract

To reduce a size of a duplicate index and prevent a decrease in a duplication removal rate by performing duplicate management in segment units.SOLUTION: A similarity determination unit calculates a finger print of a segment newly written into a disk 2a using a hash function for similarity determination. The similarity determination unit determines whether the segment newly written into the disk 2a is similar to any of segments stored in the disk 2a using the calculated finger print. When the similarity determination unit determines that there is a segment similar to the segment newly written into the disk 2a, a difference processing unit calculates a difference with the similar segment, and writes the calculated difference into the disk 2a.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing device, a duplication removal method, and a duplication removal program. [Background technology]

[0002] Composable architecture is being considered as an ICT (Information and Communication Technology) infrastructure architecture. In composable architecture, compute nodes and storage nodes can be virtualized and managed in an integrated manner, and compute nodes and storage nodes can be scaled out independently. In composable architecture, the storage node side is JBOD (Just a Bunch Of Disks) / JBOF (Just a Bunch Of Flash), and storage functions such as deduplication are realized on the compute node side. Therefore, composable architecture realizes a storage node that is inexpensive and can be scaled out.

[0003] Fig. 13 is a diagram for explaining a composable architecture. As shown in Fig. 13, a compute node 91 and a storage node 92 are managed in an integrated manner. In the composable architecture, a RAM (Random Access Memory) 91a of the compute node 91 is used for compute, deduplication, and other storage functions. If a large amount of RAM 91a is used for deduplication, the amount of RAM 91a used by the compute node 91 decreases. For this reason, a deduplication method with high RAM usage efficiency is required in the composable architecture.

[0004] FIG. 14 is a diagram for explaining deduplication. As shown in FIG. 14, write data is divided into chunks of a predetermined size, and a hash value called a fingerprint is calculated for each chunk. Then, an index that associates the storage location of the chunk with the fingerprint is referenced, and if there is an identical fingerprint, the chunk is a duplicate and is not written to the disk 92a. On the other hand, if there is no identical fingerprint, the chunk is not a duplicate and is written to the disk. Note that the disk 92a is included in the storage node 92. Furthermore, when a new chunk is written to the disk 92a, the storage location of the chunk and the fingerprint of the chunk are registered in the index.

[0005] In FIG. 14, the write data consists of chunks A, B, C', and D, and the fingerprints of A, B, C, and D are registered in the index, so a new chunk C' is written to disk 92a.

[0006] Since the index is stored in the RAM 91a, it is possible to make the index smaller in order to achieve deduplication with high RAM usage efficiency. For example, if the chunk size is 4 KB (kilobytes) and there is 10 TB (terabytes) of unique data, 40 B (bytes) of metadata per chunk is required for the index. In this case, the capacity of the index is 40 x (10 x 10 12 ÷(4×10 3 ))=10 11 B=100GB.

[0007] Therefore, it is possible to reduce the size of the index by performing deduplication for each segment that groups together multiple chunks. For example, if the segment size is 64 KB, the capacity of the index can be reduced to 1 / 16, or 6.25 GB. By reducing the capacity of the RAM 91a used as the index, it is possible to increase the capacity of the RAM 91a used as a cache, for example, and improve the access performance to the storage node 92.

[0008] As a conventional technique for backup, there is a storage control device capable of reducing the processing load of differential backup on a file-by-file basis. This storage control device copies all files included in a first snapshot created by a copy-on-write method to a backup area. After that, this storage control device creates a second snapshot. Then, this storage control device identifies areas M1 and M2 that have been updated after the creation of the first snapshot among metadata areas in the second snapshot in which metadata related to all files included in the second snapshot is recorded. This storage control device identifies areas M1 and M2 based on management information for managing the positions of data blocks updated after the creation of the first snapshot. Then, this storage control device identifies files F1 and F2 that have been updated after the creation of the first snapshot based on the metadata included in areas M1 and M2, and copies them to the backup area.

[0009] Furthermore, a conventional technique for deduplication is a storage control device that first compares hash codes to narrow down the data to be compared and quickly detects duplicate data. This storage control device sets a hash code to data received from a host. Data with a hash code is stored in a logical volume. The storage control device then compares the hash codes of each data to be compared. If the hash codes match, the storage control device compares the data to determine whether it is duplicate data. If duplicate data is detected, the storage control device eliminates the duplicate data. [Prior art documents] [Patent documents]

[0010] [Patent Document 1] US Patent Application Publication No. 2019 / 0250818 [Patent Document 2] JP 2018-028715 A [Patent Document 3] JP 2009-251725 A Summary of the Invention [Problem to be solved by the invention]

[0011] There is a problem in that the deduplication rate decreases when deduplication is performed for each segment. Fig. 15 is a diagram for explaining the decrease in the deduplication rate when deduplication is performed for each segment. As shown in Fig. 15, when application 3 instructs writing segment A consisting of chunks a, b, c, and d to disk 92a, fingerprint A of segment A is calculated and the duplication index is referenced. Since fingerprint A is not in the duplication index, segment A is written to disk 92a, and fingerprint A and the storage destination are registered in the duplication index.

[0012] After that, chunk c of segment A is updated to chunk c', and application 3 instructs writing segment B consisting of chunks a, b, c', and d to disk 92a. Then, fingerprint B of segment B is calculated, and the duplicate index is referenced. Since fingerprint B is not in the duplicate index, segment B is written to disk 92a, and fingerprint B and the storage destination are registered in the duplicate index.

[0013] In this way, although segment B differs only partially from segment A, the entirety of segment B is written to disk 92a, so chunk a, chunk b, and chunk d are written in duplicate to disk 92a.

[0014] In one aspect, the present invention aims to improve the deduplication rate and make effective use of storage resources when deduplication is performed for each unit that groups together a predetermined number of chunks, such as a segment. [Means for solving the problem]

[0015] In one embodiment, the information processing device includes a similarity determination unit and a writing unit. The similarity determination unit determines whether or not similar data similar to write data of a first size to a storage device is present in the storage device. When the similarity determination unit determines that similar data is present, the writing unit generates storage data based on the write data and the similar data, and writes the generated storage data to the storage device. Effect of the Invention

[0016] In one aspect, the present invention can improve the deduplication rate and make effective use of storage resources. [Brief description of the drawings]

[0017] [Figure 1] FIG. 1 is a diagram for explaining similarity detection by a compute node according to an embodiment. [Diagram 2] FIG. 2 is a diagram illustrating a configuration of a compute node according to the embodiment. [Diagram 3] FIG. 3 is a diagram illustrating an example of a duplicate index. [Figure 4] FIG. 4 is a diagram illustrating an example of the similarity index. [Diagram 5] FIG. 5 is a diagram for explaining a hash function for removing duplicates and a hash function for determining similarity. [Figure 6] FIG. 6 is a diagram showing an example of logical-physical address mapping information. [Figure 7] FIG. 7 is a diagram for explaining a method of calculating the difference. [Figure 8] FIG. 8 is a flowchart showing the flow of the write process. [Figure 9] FIG. 9 is a flowchart showing the flow of the mapping information update process. [Figure 10] FIG. 10 is a diagram comparing the conventional overlap management with the overlap management according to the embodiment. [Figure 11]FIG. 11 is a diagram for explaining a method of calculating the exclusive OR of a new segment and a similar segment, compressing the result, and then writing the result to a disk. [Figure 12] FIG. 12 is a diagram showing an example of logical-physical address mapping information when an exclusive OR is taken between a new segment and a similar segment. [Figure 13] FIG. 13 is a diagram for explaining the composable architecture. [Figure 14] FIG. 14 is a diagram for explaining the overlap removal. [Figure 15] FIG. 15 is a diagram for explaining a decrease in the duplication removal rate when duplication removal is performed for each segment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] Hereinafter, an information processing device, a method for removing duplicates, and a program for removing duplicates disclosed in the present application will be described in detail with reference to the accompanying drawings. Note that the disclosed technology is not limited to the embodiments. EXAMPLES

[0019] First, similarity detection by a compute node (information processing device) according to the embodiment will be described. Fig. 1 is a diagram for explaining similarity detection by a compute node according to the embodiment. In Fig. 1, it is assumed that a segment A consisting of chunks a, b, c, and d has already been written to a disk 2a. As shown in Fig. 1, an application 3 instructs writing of a segment B consisting of chunks a, b, c', and d to a disk 2a (t1).

[0020] Then, the compute node according to the embodiment calculates a fingerprint B for deduplication and refers to the duplication index. Then, since fingerprint B is not in the duplication index, the compute node according to the embodiment calculates a fingerprint for similarity detection and refers to the similarity index. Here, a fingerprint for similarity detection is a fingerprint whose value is the same for similar segments. Since segment B is similar to segment A, fingerprint A is calculated as the fingerprint for similarity detection of segment B.

[0021] Then, the compute node according to the embodiment identifies that there is a segment A similar to segment B because segment B has the same fingerprint for detecting similarity as segment A (t2). Then, the compute node according to the embodiment reads the similar segment A from disk 2a, calculates the difference between the similar segment A and writes the difference to disk 2a (t3). In Fig. 1, the difference is chunk c', so chunk c' is written to disk 2a.

[0022] In this way, the compute node of the embodiment uses the similarity index to identify existing similar segments that are similar, and writes the differences between the identified similar segments to the disk 2a, thereby preventing a decrease in the duplicate elimination rate.

[0023] Next, the configuration of a compute node according to the embodiment will be described. Fig. 2 is a diagram showing the configuration of a compute node according to the embodiment. As shown in Fig. 2, a compute node 1 according to the embodiment includes a RAM 1a, a CPU (Central Processing Unit) 1b, an I / F 1c, an ODD (Optical Disk Drive) 1d, and a LAN (Local Area Network) interface 1e.

[0024] The RAM 1a is a memory that stores programs, data necessary for executing the programs, etc. The RAM 1a stores a duplication index 11, a similarity index 12, logical-physical address mapping information 13, and cache data 14.

[0025] The duplicate index 11 is an index used for deduplication for each segment. FIG. 3 is a diagram showing an example of the duplicate index 11. As shown in FIG. 3, the duplicate index 11 associates a segment with a fingerprint and a constituent chunk physical address. A segment is a unit of data for which deduplication is performed. A fingerprint is a hash value of a segment. A constituent chunk physical address is a physical address of the disk 2a where the chunks constituting the segment are stored.

[0026] For example, the fingerprint of a segment consisting of chunk a, chunk b, chunk c, and chunk d is "0xdeadbeef", where "0x" represents a hexadecimal number, and the physical addresses of chunk a, chunk b, chunk c, and chunk d are "200", "201", "202", and "203", respectively.

[0027] When a segment consisting of chunks a, b, c, and d exists and a segment consisting of chunks a, b, c', and d is written, only chunk c' is written. For chunks a, b, and d, the physical addresses that already exist are referenced.

[0028] The similarity index 12 is an index used to identify similar segments. FIG. 4 is a diagram showing an example of the similarity index 12. As shown in FIG. 4, the similarity index 12 associates a segment with a fingerprint and a constituent chunk physical address. A segment is a unit of data for which similarity determination is performed. A fingerprint is a hash value of a segment. A constituent chunk physical address is a physical address of the disk 2a where the chunks constituting the segment are stored.

[0029] For example, the fingerprint of a segment consisting of chunk a, chunk b, chunk c, and chunk d is "0xbaadf00d." Furthermore, the physical addresses of chunk a, chunk b, chunk c, and chunk d are "200," "201," "202," and "203," respectively.

[0030] FIG. 5 is a diagram for explaining hash functions for deduplication and hash functions for similarity judgment. FIG. 5(a) shows the values ​​taken by the hash function for deduplication, and FIG. 5(b) shows the values ​​taken by the hash function for similarity judgment. In FIG. 5, A and B are similar data. As shown in FIG. 5(a), a hash function is used for deduplication that is not biased in the values ​​generated, does not generate close values ​​for input of similar data, and is unlikely to cause collisions (outputting the same value for different inputs). For example, a SHA-1 function is used as a hash function for deduplication. In FIG. 5(a), the hash values ​​of A and B are completely different.

[0031] On the other hand, as shown in Figure 5(b), a hash function is used to judge similarity, which takes the same value for similar inputs and a different value otherwise. An example of a definition of similarity is the Hamming distance. For example, minHash, a type of LSH (Locality Sensitive Hashing), is used as a hash function for similarity judgment. In Figure 5(b), the hash values ​​of A and B are the same.

[0032] Returning to FIG. 2, the logical-physical address mapping information 13 is information that associates logical addresses with physical addresses for chunks. FIG. 6 is a diagram showing an example of the logical-physical address mapping information 13. As shown in FIG. 6, the logical-physical address mapping information 13 associates chunks with logical addresses and physical addresses. A chunk is data that is associated with a logical address and a physical address. A logical address is an address at which a chunk is stored in application 3. A physical address is an address at which a chunk is stored on disk 2a. For example, the logical address of chunk a is "100" and the physical address is "200".

[0033] The cache data 14 is a portion of the data stored in the disk 2a. The cache data 14 is used to speed up access to the data stored in the disk 2a.

[0034] The CPU 1b executes the deduplication program stored in the RAM 1a to realize a duplication determination unit 21, a similarity determination unit 22, a difference processing unit 23, and an update unit 24. The deduplication program is stored in a DVD, which is an example of a recording medium readable by the compute node 1, and is read from the DVD by the ODD 1d and installed on the disk 2a. Alternatively, the deduplication program is stored in a database or the like of a computer system connected via the LAN interface 1e, and is read from the database and installed on the disk 2a. The installed deduplication program is then read from the disk 2a to the RAM 1a and executed by the CPU 1b.

[0035] The duplication determination unit 21 calculates the fingerprint of the new segment using a hash function for deduplication, and determines whether the new segment overlaps with any of the segments stored in the disk 2a using the duplication index 11. Here, the new segment is a segment that is newly written to the disk 2a.

[0036] The similarity determination unit 22 calculates the fingerprint of the new segment using a hash function for similarity determination, and determines, using the similarity index 12, whether the new segment is similar to any of the segments stored on the disk 2a.

[0037] The difference processing unit 23 calculates the difference between the new segment and the similar segment, and writes the calculated difference to the disk 2a. The difference processing unit 23 breaks the segment into chunks, and judges whether or not there is a match on a chunk-by-chunk basis. Then, the difference processing unit 23 identifies, as a difference, a chunk that does not match an existing chunk among the chunks that constitute the new segment. The difference processing unit 23 judges whether or not the chunks match by using a hash value. For example, the difference processing unit 23 uses SHA-1, md5, or the like as a hash function. Note that the difference processing unit 23 may judge whether or not the chunks match without using a hash value. Also, the difference processing unit 23 corresponds to a writing unit in the claims.

[0038] Fig. 7 is a diagram for explaining a method of calculating a difference. As shown in Fig. 7, when a new segment is composed of chunk a, chunk b, chunk c', and chunk d, and a written segment is composed of chunk a, chunk b, chunk c, and chunk d, the difference processing unit 23 identifies chunk c' as the difference.

[0039] If the new segment is a duplicated segment, the update unit 24 adds to the logical-physical address mapping information 13 the logical addresses of the chunks constituting the new segment and the physical addresses of the corresponding chunks that have already been written.

[0040] Furthermore, if the new segment is a similar segment, the updating unit 24 updates the logical-physical address mapping information 13 as follows. That is, the updating unit 24 adds to the logical-physical address mapping information 13 the logical addresses of the chunks constituting the new segment and the physical addresses of the corresponding chunks that have already been written or the chunks that have been newly written to the disk 2a.

[0041] In addition, when a new segment is written as a new segment that is neither a duplicate segment nor a similar segment, the update unit 24 adds the logical address of the written chunk and the physical address of the written chunk to the logical-physical address mapping information 13.

[0042] Furthermore, the update unit 24 updates the duplication index 11 and the similarity index 12 if the new segment is a segment other than a duplicate segment.

[0043] The I / F 1c is an interface that connects the compute node 1 to the disk 2a. The disk 2a stores logical-physical address mapping information 31 and data 32. The logical-physical address mapping information 31 is read into the RAM 1a and stored as logical-physical address mapping information 13. The updated logical-physical address mapping information 13 is stored in the disk 2a as the logical-physical address mapping information 31.

[0044] Next, the flow of the write process will be described. Fig. 8 is a flowchart showing the flow of the write process. As shown in Fig. 8, the compute node 1 stores an input chunk in a write buffer (step S1), and when a certain size (segment size) of the input chunk is stored, the input chunk is segmented (step S2).

[0045] Then, the compute node 1 calculates a hash value for deduplication and a hash value for similarity determination for each segment (step S3), and searches the duplicate index 11 using the hash value for deduplication to determine whether or not there is a duplicate segment (step S4). If there is a duplicate segment, the compute node 1 proceeds to step S8.

[0046] On the other hand, if there is no overlapping segment, the compute node 1 determines whether there is a similar segment (step S5), and if there is no similar segment, writes it as a new segment (step S6).On the other hand, if there is a similar segment, the compute node 1 calculates the difference from the similar segment and writes only the difference (step S7).

[0047] Then, the compute node 1 performs a mapping information update process for updating the logical-physical address mapping information 13 and the like (step S8).

[0048] In this way, when a similar segment is found, the compute node 1 calculates the difference between the similar segment and the new segment and writes only the difference to the disk 2a, thereby preventing a decrease in the deduplication rate.

[0049] Fig. 9 is a flowchart showing the flow of the mapping information update process. As shown in Fig. 9, the compute node 1 judges whether the new segment is an overlapping segment (step S11). If it is an overlapping segment, the compute node 1 adds the logical address of the chunk constituting the overlapping segment and the physical address of the existing corresponding chunk to the logical-physical address mapping information 13 (step S16).

[0050] On the other hand, if it is not a duplicate segment, the compute node 1 updates the duplicate index 11 and the similar index 12 (step S12), and determines whether or not only the difference has been written as a similar segment (step S13). If only the difference has not been written, the compute node 1 adds the logical address of the written chunk and the physical address of the written chunk to the logical-physical address mapping information 13 (step S14).

[0051] On the other hand, if only the difference is written, the compute node 1 adds the logical address of the written chunk and the physical address of the existing corresponding chunk or the newly written chunk to the logical-physical address mapping information 13 (step S15).

[0052] In this way, the compute node 1 can appropriately manage writing to the disk 2a by using the duplicate index 11 and similarity index 12 in segment units and the logical-physical address mapping information 13 in chunk units.

[0053] Fig. 10 is a diagram comparing conventional duplicate management with duplicate management according to an embodiment. Fig. 10(a) shows conventional duplicate management, and Fig. 10(b) shows duplicate management according to an embodiment. As shown in Fig. 10, when an application 3 updates some chunks in a segment, conventionally both the segment before the update and the segment after the update are stored on disk 2a, but in the embodiment, only the segment before the update and the difference are stored.

[0054] 10, among chunk a, chunk b, chunk c, and chunk d, chunk c is updated to chunk c'. Conventionally, a segment consisting of chunk a, chunk b, chunk c, and chunk d, and a segment consisting of chunk a, chunk b, chunk c', and chunk d are stored on disk 2a. On the other hand, in the embodiment, a segment consisting of chunk a, chunk b, chunk c, and chunk d, and chunk c' are stored on disk 2a.

[0055] Therefore, compared to conventional duplicate management, the duplicate management according to the embodiment can improve the duplicate removal rate, and is more effective for a workload with many partial updates.

[0056] As described above, in the embodiment, the similarity determination unit 22 calculates the fingerprint of the segment to be newly written to the disk 2a by using a hash function for similarity determination. Then, the similarity determination unit 22 determines whether the segment to be newly written to the disk 2a is similar to any of the segments stored in the disk 2a by using the calculated fingerprint and the similarity index 12. Then, when the similarity determination unit 22 determines that there is a segment similar to the segment to be newly written to the disk 2a, the difference processing unit 23 calculates the difference from the similar segment and writes the calculated difference to the disk 2a. Therefore, the compute node 1 can reduce the size of the duplicate index 11 and prevent a decrease in the duplicate removal rate by performing duplicate management on a segment basis. Therefore, the compute node 1 can effectively utilize storage resources.

[0057] Also, in the embodiment, a segment is composed of multiple chunks, and the differential processing unit 23 determines whether or not there is duplicate data for each chunk, and calculates the differential based on the presence or absence of duplicate data for each chunk, so that the differential can be calculated appropriately.

[0058] In the embodiment, the similarity index 12 associates a segment, a fingerprint, and a constituent chunk physical address, and the logical-physical address mapping information 13 associates a logical address and a physical address for the chunk. Then, when there is a similar segment similar to the new segment, the update unit 24 updates the similarity index 12 and the logical-physical address mapping information 13. Therefore, the compute node 1 can appropriately manage the similar segments.

[0059] Incidentally, instead of taking the difference by comparing chunk units, the difference processing unit 23 may take the exclusive OR of the new segment and the similar segment, compress it, and then write it to the disk 2a. Figure 11 is a diagram for explaining a method of taking the exclusive OR of the new segment and the similar segment, compressing it, and then writing it to the disk 2a. Figure 11(a) shows the time of writing, and Figure 11(b) shows the time of reading.

[0060] As shown in Fig. 11(a), when writing, the differential processing unit 23 performs an exclusive OR operation on the new segment and the similar segment, compresses it, and writes it to the disk 2a. In Fig. 11(a), "xxyyzz" is the compressed data. When reading, the compute node 1 expands the compressed data, performs an exclusive OR operation on the similar segment, and sets it as read data.

[0061] When the exclusive OR of the new segment and the similar segment is taken, the compute node 1 manages data in units of segments instead of managing data in units of chunks. Fig. 12 is a diagram showing an example of logical-physical address mapping information 13 when the exclusive OR of the new segment and the similar segment is taken.

[0062] 12, the logical-physical address mapping information 13 associates a segment with a logical address, a physical address, a size, and a similar segment P. A segment is data in which a logical address and a physical address are associated with each other. A logical address is an address at which a segment is stored in an application 3.

[0063] The physical address is the address where the segment is stored on disk 2a if there is no similar segment. If there is a similar segment, the physical address is the address where the XORed compressed data is stored on disk 2a. The size is the size of the segment or compressed data stored on disk 2a. The similar segment P is a pointer to the similar segment. If there is no similar segment, the similar segment P is "null".

[0064] For example, for segment abcd, the logical address is "100-103", the physical address is "200", the size is "8kB", and there is no similar segment. For segment abc'd, the logical address is "104-107", the physical address is "202", the size of the compressed data is "4kB", and the similar segment is abcd.

[0065] In this way, the compute node 1 can manage the difference with the similar segment by compressing the new segment by taking the exclusive OR of the new segment and the similar segment, instead of taking the difference by comparing chunk units. [Explanation of symbols]

[0066] 1,91 Compute nodes 1a,91a RAM 1b CPU 1c Interface 1d ODD 1e LAN interface 2a,92a Disc 3. Applications 11 Duplicate Indexes 12 Similarity Index 13 Logical-physical address mapping information 14 Cached Data 21 Duplication determination section 22 Similarity determination section 23 Differential Processing Section 24 Update section 31 Logical-physical address mapping information 32 Data 92 Storage Nodes

Claims

1. a similarity determination unit that, when it is determined that identical data identical to data written to a storage device is not present in the storage device using a first hash function that takes different values ​​for data that are similar to each other, determines whether or not similar data similar to the write data is present in the storage device using a second hash function that is different from the first hash function and that takes the same values ​​for data that are similar to each other, based on a Hamming distance; a write unit that writes a difference between the write data and the similar data or an exclusive OR of the write data and the similar data into the storage device when the similarity determination unit determines that the write data and the similar data having the same value as the value of the second hash function are present in the storage device; An information processing device having the above configuration.

2. The write data is divided into a plurality of chunks; the similarity determination unit determines whether or not there is duplicate data that overlaps with the write data for each of the chunks, and calculates the difference based on the presence or absence of duplicate data for each of the chunks. The information processing device according to claim 1 .

3. a storage unit that associates the data, a fingerprint used for determining similarity of the data, and a physical address of a chunk included in the data, and stores the associated data as a similarity index, and also associates the chunk, a logical address, and a physical address of the chunk, and stores the associated data as logical-physical information; an update unit that updates the similarity index and the logical-physical information based on the write data when the similarity determination unit determines that the similar data exists; The information processing apparatus according to claim 2 , further comprising:

4. The computer when it is determined that the storage device does not contain identical data that is identical to the write data to the storage device using a first hash function that takes different values ​​for similar data, whether or not the storage device contains similar data that is similar to the write data is determined based on a Hamming distance using a second hash function that is different from the first hash function and that takes the same values ​​for similar data; when it is determined that the similar data having the same value as the write data and the second hash function is present in the storage device, a difference between the write data and the similar data or an exclusive OR of the write data and the similar data is written to the storage device; The deduplication method to perform the process.

5. On the computer, when it is determined that the storage device does not contain identical data that is identical to the write data to the storage device using a first hash function that takes different values ​​for similar data, whether or not the storage device contains similar data that is similar to the write data is determined based on a Hamming distance using a second hash function that is different from the first hash function and that takes the same values ​​for similar data; when it is determined that the similar data having the same value as the write data and the second hash function is present in the storage device, a difference between the write data and the similar data or an exclusive OR of the write data and the similar data is written to the storage device; A deduplication program to run the process.

Images