Control method of storage system

Abstract

A control method includes: dividing a source data unit of a plurality of source data units received from a host into a plurality of source chunks; generating a unique value for each of the plurality of source chunks; inserting the unique value into a source Bloom filter of the source data unit; calculating, for the source Bloom filter, a Hamming similarity to a storage Bloom filter of each of the plurality of active storage data units; selecting the active storage data unit as a target data unit based on the highest Hamming similarity; classifying each of the plurality of source chunks as a first de-reorganized chunk, a de-reorganized chunk, or a unique chunk using a unique value; and storing at least one of the first de-reorganized blocks in the target data unit, such that spatial locality of each block may be improved and fragmentation may be mitigated.

Description

[0001] Cross-reference of related applications

[0002] This application claims the benefit of priority to Korean Patent Application No. 10-2024-0120322 filed on September 4, 2024, with the Korean Intellectual Property Office, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] Some example embodiments relate to a method for controlling a storage system. Background Technology

[0004] Storage devices can store data in response to write requests and output stored data in response to read requests. As the size of data stored in storage devices increases, it may be important to effectively manage deduplicated data within the storage device to efficiently utilize its limited capacity. Deduplication can be a technique that enables efficient use of storage device capacity. Deduplication may include determining whether data to be stored in the storage device is duplicated with data already stored in the storage system, and when it is determined that the data to be stored is duplicated, referring to the already stored data instead of storing the data to be stored in the storage device. In this case, data can be divided into blocks, such as blocks of a predetermined size, and deduplication can be performed on a block-by-block basis. Summary of the Invention

[0005] Some example embodiments may provide a control method for a storage system that can mitigate source block fragmentation by storing at least one deduplicated source block from a plurality of source blocks included in a source data unit received from a host in a storage data unit based on correlation.

[0006] According to some example embodiments, a method for controlling a storage system includes: dividing source data units, which are the target for deduplication, from a plurality of source data units received from a host into a plurality of source blocks; generating a unique value for each of the plurality of source blocks; inserting the unique value into a source Bloom filter of the source data unit and updating the source Bloom filter; calculating a Hamming similarity between the source Bloom filter and a storage Bloom filter of each of the plurality of active storage data units; selecting the active storage data unit with the highest Hamming similarity among the plurality of active storage data units as the target data unit; classifying each of the plurality of source blocks into one of a first deduplication block, a deduplication block, or a unique block using the unique value; and storing at least one of the first deduplication blocks in the target data unit.

[0007] According to some example embodiments, alternatively or additionally, a method for controlling a storage system includes: dividing a source data unit, which is the target for deduplication, from a plurality of source data units received from a host into a plurality of source blocks; generating a unique value for each of the plurality of source blocks; inserting the unique value into a source Bloom filter of the source data unit and updating the source Bloom filter; calculating a Hamming similarity for the source Bloom filter to a storage Bloom filter of each of a plurality of active storage data units in which the deduplication blocks are stored; selecting a target data unit from the plurality of active storage data units using the Hamming similarity; and adding at least one first deduplication block from the plurality of source blocks to the target data unit.

[0008] According to some example embodiments, alternatively or additionally, a method for controlling a storage system includes: dividing source data units, which are the target of deduplication, into a plurality of source blocks; generating a unique value for each of the plurality of source blocks; inserting the unique value into a source Bloom filter of the source data unit and updating the source Bloom filter; calculating a Hamming similarity for the source Bloom filter to a storage Bloom filter of each of a plurality of active storage data units in which the deduplication blocks are stored; selecting a target data unit from the plurality of active storage data units using the Hamming similarity; classifying each of the plurality of source blocks into one of a first deduplication block, a deduplication block, or a unique block using the unique value; adding the first deduplication block to the target data unit and accessing the first deduplication block by referring to the location of the target data unit; and

[0009] Insert the first decomposition block into the target Bloom filter of the target data unit and update the bits of the target Bloom filter. Attached Figure Description

[0010] The above and other aspects, features, and advantages of some exemplary embodiments will become clearer from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0011] Figure 1 This is a block diagram illustrating a storage system according to some example embodiments;

[0012] Figure 2 This is a block diagram illustrating a controller for a storage device according to some example embodiments;

[0013] Figure 3 This is a block diagram illustrating a memory device according to some example embodiments;

[0014] Figure 4 This is a diagram illustrating a 3DV-NAND structure applicable to a storage device according to some example embodiments;

[0015] Figure 5This is a diagram illustrating the hierarchical structure of a storage system according to some example embodiments;

[0016] Figure 6 This is a diagram illustrating source data groups, repeating data groups, and reference groups according to some example embodiments;

[0017] Figure 7 This is a block diagram illustrating a deduplication engine according to some example embodiments;

[0018] Figure 8 This is a flowchart illustrating the deduplication process of source data units according to some example embodiments;

[0019] Figure 9 This is a flowchart illustrating a process for selecting target data units by detecting similarity, according to some example embodiments;

[0020] Figure 10 This is a flowchart illustrating a process for performing deduplication by distinguishing source chunks, according to some example embodiments;

[0021] Figure 11 This is a diagram illustrating source data cells and source Bloom filters according to some example embodiments;

[0022] Figure 12 and Figure 13 This is a diagram illustrating deduplication processing according to some example embodiments;

[0023] Figure 14 and Figure 15 This is a diagram illustrating deduplication processing according to some example embodiments;

[0024] Figure 16 and Figure 17 This is a diagram illustrating deduplication processing according to some example embodiments;

[0025] Figure 18 and Figure 19 This is a diagram illustrating deduplication processing according to some example embodiments;

[0026] Figure 20 and Figure 21 This is a diagram illustrating deduplication processing according to some example embodiments;

[0027] Figure 22 This is a diagram illustrating the effect of deduplication according to some example embodiments;

[0028] Figure 23 These are diagrams illustrating the deduplication effect according to some example embodiments; and

[0029] Figure 24This is a diagram illustrating a system of application storage devices according to some example embodiments. Detailed Implementation

[0030] In the following description, exemplary embodiments of this disclosure will be described with reference to the accompanying drawings.

[0031] Figure 1 This is a block diagram illustrating a storage system according to some example embodiments.

[0032] Figure 1 Storage system 1 is shown. Storage system 1 can be configured as a computing system for processing various types of information, such as one or more of a data center, server, personal computer, laptop, tablet PC, server, workstation, tablet PC (personal computer), smartphone, digital camera, black box, etc. Host 10 may include at least one operating system (OS). The operating system can manage and control some or all of the functions and / or operations of host 10 (e.g., overall functions and / or operations).

[0033] Storage system 1 may include host 10, bus 20, deduplication engine 30, and multiple storage devices 100a to 100z. The multiple storage devices 100a to 100z may also be referred to as first storage device 100a to Nth storage device 100z. N can be any natural number.

[0034] Host 10 can control some or all (e.g., overall operation) of, for example, the operation of storage system 1. For example, host 10 can communicate with multiple storage devices 100a to 100z via bus 20. Host 10 can store data to or read data stored in the multiple storage devices 100a to 100z.

[0035] Bus 20 provides an interface for communication between host 10, deduplication engine 30, and multiple storage devices 100a to 100z. For example, bus 20 may be configured as a Peripheral Component Interconnect High Speed ​​(PCIe) bus. Bus 20 may be implemented as a wired bus and / or a wireless bus; the example embodiments are not limited thereto. Host 10 can read data stored in the multiple storage devices 100a to 100z and / or write data to the multiple storage devices 100a to 100z via bus 20. The multiple storage devices 100a to 100z can communicate with each other in a peer-to-peer (P2P) manner via bus 20.

[0036] Each of the plurality of storage devices 100a to 100z may include a storage medium for storing data in response to a request from host 10. For example, each of the plurality of storage devices 100a to 100z may include at least one of a solid-state drive (SSD), embedded memory, and removable external memory. Each of the plurality of storage devices 100a to 100z may include storage media of the same type and / or different types, having the same or different capacities and / or the same or different read and / or write times. When the plurality of storage devices 100a to 100z are configured as SSDs, the plurality of storage devices 100a to 100z may be devices compliant with the Fast Non-Volatile Memory (NVMe) standard. Alternatively or additionally, when the plurality of storage devices 100a to 100z are embedded memory or external memory, the plurality of storage devices 100a to 100z may be devices compliant with Universal Flash Memory (UFS) and / or Embedded Multimedia Card (eMMC) standards. The host 10 and each of the plurality of storage devices 100a to 100z can generate packets according to the standard protocol adopted, and can send the packets.

[0037] Each of the plurality of storage devices 100a to 100z may store data received from host 10, or may provide the stored data to host 10. Each of the plurality of storage devices 100a to 100z may communicate with each other, for example, in a one-way, two-way or multiplexing communication method such as a broadcast mechanism, in order to share and exchange data (such as, but not limited to, data and / or instructions) encoded in analog and / or digital formats in a serial and / or parallel manner; the exemplary embodiments are not limited thereto.

[0038] Each of the plurality of storage devices 100a to 100z may include controllers 102a to 102z (102) and memory devices 104a to 104z (104). The functions and structures of the plurality of storage devices 100a to 100z may be similar to each other; the example embodiments are not limited thereto.

[0039] Even when not powered, the memory device 104 can maintain the stored data. The memory device 104 can store data supplied from the host 10 via programming operations that may involve setting the voltage of the memory cells, and can output the data stored in the memory device 104 via write and / or read operations. The memory device 104 may include multiple memory blocks, each of which may include multiple pages, and each of which may include multiple memory cells connected to word lines.

[0040] When each of the plurality of storage devices 100a to 100z includes a flash memory in its storage device 104a to 104z, the flash memory may include a 2D NAND memory array and / or a 3D (or vertical) NAND (VNAND) memory array. Alternatively or additionally, each of the plurality of storage devices 100a to 100z may include various types of non-volatile memory. For example, each of the plurality of storage devices 100a to 100z may include one or more of magnetic RAM (MRAM), spin-transfer torque MRAM, conductive bridged RAM (CBRAM), ferroelectric RAM (FeRAM), phase RAM (PRAM), resistive RAM, and various other types of memory, and each of the plurality of storage devices 100a to 100z may include the same or different memory types.

[0041] The controller 102 can control the memory device 104 in response to a request from the host 10. For example, the controller 102 can provide data read from the memory device 104 to the host 10, and can program the data provided from the host 10 to the memory device 104.

[0042] The deduplication engine 30 can manage duplicate data stored in multiple storage devices 104a to 104z. Depending on various factors such as application and file characteristics, user characteristics, and the availability of backup data, duplicate data with the same content as data provided from host 10 can be stored in at least one of the multiple storage devices 104a to 104z. Repeated copying and storage of data with identical content may lead to unnecessary waste of storage system 1 capacity.

[0043] The deduplication engine 30 can perform deduplication of multiple identical data entries. Deduplication can be, may include, or may be included in an operation that stores only one data entry and manages other data for reference, rather than storing multiple identical data entries as a whole. For example, when performing deduplication of duplicate data, duplicate data stored in a memory device can be deleted, and / or duplicate data temporarily stored in volatile memory such as a write buffer memory can be discarded. By performing deduplication, the capacity of the storage system 1 can be ensured, or the capacity of the storage system can be ensured by the size of the deduplicated data, thus providing a storage system 1 with improved capacity efficiency.

[0044] The deduplication engine 30 can communicate with multiple storage devices 100a to 100z in a P2P manner. For example, the deduplication engine 30 can communicate in a P2P manner through one or more of various standards such as the controller memory buffer (CMP) and permanent memory region (PMR) of high-speed non-volatile memory (NVMe), high-speed compute link (CXL), or accelerator cache coherent interconnect (CCIX).

[0045] In some example embodiments, the deduplication engine 30 may divide data into multiple chunks and perform deduplication on each of the multiple chunks. In some example embodiments, the deduplication engine 30 may divide data units received from host 10 into multiple chunks and generate a unique value for each of the multiple chunks. For example, a data unit may have an object structure in which data and metadata about the data are combined. However, example embodiments thereof are not limited to this.

[0046] When at least one of the multiple storage devices 100a to 100z has included a data unit received from the host 10 and a unique value to be deduplicated, and / or when the unique value to be deduplicated is included in the received data unit, deduplication can be performed on at least one block corresponding to the unique value to be deduplicated. For example, the deduplication engine 30 may store only a single block among the multiple blocks that has the unique value to be deduplicated, and other blocks can be accessed by referring to the stored block.

[0047] In typical storage systems, blocks can be stored at non-contiguous memory addresses because deduplication is repeatedly performed on data units. Since deduplication is repeatedly performed on blocks initially stored at contiguous memory addresses, blocks can be physically distributed and stored. Therefore, each data block may lose spatial locality, and data block fragmentation may occur.

[0048] However, in some example embodiments, the deduplication engine 30 may perform deduplication on at least one first deduplication block of a data unit that may be the deduplication target, and may determine the location where the deduplicated block is stored to prevent fragmentation of the data unit. The deduplicated block from the data unit received from the host may be stored in a storage data unit. A storage data unit may refer to a data unit that stores the deduplicated block.

[0049] When deduplicated blocks and identical blocks have been stored in a storage data unit, only the location information needs to be updated to reference the location of the stored blocks. When a deduplicated block is not stored in a storage data unit as the first deduplicated block, the block can be stored in one of the storage data units, and the location information can be updated.

[0050] In some example embodiments, the deduplication engine 30 may determine the storage data units in which the first deduplication block can be stored based on the correlation between the data unit that may be the deduplication target and a plurality of storage data units. For example, among the plurality of storage data units, the first deduplication block may be stored in the storage data unit that includes the maximum number of data blocks, which may be the same as the data blocks in the data unit that may be the deduplication target. For example, the probability that related data blocks are included in the same storage data unit may be increased.

[0051] At least one first deduplication block can be stored in contiguous memory addresses of the stored data units. This improves the spatial locality of each block, thereby mitigating or improving fragmentation. Consequently, the number of block reads can be reduced when the output is a data unit that may be the target for deduplication.

[0052] Figure 2 This is a block diagram illustrating a controller for a storage device according to some example embodiments.

[0053] Figure 2 This may correspond to the embodiment in some example embodiments. Figure 1 A block diagram of controllers 102a to 102z (102) of the storage device is shown. Controller 200 can control the storage device in response to a request from the host. For example, controller 200 can provide data read from the storage device to the host, and can program the data provided from the host to the storage device. In this case, the host can correspond to... Figure 1 The host 10 in the middle, and the memory device may correspond to Figure 1 One of the multiple memory devices 104a to 104z.

[0054] Controller 200 may include a host interface (I / F) 210, a memory interface 220, and a central processing unit (CPU) 230. Furthermore, controller 200 may also include a flash translation layer (FTL) 240, a packet manager 250, a buffer memory 260, an error correction code (ECC) engine 270, and an Advanced Encryption Standard (AES) engine 280. Controller 200 may also include working memory (not shown), in which the flash translation layer 240 is loaded, and data programming and reading operations for the memory device can be controlled by the CPU 230 running the flash translation layer 240.

[0055] Host interface 210 can send packets to and / or receive packets from the host. Packets sent from the host to host interface 210 may include commands and / or data to be programmed into a memory device, and packets sent from host interface 210 to the host may include commands and / or data to be programmed into a memory device. Packets may include responses to commands and / or data read from the memory device.

[0056] The memory interface 220 can send data to be programmed into the memory device to the memory device or receive data read from the memory device. The memory interface 220 can be implemented in accordance with standard protocols such as Toggle and / or Open NAND Flash Interface (ONFI).

[0057] CPU 230 can control some (e.g., overall operation) of the operation of controller 200.

[0058] The flash translation layer 240 can perform various functions, such as address mapping, wear leveling, and garbage collection. Address mapping can be an operation that changes a logical address received from the host to a physical address used to actually store data in the memory device. Wear leveling can be a technique that prevents or reduces excessive degradation of a particular block by allowing blocks in the memory device to be used evenly, and can be implemented through firmware techniques for balancing, for example, the erase counts of physical blocks. Garbage collection can be a technique that ensures or helps ensure available capacity in the memory device by copying valid data from a block to a new block and erasing the existing block.

[0059] The packet manager 250 can generate packets according to the protocol of the interface agreed upon with the host, or parse various information from packets received from the host. Furthermore, the buffer memory 260 can temporarily store data to be programmed into or read from the memory device. The buffer memory 260 can be configured to be located within the controller 200, or it can be located externally to the controller 200.

[0060] ECC engine 270 performs error detection and correction functions on read data read from the memory device. In some example embodiments, ECC engine 270 can generate parity bits for write data to be written to the memory device, and the generated parity bits can be stored in the memory device along with the write data. When reading data from the memory device, ECC engine 270 can use the parity bits read from the memory device along with the read data to correct errors in the read data, and can output error-corrected read data.

[0061] The AES engine 280 can, for example, perform at least one of encryption and decryption operations on the data input to the controller 200 by using a symmetric key algorithm.

[0062] When duplicate data with identical content is stored in a memory device, deduplication can be performed on the duplicate data. In this case, deduplication can correspond to inline deduplication, post-processing deduplication, or incremental compression.

[0063] For example, in inline deduplication where deduplication is performed in real time before the data is stored in the memory device, each piece of data provided by the host may first be stored in the buffer memory 260 before deduplication is complete. After deduplication is complete, the non-duplicate data can be stored in the memory device.

[0064] For example, in post-processing deduplication, data provided from the host can be stored in a memory device, and deduplication can then be performed on the duplicate data. Deduplication can be performed in the order the data was provided from the host. Duplicate data can be deleted from the memory device, and the duplicate data can be accessed by referring to the location of previously stored data.

[0065] Figure 3 This is a block diagram illustrating a memory device according to some example embodiments. Figure 4 This is a diagram illustrating a 3D V-NAND structure applicable to a storage device according to some example embodiments.

[0066] Reference Figure 3 The memory device 300 may include control logic circuitry 320, memory cell array 330, page buffer 340, voltage generator 350, and row decoder 360. The memory device 300 may also include memory interface circuitry for receiving commands (CMD) and addresses (ADDR) from external entities and exchanging data (DATA) with external entities, and may also include column logic, pre-decoder, temperature sensor, command decoder, address decoder, etc.

[0067] The control logic circuit 320 generally controls various operations within the memory device 300. The control logic circuit 320 can output various control signals in response to commands CMD and / or addresses ADDR from the memory interface circuit 310. For example, the control logic circuit 320 can output voltage control signals CTRL_vol, row address X-ADDR, and column address Y-ADDR.

[0068] The memory cell array 330 may include multiple memory blocks BLK1 to BLKz (where z is a positive integer), and each of the multiple memory blocks BLK1 to BLKz may include multiple memory cells. The memory cell array 330 can be connected to the page buffer 340 via the bit line BL, and can be connected to the line decoder 360 via the word line WL, the serial select line SSL, and the ground select line GSL.

[0069] As an example, the memory cell array 330 may include a three-dimensional memory cell array, and the three-dimensional memory cell array may include a plurality of NAND strings. Each NAND string may include memory cells respectively connected to word lines stacked vertically on a substrate. In some example embodiments, the memory cell array 330 may include a two-dimensional memory cell array, and the two-dimensional memory cell array may include a plurality of NAND strings arranged along row and column directions. The memory cell array 330 may include one or more of single-level cell (SLC), multi-level cell (MLC), three-level cell (TLC), or four-level cell (QLC); the example embodiments are not limited thereto.

[0070] Page buffer 340 may include multiple page buffers PB1 to PBn (where n is an integer equal to or greater than 3), and the multiple page buffers PB1 to PBn may be connected to memory cells via multiple bit lines BL. Here, n may be greater than, less than, or equal to z. Page buffer 340 may select at least one bit line from the bit lines BL in response to column address Y-ADDR. Page buffer 340 may operate as a write driver or a sense amplifier depending on the operating mode. For example, in a programming operation, page buffer 340 may apply a bit line voltage corresponding to the data to be programmed to the selected bit line. In a read operation, page buffer 340 may read data stored in the memory cell by sensing the current or voltage of the selected bit line.

[0071] Voltage generator 350 can generate various types of voltages for performing programming, reading, and erasing operations based on the voltage control signal CTRL_vol. For example, voltage generator 350 can generate programming voltage, read voltage, programming verification voltage, erase voltage, etc. as word line voltage VWL.

[0072] The row decoder 360 can select one of multiple word lines (WL) and multiple string select lines (SSL) in response to the row address X-ADDR. For example, during a programming operation, the row decoder 360 can apply a programming voltage to the selected word line and can program a verification voltage, and during a read operation, the row decoder 360 can apply a read voltage to the selected word line.

[0073] Figure 4 This illustrates a 3D V-NAND structure applicable to some example embodiments. When the memory device of the storage device ( Figure 3 When 300 in the memory device is implemented as or includes 3D V-NAND type flash memory, each of the plurality of memory blocks included in the memory device can be represented as follows: Figure 4 The equivalent circuit shown. For example, multiple memory NAND strings included in the memory block BLKi can be formed in a direction perpendicular to the substrate.

[0074] Reference Figure 4 The memory block BLKi may include multiple memory NAND strings NS11 to NS33 connected between bit lines BL1, BL2, and BL3 and the common-source line CSL. Each of the multiple memory NAND strings NS11 to NS33 may include a string select transistor SST, multiple memory cells MC1, MC2, ..., MC8, and a ground select transistor GST. Figure 3 In this context, multiple memory NAND strings NS11 to NS33 may include eight memory cells MC1, MC2, ..., MC8, but the example embodiments are not limited thereto.

[0075] The string select transistor SST can be connected to the corresponding string select lines SSL1, SSL2, and SSL3. Multiple memory cells MC1, MC2, ..., MC8 can be connected to the corresponding gate lines GTL1, GTL2, ..., GTL8, respectively. Gate lines GTL1, GTL2, ..., GTL8 can correspond to word lines, and a portion of gate lines GTL1, GTL2, ..., GTL8 can correspond to pseudo-word lines. The ground select transistor GST can be connected to the corresponding ground select lines GSL1, GSL2, and GSL3. The string select transistor SST can be connected to the corresponding bit lines BL1, BL2, and BL3, and the ground select transistor GST can be connected to the common-source line CSL.

[0076] Word lines at the same level (e.g., WL1) can be connected together, and ground select lines GSL1, GSL2, and GSL3, as well as string select lines SSL1, SSL2, and SSL3, can be isolated from each other. Figure 3 In this example, the memory block BLK can be connected to eight gate lines GTL1, GTL2, ..., GTL8 and three bit lines BL1, BL2 and BL3, and the example embodiments are not limited thereto.

[0077] In some example embodiments, from Figure 1 Among the multiple data units received by host 10, the source data unit that is the target for deduplication can be divided into multiple blocks and deduplicated. At least one first deduplicated block among the multiple blocks can be stored in one of the multiple storage data units. The storage data unit can store the deduplicated block. For example, when it is determined that two or more identical blocks are duplicated, the block can be stored in the storage data unit.

[0078] According to some example embodiments, a storage data unit can be configured as a storage space, wherein consecutive memory addresses of memory cells arranged in physically adjacent regions on the memory cell array 330 point to the storage space. In other words, at least one first decomposition block of the source data unit can be stored in consecutive memory addresses of the storage data unit.

[0079] Refer to together Figure 3 The storage data unit may correspond to a plurality of memory blocks BLK1 to BLKz, or a portion of an adjacent memory block among the plurality of memory blocks BLK1 to BLKz. Alternatively, the storage data unit may correspond to one of the plurality of memory blocks BLK1 to BLKz. See also Figure 4 The storage data unit may correspond to a portion of an adjacent memory NAND string among multiple memory NAND strings NS11 to NS33. However, the configuration of the storage data unit is not limited to this.

[0080] Figure 5 This is a diagram illustrating the hierarchical structure of a storage system according to some example embodiments.

[0081] Reference Figure 5 According to some example embodiments, the storage system may have a software layered structure including a host layer, a deduplication layer, and a storage layer.

[0082] The host layer may include applications and a file system. Applications may be or may include software programs designed to perform specific functions. For example, applications may include one or more of word processors, database programs, web browsers, image editing programs, etc. Applications may communicate with the user of the storage system through user interface devices such as touch screens, monitors, mice, keyboards, microphones, speakers, etc.

[0083] A file system manages files. It systematically manages one or more operations on files, such as creating, accessing, opening, modifying, and deleting them, in response to operations from the operating system or requests from applications executed by the operating system. A file can be a series of data. The series of data corresponding to a file can be stored in the storage layer.

[0084] File systems may include one or more of the following: FAT, FAT32, NTFS, HFS (Hierarchical File System), JSF2 (Journal File System 2), XFS, ODS-5 (Disk Structure-5), UDF, ZFS, UFS (UNIX File System), ext2, ext3, ext4, ReiserFS, Reiser4, ISO 9660, Gnome, VFS, EFS, or WinFS.

[0085] The deduplication layer may include a deduplication engine, which manages the performance of deduplication for multiple data entries stored in the storage layer under the control of the file system. The deduplication engine can manage the performance of inline deduplication or post-processing deduplication. In inline deduplication, the deduplication engine performs deduplication before the data is stored on the storage device, and non-duplicate data can be stored on the storage device. In post-processing deduplication, the file system can store non-duplicate data on the storage device, and the deduplication engine can search the data stored on the storage device and perform deduplication. However, the types of deduplication are not limited to these.

[0086] The storage layer may include a first storage device through an Nth storage device. The first to Nth storage devices may include a source data group, a duplicate data group, and a reference group. Each of the first to Nth storage devices may be included in one of the groups. Alternatively, the first to Nth storage devices may be included in all groups.

[0087] The deduplication engine can communicate with the storage layer and can deduplicat and manage multiple identical blocks. A block can be a unit obtained by dividing data into predetermined sizes. When multiple identical blocks are determined to be duplicates, the blocks can be stored in a duplicate data group. To access blocks stored in the duplicate data group, blocks included in the source data group can be managed to reference blocks stored in the duplicate data group. Reference information for each block can be stored in a reference group.

[0088] In some example embodiments, a repeating data group may include multiple storage data units. The storage data units may store the group at consecutive memory addresses.

[0089] See below for reference. Figure 6 The description includes the source data group, duplicate data group, and reference group in the storage layer.

[0090] Figure 6 This is a diagram illustrating source data groups, repeating data groups, and reference groups according to some example embodiments.

[0091] In some example embodiments, the storage system can deduplicate multiple identical data entries, and deduplication allows for efficient use of the storage system's capacity. Specific example embodiments of the storage system may be similar to those described in the references. Figures 1 to 5 Example of the description.

[0092] From the host ( Figure 110) The data provided can be, or may be referred to as, a data unit, and a data unit can be an object structure in which data and metadata about the data are combined. Deduplication can be performed on chunks obtained by dividing (e.g., splitting) data units into sizes such as a predetermined size. For example, a data unit can be a megabyte (MB) unit, and a chunk can be a kilobyte (KB) unit. However, data units and / or chunks are not limited to this.

[0093] Multiple storage devices may include source data groups and duplicate data groups. One of the same multiple blocks may be stored in the duplicate data group, and the source data group may reference the stored block. In some example embodiments, non-duplicate blocks may be stored in the source data group. In this case, the source data group may store blocks by source data units (SCUs) provided from the host, or may reference stored blocks in the duplicate data group. The duplicate data group may include active storage data units (ASUs) and / or storage data units.

[0094] An active storage data unit (ASU) may be or may correspond to a data unit in which new blocks can be further stored, and a storage data unit may be or may correspond to a data unit in which new blocks cannot be further stored. Specifically, an allowable size can be determined for an active storage data unit (ASU), and this allowable size may be a storage capacity such as a predetermined storage capacity. Active storage data units (ASUs) and storage data units can be distinguished based on their allowable sizes.

[0095] The size of an Active Storage Unit (ASU) can be smaller than the allowed size. When a block is stored in an Active Storage Unit (ASU) and the size of the Active Storage Unit (ASU) is equal to or greater than the allowed size, the Active Storage Unit (ASU) can become a Storage Unit, allowing new blocks to be prevented from being stored further.

[0096] In some example embodiments, deduplication can be performed using the unique value of the chunk. The unique value of the chunk may have a fixed size and, in some cases, may be generated by a hash function. For example, using Secure Hash Algorithm-1 (SHA-1), the unique value of the chunk may have a fixed size of 20 bytes. SHA-1 can be a fingerprint or hash of the chunk; in some example embodiments, the hash may or may not be stored with other information (such as, but not limited to, the time of day). However, the type of hash function and / or the size of the unique value are not limited to these.

[0097] In some example embodiments, the storage system may divide the source data unit (SCU) into multiple source blocks and generate a unique value for each of the multiple source blocks. The storage system can use the unique value of the source data unit to determine whether the block should be deduplicated.

[0098] Deduplication can be performed in the order in which the source data units are provided from the host. Figure 6 In some example embodiments shown, a first source data unit SCU_1 and a second source data unit SCU_2 may be provided sequentially.

[0099] The first source data unit SCU_1 may include multiple blocks (such as the first source block to the third source block) and may be divided or segmented into sizes such as a predetermined size. The unique values ​​of each of the first to third source blocks may be unique values ​​a (UVa), c (UVc), and d (UVd), respectively. The second source data unit SCU_2 may include the first to third source blocks and may be divided or segmented into sizes such as a predetermined size. The unique values ​​of each of the first to third source blocks may be unique values ​​a (UVa), b (UVb), and c (UVc), respectively.

[0100] Before the first source data unit SCU_1 is provided from the host, the previously provided source data unit SCU may not exist. Since there are no blocks with the same unique value among the three source blocks of the first source data unit SCU_1, the three source blocks of the first source data unit SCU_1 do not need to be deduplicated. In this case, the three source blocks of the first source data unit SCU_1 can be stored in the source data group, and there may be no stored blocks in the duplicate data group.

[0101] Subsequently, a second source data unit SCU_2 can be provided from the host. The second source data unit SCU_2 may include the same unique values ​​as the first source data unit SCU_1. For example, the unique values ​​of the first source block and the third source block of the second source data unit SCU_2 may be the same as the unique values ​​of the first source block and the second source block of the first source data unit SCU_1, respectively. Deduplication can be performed on the first source block of the first source data unit SCU_1 and the first source block of the second source data unit SCU_2 corresponding to the unique value a (UVa). Deduplication can also be performed on the second source block of the first source data unit SCU_1 and the third source block of the second source data unit SCU_2 corresponding to the unique value c (UVc). For example, the block can be classified as a first deduplication block.

[0102] The third source block of the first source data unit SCU_1 corresponding to the unique value d (UVd) and the second source block of the second source data unit SCU_2 corresponding to the unique value b (UVb) can be classified as non-repeating unique blocks.

[0103] The storage system can generate a first active storage data unit to store the blocks to be deduplicated. The active storage data unit may include a unique name. The name of the first active storage data unit can be "ASU_aaa".

[0104] In the first source block of the first source data unit SCU_1 and the first source block of the second source data unit SCU_2 corresponding to the unique value a (UVa), the block corresponding to the unique value a (UVa) can be stored in the first active storage data unit ASU_aaa. To access the first source block of the first source data unit SCU_1 or the first source block of the second source data unit SCU_2, the block stored in the first active storage data unit ASU_aaa can be referenced. In this case, information such as the location of the first active storage data unit ASU_aaa, in which the block corresponding to the unique value a (UVa) is stored, can be referenced, and this reference information can be stored in both the first source data unit SCU_1 and the second source data unit SCU_2.

[0105] In the second source block of the first source data unit SCU_1 and the third source block of the second source data unit SCU_2 corresponding to the unique value c (UVc), the block corresponding to the unique value c (UVc) can be stored in the first active storage data unit ASU_aaa. To access the second source block of the first source data unit SCU_1 or the third source block of the second source data unit SCU_2, the block stored in the first active storage data unit ASU_aaa can be referenced. In this case, information such as the location of the first source storage data unit ASU_aaa, in which the block corresponding to the unique value c (UVc) is stored, can be referenced, and the reference information can be stored in the first source data unit SCU_1 and the second source data unit SCU_2.

[0106] For reference Figure 6 As shown in some example embodiments described, reference information can be formed and stored in a tabular format, but such example embodiments are not limited thereto. Reference information may include the length of a block referenced by an offset from a source data unit, the name of the active storage data unit where the referenced block is stored, and the offset of the active storage data unit where the referenced block is stored. The offset may indicate the relative distance of the block within the data unit and, in some cases, may begin at "0".

[0107] As an example, in the first source block of deduplication in the first source data unit SCU_1, the offset of the first source block can be "0". The length (run length) of the reference block can be "1", the name of the active storage data unit storing the reference block can be "ASU_aaa", and the offset of the active storage data unit storing the reference block can be "0".

[0108] As another example, in the deduplication of the third source block of the second source data unit SCU_2, the offset of the third source block can be "2". The length of the reference block can be "1", where the name of the active storage data unit storing the reference block can be "ASU_aaa", and the offset of the active storage data unit storing the reference block can be "1".

[0109] Non-repeating blocks can be stored in the source data group. Specifically, the third source block of the first source data unit SCU_1 corresponding to the unique value d (UVd) can be stored in the first source data unit SCU_1. The second source block of the second source data unit SCU_2 corresponding to the unique value b (UVb) can be stored in the second source data unit SCU_2.

[0110] In some example embodiments, the multiple storage devices may further include a reference group. The reference group may include information about the storage location of each decompiled block and a reference source data unit. For example... Figure 6 Some of the example embodiments shown may form and store each piece of information in a tabular form for each decomposition block, but the example embodiments are not limited thereto.

[0111] For example, the decompiled block corresponding to the unique value a (UVa) can be stored in the first active storage data unit ASU_aaa, and the offset can be "0". The source data unit referencing the decompiled block corresponding to the unique value a (UVa) can be the first source data unit SCU_1 and the second source data unit SCU_2. In another example, the decompiled block corresponding to the unique value c (UVc) can be stored in the first active storage data unit ASU_aaa, and the offset can be "1". The source data unit referencing the decompiled block corresponding to the unique value c (UVc) can be the first source data unit SCU_1 and the second source data unit SCU_2.

[0112] Subsequently, multiple source data units can be provided from the host, and deduplication can be repeatedly performed on multiple source data units. The number of source data units included in the source data group can be increased, and the number of active storage data units included in the duplicate data group can be increased. The number of deduplicated blocks stored in the active storage data units can be increased.

[0113] For example, as the decompiled block is further stored in the first active data unit ASU_aaa, the size of the first active data unit ASU_aaa may become equal to or greater than the allowed size. In this case, the first active data unit ASU_aaa may be changed to the first data unit ASU_aaa, so that the new block cannot be further stored in the first data unit ASU_aaa.

[0114] In some example embodiments, the storage system may select a location where correlation can be used to store active storage data units of the decompiled block. In this case, the similarity between the source data unit and multiple active storage data units can be calculated, as will be referred to below. Figure 7 and Figure 8 To describe in more detail.

[0115] Figure 7 This is a block diagram illustrating a deduplication engine according to some example embodiments. Figure 8 This is a flowchart illustrating the deduplication process of source data units according to some example embodiments. Figure 9 This is a flowchart illustrating a process for selecting target data units by detecting similarity, according to some example embodiments. Figure 10 This is a flowchart illustrating a process for performing deduplication by distinguishing source chunks, according to some example embodiments.

[0116] A storage system may include a host, multiple storage devices, and a deduplication engine. When data provided from the host to the multiple storage devices is programmed, the deduplication engine can more efficiently manage the storage system's capacity by removing duplicate data. The data provided from the host may be source data units. Specific, non-limiting example embodiments of the storage system may be similar to those described in the references. Figures 1 to 6 Example of the description.

[0117] Reference Figure 7 According to some example embodiments, the deduplication engine 400 may include a chunk generator 410, a unique value generator 420, a target data unit selector 430, and a chunk controller 440. The target data unit selector 430 may include a Bloom filter 432, a Hamming similarity calculator 434, and an active storage data unit table 436. The chunk controller 440 may include a unique value table 442.

[0118] For example, unique value table 442 may include source data units that have undergone deduplication, multiple active storage data units, and unique values ​​of blocks stored in at least one storage data unit. Figure 6 In some example embodiments shown, the unique value table 442 may have a form similar to that of the reference group, but some example embodiments are not limited thereto.

[0119] The components of the target data unit selector 430 can communicate with each other to send and / or receive information (such as, but not limited to, data and / or commands), which can be sent via a bus (such as, but not limited to, a wired bus and / or a wireless bus). The information can be sent and / or received in a broadcast, one-way, or two-way manner, and can be sent and / or received serially and / or in parallel, and can be encoded digitally and / or analogously. Example embodiments are not limited thereto.

[0120] Refer to together Figure 7 and Figure 8 The chunk generator 410 can receive source data units and divide (e.g., split) the source data units into multiple source chunks (S100). A source chunk can be or corresponds to a data unit obtained by dividing the source data unit into sizes such as, but not limited to, a predetermined size. The unique value generator 420 can generate a unique value for each of the multiple source chunks (S110). The unique value of a chunk can have a fixed size generated by a hash function.

[0121] The target data unit selector 430 can use the unique value of the source data unit to detect the similarity between multiple active storage data units and the source data unit (S120). The target data unit selector 430 can then select a target data unit from the multiple active storage data units using the calculated similarity (S130). See also... Figure 9 The processing of S120 and S130 is described in more detail.

[0122] Also refer to Figure 9 The target data cell selector 430 can update the source Bloom filter by inserting the unique value of the source data cell into the source Bloom filter 432 of the source data cell (S210).

[0123] The target data unit selector 430 can use a source Bloom filter and a storage Bloom filter to detect similarity. In this case, a storage Bloom filter can be loaded from the active storage data unit table 436. The active storage data unit table 436 may include information about the name of the active storage data unit, the storage Bloom filter, and its current size.

[0124] In some example embodiments, the source Bloom filter and the storage Bloom filter may be or include (or be included in) a data structure in which the resulting values ​​of the unique values ​​of the source data cells or storage data cells input to the hash function are indexed as bit arrays. For example, the source Bloom filter may probabilistically indicate whether a source block with a specific unique value is included in the corresponding source data cell, and the storage Bloom filter may probabilistically indicate whether a block with a specific unique value is included in the corresponding storage data cell.

[0125] As a non-limiting example, the similarity can be Hamming similarity, for example, a count similarity of one. The Hamming similarity calculator 434 can calculate the Hamming similarity to the storage Bloom filter of each of the plurality of active storage data units of the source Bloom filter (S220). The storage Bloom filters of each of the plurality of active storage data units can be stored in the active storage data unit table 436.

[0126] The target data unit selector 430 can use the Hamming similarity calculated in the S220 process to select the target data unit (S230 to S280). When the Hamming similarity is greater than a threshold and multiple Hamming similarities are not the highest (No in S230 and No in S240), the target data unit selector 430 can select the unit with the highest Hamming similarity as the target data unit from multiple active storage data units (S260). In this case, the threshold can be configured to a predetermined positive number.

[0127] When the Hamming similarity is below the threshold (S230 is) or multiple Hamming similarities are the highest (S240 is), the target data unit selector 430 can select the target data unit based on the number of multiple active storage data units (S250).

[0128] When the number of multiple active storage data units is the maximum allowable number (Yes in S250), the target data unit selector 430 can select the active storage data unit with the smallest size among the multiple active storage data units as the target data unit (S270).

[0129] When the number of multiple active storage data units is equal to or less than the maximum allowed number (No in S250), the target data unit selector 430 can generate a new active storage data unit and select that unit as the target data unit (S280).

[0130] Reference Figure 7 and Figure 8 The chunk controller 440 can use unique values ​​to classify each of the multiple source chunks as a first decompiled chunk, a decompiled chunk, or a unique chunk (S140). In this case, the chunk controller 440 can use the unique values ​​of the stored chunks in the unique value table 442. Thereafter, the chunk controller 440 can perform deduplication on the multiple source chunks (S150). See also... Figure 10 The processing of S140 and S150 is described in more detail.

[0131] When a unique value identical to the unique value of the source block is not included in the active storage data unit or the storage data unit, and a unique value identical to the unique value of the source block is not included in the deduplicated source data unit (No in S300 and No in S310), the block controller 440 may classify the source block as a unique block and store the block in the source data unit (S320).

[0132] Among multiple source chunks, non-repeating chunks can be classified as unique chunks. In other words, among multiple source chunks, chunks other than the first decompiled chunk and the decompiled chunks can be classified as unique chunks.

[0133] When a unique value identical to the source block's unique value is not included in the active storage data unit or the stored data unit (S300: No), but a unique value identical to the source block's unique value is included in a deduplicated source data unit (S310: Yes), the block controller 440 may classify the source block as a first deduplicated block and may store the block in the target data unit (S330). The block controller 440 may refer to the blocks stored in the source data unit (S340).

[0134] The block controller 440 can update the active storage data unit table and reference information (S350). Some specific non-limiting example embodiments of the reference information may be similar to the reference information. Figure 6 Example of the description.

[0135] When a unique value identical to the unique value of the source block is included in the source data unit or the stored data unit (as in S300), the block controller 440 can classify the source block as a decompiled block and refer to the block stored in the source data unit (S360). The block controller 440 can update the reference information (S370).

[0136] According to some example embodiments, target data units can be selected based on similarity, and at least one first deduplication block can be stored in the target data unit. In other words, at least one first deduplication block can be stored in contiguous memory addresses. Therefore, fragmentation of blocks can be mitigated, reducing the number of times blocks are read to output data units, which can be deduplication targets.

[0137] Figure 11 This is a diagram illustrating source data cells and source Bloom filters according to some example embodiments.

[0138] According to some example embodiments, source data units can be divided (e.g., segmented) into multiple source blocks, which may or may not have the same size as each other, and a unique value can be generated for each of the multiple source blocks. The unique values ​​can be inserted into a Bloom filter, allowing the source Bloom filter of the source data unit to be updated. In this case, at least one hash function can be used. The source Bloom filter can be configured as a bit array. The source Bloom filter can have a predetermined size corresponding to the number of bits.

[0139] exist Figure 11 In some example embodiments shown, the source data unit may be divided or segmented into a first source block CHK1 to a sixth source block CHK6. For each of the first source blocks CHK1 to the sixth source block CHK6, a unique value a (UVa) to a unique value f (UVf) may be generated. For example, the unique value of the first source block CHK1 may correspond to the unique value a (UVa), and the unique value of the sixth source block CHK6 may correspond to the unique value f (UVf).

[0140] according to Figure 11 In some example embodiments shown, the source Bloom filter can use 15 bits. Unique values ​​a (UVa) to f (UVf) can be inserted into the source Bloom filter of the source data unit, allowing the source Bloom filter to be updated. The values ​​of the bits corresponding to the unique values ​​a (UVa) to f (UVf) can be configured from the initial state (0) to the configuration state (1). Bits 1, 4, 6, 10, 11, and 15 of the source Bloom filter can be configured from the initial state (0) to the configuration state (1).

[0141] Figure 12 and Figure 13 This is a diagram illustrating deduplication processing according to some example embodiments.

[0142] The storage device can calculate the Hamming similarity between multiple active storage data units and each storage Bloom filter used for the source Bloom filter. The calculated Hamming similarity can be used to select a target data unit from the multiple active storage data units. In this case, the storage Bloom filters can be loaded from the active storage data unit table. This process corresponds to the above. Figure 9 The processing steps are S220 to S280.

[0143] For reference Figure 12 This describes the active storage data unit table in some example embodiments. Figure 12 The active storage unit table in some example embodiments shown may include information about the active storage unit's name (ASU_NAME), storage Bloom filter (ASU_BF), and current size (ASU_SIZE). The active storage unit table can be updated whenever deduplication is performed.

[0144] The active storage data unit table may also include the generation time of the active storage data unit, the address of the storage device containing the active storage data unit, the identifier ID of the storage device containing the active storage data unit, the last time the block was stored in the active storage data unit, and / or the identifier ID of the deduplication engine.

[0145] exist Figure 12 In some example embodiments shown, the name of the active storage data unit, ASU_NAME, may first include a unique value for the stored block. The name of the first active storage data unit may correspond to "ASU_aaa", and the name of the second active storage data unit may correspond to "ASU_bbb". However, some example embodiments are not limited to this.

[0146] The unique value of at least one block stored in the active storage data unit can be inserted into the storage Bloom filter ASU_BF of the active storage data unit, allowing the storage Bloom filter ASU_BF to be updated. In this case, at least one hash function can be used. The storage Bloom filter can be configured as a bit array. The Bloom filter can have a predetermined size corresponding to the number of bits. Figure 12 In some example embodiments shown, the storage Bloom filter ASU_BF can use 15 bits.

[0147] Refer to together Figure 11 The source Bloom filter and the storage Bloom filter can be the same size. Furthermore, the source Bloom filter and the storage Bloom filter can be larger than the number of blocks allowed in each of the multiple active storage data units. For example, the source Bloom filter and the storage Bloom filter can both be 15 bits. In this case, the number of blocks allowed for the source storage data unit can be less than 15. In other words, 14 or fewer blocks can be stored in the source storage data unit.

[0148] The current size of the source Bloom filter, ASU_SIZE, represents the current storage capacity of the source Bloom filter. The size of the source Bloom filter can increase as blocks are stored. That is, the size of the source Bloom filter can change when deduplication is performed repeatedly.

[0149] Figure 12 This can represent the active storage data unit table before and after deduplication is performed on the source data units. Referring to the active storage data unit table before deduplication is performed, the duplicate data group may include a first active storage data unit ASU_aaa and a second active storage data unit ASU_bbb. The allowed size of the first active storage data unit ASU_aaa and the second active storage data unit ASU_bbb may be the same.

[0150] To select a target data unit from the first active storage data unit ASU_aaa and the second active storage data unit ASU_bbb, the Hamming similarity of each of the first active storage data unit ASU_aaa and the second active storage data unit ASU_bbb can be calculated. Specifically, for Figure 11 The source Bloom filter in some example embodiments shown can calculate the Hamming similarity to the storage Bloom filter of each of the first active storage data unit ASU_aaa and the second active storage data unit ASU_bbb.

[0151] Hamming similarity represents the number of positions in both the source and storage Bloom filters where each bit value is "1". A Bloom filter can roughly indicate whether a block with a specific unique value is included in the corresponding data unit. A higher Hamming similarity likely indicates a larger number of blocks with the same unique value in both the source and active storage data units. Figure 12 In the source Bloom filter and the storage Bloom filter, the bit that has a value of "1" can be shown as the first bit.

[0152] exist Figure 12 In some example embodiments shown, Hamming similarity can be described by referring to the Active Storage Unit table before deduplication is performed. For Figure 11 In some example embodiments shown, the source Bloom filter in the storage Bloom filter of the first active storage data unit ASU_aaa can have a value of "1" for each bit at the same position, which can be 4. For Figure 11 The source Bloom filter in some of the example embodiments shown may have a Hamming similarity of "4" with the Bloom filter of the first active storage data unit ASU_aaa.

[0153] for Figure 11 The source Bloom filter shown in some example embodiments can have a number of positions where each bit at the same location in the storage Bloom filter of the second active storage data unit ASU_bbb is "1", which can be 1. For Figure 11 The source Bloom filter in some of the example embodiments shown can have a Hamming similarity of "1" with the Bloom filter of the second active storage data unit ASU_bbb.

[0154] In some example implementations, when the Hamming similarity is higher than a threshold and multiple Hamming similarities are not the highest, the target data unit can be selected from multiple active storage data units, such as the unit with the highest Hamming similarity.

[0155] The threshold can be configured as a predetermined positive number. The threshold can vary based on the number of active storage data units. Alternatively or additionally, the threshold can vary based on the distribution of Hamming similarity. For example, when the number of active storage data units is less than a predetermined number, the unit may be sensitive to even small Hamming similarities and may not generate new active storage data units. In this case, the threshold can be adjusted upwards.

[0156] In another example, when the distribution of Hamming similarity is below a certain or predetermined level, chunks can be densely stored in specific active storage data units with high Hamming similarity. In this case, the threshold can be adjusted downwards to be sensitive to Hamming similarity.

[0157] exist Figure 12 and Figure 13 In some example embodiments, the threshold can be set to "1". In this case, the calculated Hamming similarities 1 and 4 can be equal to or higher than the threshold 1. Furthermore, since only one of the calculated Hamming similarities 1 and 4 has the highest Hamming similarity, multiple Hamming similarities are not necessarily the highest. Therefore, the first active storage data unit ASU_aaa with the highest Hamming similarity 4 can be selected as the target data unit. Figure 12 and Figure 13 The example embodiments in the reference may correspond to the following: Figure 9 The example described in operation S260.

[0158] Reference Figure 13 This can describe the process of storing the first deduplicated source block from multiple source blocks in a target data unit. Figure 11 The source data units of some example embodiments shown can be divided into a first source block CHK1 to a sixth source block CHK6 having unique values ​​a (UVa) to f (UVf).

[0159] Source blocks CHK1, CHK2, CHK4, and CHK5, corresponding to unique values ​​a (UVa), b (UVb), d (UVd), and e (UVe), can be classified as decompiled blocks. These unique values ​​can also be included in the source data unit and either the first active storage data unit ASU_aaa or the second active storage data unit ASU_bbb. Source data units can be managed such that each reference in source blocks CHK1, CHK2, CHK4, and CHK5 is stored at a location in either the first active storage data unit ASU_aaa or the second active storage data unit ASU_bbb.

[0160] The source block CHK3, corresponding to the unique value c (UVc) also included in the source data unit and the deduplicated source data unit (not shown), can be classified as the first deduplicated block. The source block CHK3 can be stored in the target data unit ASU_aaa, and the source data unit can be managed such that the source block can refer to the location stored in the target data unit ASU_aaa.

[0161] refer to Figure 12In some example embodiments shown, after deduplication, the unique value c (UVc) can be inserted into the source Bloom filter of the first active storage data unit ASU_aaa, thereby updating the source Bloom filter. For example, the fourth bit of the source Bloom filter of the first active storage data unit ASU_aaa can be configured from the initial state (0) to the configuration state (1). Furthermore, the current size of the first active storage data unit ASU_aaa can be increased by S. In this case, S can be the capacity of the source block CHK3 stored in the first active storage data unit ASU_aaa.

[0162] Figure 14 and Figure 15 This is a diagram illustrating the deduplication process according to an example embodiment.

[0163] Figure 14 The active storage data unit table shown and Figure 15 The process of storing source blocks into target data units, as shown above, can be similar to the process described in the reference above. Figure 12 and Figure 13 Examples of descriptions. In the following descriptions, [the following may describe / describe / etc.]... Figure 12 and Figure 13 The differences.

[0164] Reference Figure 14 ,exist Figure 11 In some of the example embodiments shown, the Hamming similarity of the source Bloom filter with the Bloom filter of the first active storage data unit ASU_aaa and the second active storage data unit ASU_bbb can both be "4".

[0165] The calculated Hamming similarity score of 4 can be higher than the threshold of 1. However, multiple calculated Hamming similarities of 4 can be the highest possible. Figure 12 and Figure 13 In some example embodiments, the number of multiple active storage data units (the number is 2) may be equal to or less than the maximum allowable number N. See reference... Figure 9 As described in operation S280, a new third active storage data unit ASU_ccc can be generated, and the third active storage device data unit ASU_ccc can be selected as the target data unit.

[0166] Source blocks CHK1, CHK2, CHK4, and CHK5, corresponding to the unique values ​​a (UVa), b (UVb), d (UVd), and e (UVe), can be classified as decompiled blocks. These unique values ​​are also included in the source data unit and either the first active storage data unit ASU_aaa or the second active storage data unit ASU_bbb. The source data units are managed such that each of the source blocks CHK1, CHK2, CHK4, and CHK5 can be referenced to its location in either the first active storage data unit ASU_aaa or the second active storage data unit ASU_bbb.

[0167] The source block CHK3, corresponding to the unique value c (UVc) also included in the source data unit and the deduplicated source data unit (not shown), can be classified as the first deduplicated block. The source block CHK3 can be stored in the target data unit ASU_ccc, and the source data unit can be managed such that the source block can refer to the location stored in the target data unit ASU_ccc.

[0168] Reference Figure 14 In some example embodiments shown, after deduplication, the unique value c (UVc) can be inserted into the source Bloom filter of the third active storage data unit ASU_ccc, making the source Bloom filter updatable. In this case, the fourth bit of the source Bloom filter of the third active storage data unit ASU_ccc can be configured from the initial state (0) to the configuration state (1). Furthermore, the current size of the third active storage data unit ASU_ccc can be increased by S. In this case, S can be the capacity of the source block CHK3 stored in the third active storage data unit ASU_ccc.

[0169] Figure 16 and Figure 17 This is a diagram illustrating the deduplication process according to an example embodiment.

[0170] Will Figure 16 The active storage data unit table and Figure 17 The process of storing the source block into the target data unit, as shown above, can be referenced. Figure 14 and Figure 15 The examples described are similar. In the following description, the terms related to... Figure 14 and Figure 15 The differences will be explained.

[0171] Reference Figure 16 Duplicate data groups may include the first active storage data unit through the Nth active storage data unit. For Figure 11In some example embodiments shown, the Hamming similarity of the source Bloom filters in each of the first active storage data units ASU_aaa to the Nth active storage data unit ASU_zzz can all be "4".

[0172] The calculated Hamming similarity score of 4 can be higher than the threshold of 1. However, multiple calculated Hamming similarities of 4 can be the highest possible. Figure 16 and Figure 17 In some example embodiments, the number N of multiple active storage data units can be the maximum allowable number N. See reference... Figure 9 As described in operation S280, the Nth active storage data unit ASU_zzz, which has the smallest size among the first active storage data units ASU_aaa to the Nth active storage data units ASU_zzz, can be selected as the target data unit.

[0173] Source blocks CHK1, CHK2, CHK4, and CHK5 corresponding to unique values ​​a (UVa), b (UVb), d (UVd), and e (UVe) can be classified as decompiled blocks, wherein these unique values ​​can also be included in at least one of the source data units and the first active storage data units ASU_aaa to the Nth active storage data units ASU_zzz. Source data units are manageable such that each of the source blocks CHK1, CHK2, CHK4, and CHK5 can be referenced to a location stored in the first active storage data units ASU_aaa to the Nth active storage data units ASU_zzz.

[0174] The source block CHK3 corresponding to the unique value c (UVc) can be classified as the first deduplication block, wherein the unique value c (UVc) can be included in both the source data unit and the deduplicated source data unit (not shown). The source block CHK3 can be stored in the target data unit ASU_zzz, and the source data units can be managed such that the source block can refer to the position stored in the target data unit ASU_zzz.

[0175] Reference Figure 16 In some example embodiments shown, after deduplication, the unique value c (UVc) can be inserted into the source Bloom filter of the Nth active storage data unit table, allowing the source Bloom filter to be updated. In this case, the fourth bit of the source Bloom filter of the Nth active storage data unit ASU_zzz can be configured from the initial state (0) to the configuration state (1). Furthermore, the current size of the Nth active storage data unit ASU_zzz can be increased by S. In this case, S can be the capacity of the source block CHK3 stored in the Nth active storage data unit ASU_zzz.

[0176] Figure 18 and Figure 19 This is a diagram illustrating the deduplication process according to an example embodiment.

[0177] Will Figure 18 The active storage data unit table shown and Figure 19 The processing of the source block stored in the target data unit as shown can be similar to that described above. Figure 12 and Figure 13 Example of the description. The following will illustrate with... Figure 12 and Figure 13 The differences.

[0178] Reference Figure 18 ,for Figure 11 The source Bloom filter in the example embodiment shown can have a Hamming similarity of "1" with the Bloom filter of the first active storage data unit ASU_aaa. For Figure 11 The source Bloom filter in the example embodiment shown can have a Hamming similarity of "0" with the Bloom filter of the second active storage data unit ASU_bbb.

[0179] like Figure 18 and Figure 19 In some example embodiments, the threshold can be configured to "3". The calculated Hamming similarities of 1 and 0 may not exceed the threshold of 3. Figure 18 and Figure 19 In some example embodiments, the number 2 of multiple active storage data units may be equal to or less than the maximum allowable number N. See reference... Figure 9 As described in operation S280, a new third active storage data unit ASU_ccc can be generated, and the third active storage device data unit ASU_ccc can be selected as the target data unit.

[0180] The source block CHK2, corresponding to the unique value b (UVb) also included in the source data unit and the first active storage data unit ASU_aaa or the second active storage data unit ASU_bbb, can be classified as a decompiled block. The source data unit can be managed such that the source block CHK2 can reference the location stored in the first active storage data unit ASU_aaa.

[0181] Source blocks CHK1 and CHK3, corresponding to unique values ​​a (UVa) and c (UVc), can be classified as first deduplication blocks, where these unique values ​​are similarly included in both the source data unit and the deduplicated source data unit (not shown). Source blocks CHK1 and CHK3 can be stored in the target data unit ASU_ccc, and the source data unit can be managed so that the source blocks can reference their location stored in the target data unit ASU_ccc.

[0182] refer to Figure 18 In some example embodiments shown, after deduplication, the unique values ​​a (UVa) and c (UVc) of the active storage data unit can be inserted into the source Bloom filter of the third active storage data unit ASU_ccc, thereby updating the source Bloom filter. In this case, the fourth and first bits of the source Bloom filter of the third active storage data unit ASU_ccc can be configured from the initial state (0) to the configuration state (1). Furthermore, the current size of the third active storage data unit ASU_ccc can be increased by 2S. In this case, 2S can be the sum of the capacities of the source blocks CHK1 and CHK3 stored in the third active storage data unit ASU_ccc.

[0183] Figure 20 and Figure 21 This is a diagram illustrating the deduplication process according to an example embodiment.

[0184] Will Figure 20 The active storage data unit table shown and Figure 21 The processing of the source block stored in the target data unit shown above can be compared with the above reference. Figure 18 and Figure 19 Similar examples are described below. In the following descriptions, descriptions can be made of... Figure 18 and Figure 19 The differences.

[0185] Reference Figure 20 Duplicate data groups may include the first active storage data unit through the Nth active storage data unit. For Figure 11 In the example embodiment shown, the Hamming similarity of the source Bloom filter for each of the first active storage data units ASU_aaa to the Nth active storage data units ASU_zzz can be either "0" or "1".

[0186] The calculated Hamming similarity scores of 1 and 0 can be no higher than the threshold of 3. Figure 20 and Figure 21 In some example embodiments, the number N of multiple active storage data units can be the maximum allowable number N. See reference... Figure 9 As described in operation S270, among the first active storage data units ASU_aaa to the Nth active storage data units ASU_zzz, the Nth active storage data unit ASU_zzz with the smallest size can be selected as the target data unit.

[0187] A source block CHK2 corresponding to a unique value b (UVb) that is also included in at least one of the source data units and the first active storage data units ASU_aaa to the Nth active storage data units ASU_zzz can be classified as a decompiled block. The source data units can be managed such that the source block CHK2 can reference the location stored in the first active storage data unit ASU_aaa.

[0188] Source blocks CHK1 and CHK3, corresponding to the unique values ​​a (UVa) and c (UVc) also included in the source data unit and the deduplicated source data unit (not shown), can be classified as the first deduplicated blocks. Source blocks CHK1 and CHK3 can be stored in the target data unit ASU_ccc, and the source data unit can be managed so that the source blocks can reference the location stored in the target data unit ASU_ccc.

[0189] Reference Figure 20 In some example embodiments shown, after deduplication, unique values ​​a (UVa) and c (UVc) can be inserted into the source Bloom filter of the Nth source storage data unit ASU_zzz, allowing the source Bloom filter to be updated. In this case, the fourth and first bits of the source Bloom filter of the Nth active storage data unit ASU_zzz can be configured from the initial state (0) to the configuration state (1). Furthermore, the current size of the Nth active storage data unit ASU_zzz can be increased by 2S. In this case, 2S can be the sum of the capacities of the source blocks CHK1 and CHK3 stored in the Nth active storage data unit ASU_zzz.

[0190] Figure 22 This is a diagram illustrating the effect of deduplication according to some example embodiments.

[0191] Figure 22 Additional read operations can be illustrated in the comparative examples and the first to third example embodiments. An additional read operation can be the average number of additional read operations performed on the source block required to output the deduplicated source data unit. As the fragmentation of source data units becomes more severe, source blocks can be physically distributed and stored, potentially increasing the number of read operations performed to output the source data unit.

[0192] In the deduplication process of the first to third example embodiments, at least one first deduplication block included in the source data unit can be stored in the active storage data unit based on relevance. The active storage data unit storing the first deduplication block can be different for each source data unit.

[0193] In the first to third example embodiments, the maximum allowed number of active storage data units may differ. In the first example embodiment, up to 10 active storage data units may be allowed. In the second example embodiment, up to 100 active storage data units may be allowed. In the third example embodiment, up to 1000 active storage data units may be allowed.

[0194] In the deduplication of the comparative example and other example embodiments, the first deduplication block may be stored sequentially in storage data units. When the storage capacity of a storage data unit is exceeded, a new storage data unit may be generated. Thereafter, the first deduplication block may be stored sequentially in the new storage data unit. Unlike the example embodiments, the location for storing the first deduplication block may not be based on correlation.

[0195] exist Figure 22 In (a), the storage capacity of the storage data unit in the comparison example and the active storage data unit in the first to third example embodiments can be 4MB. Figure 22 In (b), the storage capacity of the storage data unit in the comparative example and the active storage data unit in the first to third example embodiments can be 16MB. Figure 22 In (c), the storage capacity of the storage data unit in the comparison example and the active storage data unit in the first to third example embodiments may be 64MB.

[0196] Reference Figure 22 In steps (a) to (c), compared to the additional read operations in the comparative examples, the additional read operations in the first to third example embodiments can be reduced. By performing deduplication in the first to third example embodiments, the spatial locality of each block can be improved, and fragmentation can be mitigated.

[0197] Reference Figure 22 In (a) to (c), the larger the storage capacity of the storage data unit in the comparison example and the active storage data unit in the first to third example embodiments, the fewer additional read operations are required. As the storage capacity of the active storage data unit in the first to third example embodiments increases, the size of the Bloom filter can also increase. As the size of the Bloom filter increases, the accuracy of Hamming similarity determination improves, further reducing additional read operations.

[0198] Refer to each Figure 22 In the comparative examples (a) to (c) and the first to third example embodiments, when the block size is small, the number of additional read operations can increase. When the block size is small, the source data unit can be divided into more blocks. Therefore, the deduplication rate of the blocks may increase, and the spatial locality of each block may deteriorate.

[0199] Figure 23 This is a diagram illustrating the effect of deduplication according to an example embodiment.

[0200] Figure 23 The read performance of a first comparison example, a second comparison example, and some example embodiments may be shown. Read performance may include read operations and network bandwidth.

[0201] In some example embodiments of deduplication, at least one first deduplication block included in the source data unit may be stored in the active storage data unit based on relevance. The active storage data unit storing the first deduplication block may be different for each source data unit. When deduplication is performed in the first to third example embodiments, the spatial locality of each block can be improved, and fragmentation can be mitigated.

[0202] In the first comparative example, unlike the example embodiment, deduplication may not be performed. Multiple identical blocks can be stored in the storage device, and block fragmentation may not occur. However, the storage capacity utilization efficiency of the storage system may be reduced.

[0203] In the deduplication of the second comparative example, unlike the example embodiment, the first deduplication block can be stored separately. In other words, the first deduplication block can be stored out of order in the data units and can be stored block by block in a predetermined space. Therefore, the spatial locality of each block may deteriorate, potentially exacerbating fragmentation.

[0204] Second comparison example and Figure 23 In each of the example embodiments, read operations and network bandwidth may be included for the deduplication rate, and the deduplication rate may be 25%, 50%, and 75%. As the deduplication rate increases, read operations may increase and network bandwidth may decrease.

[0205] Compared to the read operations in the first comparison example, the read operations in the second comparison example and some example embodiments can be further increased. In other words, compared to the read performance in the first comparison example, the read performance in the second comparison example and some example embodiments can be further reduced. Therefore, compared to the network bandwidth in the first comparison example, the network bandwidth in the second comparison example and some example embodiments can be further reduced.

[0206] In some example embodiments, the average number of read operations for the overall deduplication rate may be lower than in the second comparison example. Based on the read operations in the first comparison example, the increase in read operations in some example embodiments may be less than the increase in read operations in the second comparison example.

[0207] In other words, the read performance in some example embodiments may be superior to that in the second comparative example. In the example embodiments, deduplication based on relevance improves the spatial locality of each chunk, thereby mitigating fragmentation. Therefore, read performance in some example embodiments can be improved.

[0208] In the example embodiments, the average network bandwidth for the overall deduplication rate may be higher than the average network bandwidth in the second comparison example. Based on the network bandwidth in the first comparison example, the reduction in network bandwidth in some example embodiments may be less than the reduction in network bandwidth in the second comparison example.

[0209] Figure 24 This is a diagram illustrating a system of application storage devices according to some example embodiments.

[0210] Figure 24 System 1000 can be implemented as or included in a mobile system (such as one or more of a mobile phone, smartphone, tablet computer, wearable device, healthcare device, or Internet of Things (IoT) device). However, Figure 24 The system 1000 is not limited to a mobile system, and may also be implemented as or included in one or more of a personal computer, laptop computer, server, media player or automotive device such as a navigation device.

[0211] Reference Figure 24 The system 1000 may include a main processor 1100, memories 1200a and 1200b and storage devices 1300a and 1300b, and may also include one or more of an image capture device 1410, a user input device 1420, a sensor 1430, a communication device 1440, a display 1450, a speaker 1460, a power supply device 1470 and a connection interface 1480.

[0212] The main processor 1100 controls the overall operation of the system 1000, and more specifically, controls the operation of other components included in the system 1000. The main processor 1100 can be implemented as a general-purpose processor, a special-purpose processor, or an application processor.

[0213] The main processor 1100 may include one or more CPU cores 1110, and may also include a controller 1120 for controlling memories 1200a and 1200b and / or storage devices 1300a and 1300b. According to an embodiment, the main processor 1100 may also include an accelerator 1130, which may be dedicated circuitry for high-speed data operations such as artificial intelligence (AI) data operations. The accelerator 1130 may include a graphics processing unit (GPU), a neural processing unit (NPU), and / or a data processing unit (DPU), and may be implemented as a chip physically independent of other components of the main processor 1100.

[0214] Memory 1200a and 1200b may be used as the main memory device of system 1000 and may include volatile memory such as SRAM and / or DRAM, but may also include non-volatile memory such as flash memory, PRAM and / or RRAM. Memory 1200a and 1200b may also be implemented in the same package as main processor 1100.

[0215] Storage devices 1300a and 1300b can be used as non-volatile storage devices for storing data regardless of whether it is powered on, and can have a relatively large storage capacity compared to memories 1200a and 1200b. Storage devices 1300a and 1300b may include controllers 1310a and 1310b and non-volatile memory (NVM) 1320a and 1320b for storing data under the control of controllers 1310a and 1310b.

[0216] Storage devices 1300a and 1300b may be included in system 1000 in a physically separate state from main processor 1100, or may be implemented in the same package as main processor 1100. Furthermore, storage devices 1300a and 1300b may be in the form of SSDs (Solid State Devices) or memory cards, and may be detachably coupled to other components of system 1000 via interfaces such as connection interface 1480, described later. Standard specifications such as Universal Flash Memory (UFS), embedded multimedia card (eMMC), or high-speed non-volatile memory (NVMe) may be applied to storage devices 1300a and 1300b, but some example embodiments are not limited thereto.

[0217] According to some example embodiments, in storage devices 1300a and 1300b, at least one first de-organized block included in the source data unit can be stored at a location selected based on relevance. In other words, at least one first de-organized block can be stored at contiguous memory addresses of the storage data units. Therefore, the spatial locality of each block can be improved, thereby mitigating fragmentation. Furthermore, the read performance of storage devices 1300a and 1300b can be improved.

[0218] The image capture device 1410 can capture still images or moving images and can be implemented as a camera, video camera and / or webcam.

[0219] User input device 1420 can receive various types of data input from the user of system 1000, and can be implemented as or included in one or more of a touchpad, keyboard, mouse and / or microphone.

[0220] Sensor 1430 can sense various types of physical quantities obtained from external entities of system 1000 and can convert the sensed physical quantities into electrical signals. Sensor 1430 may be a temperature sensor, pressure sensor, light sensor, position sensor, acceleration sensor, biosensor, and / or gyroscope sensor.

[0221] The communication device 1440 can perform signal transmission and reception between other devices outside the system 1000 according to various communication protocols. The communication device 1440 may be implemented as including an antenna, a transceiver, and / or a modem.

[0222] The display 1450 and the speaker 1460 can be used as output devices to output visual and auditory information to the user of the system 1000, respectively.

[0223] The power supply unit 1470 can suitably convert power supplied from a battery (not shown) embedded in the system 1000 and / or an external power source, and provide power to each component of the system 1000.

[0224] Connection interface 1480 provides connectivity between system 1000 and external devices connected to and exchanging data with system 1000. Connection interface 1480 can be implemented through various interface methods, such as one or more of Advanced Technology Attachment (ATA), Serial ATA (SATA), External SATA (e-SATA), Small Computer Interface (SCSI), Serial Attached SCSI (SAS), Peripheral Component Interconnect (PCI), High Speed ​​PCI (PCIe), NVMe, IEEE 1394, Universal Serial Bus (USB), Secure Digital Card (SD), Multimedia Card (MMC), eMMC, UFS, Embedded Universal Flash Memory (eUFS), and Compact Flash Memory (CF) card interfaces.

[0225] According to the example embodiment described above, at least one block that has been deduplicated from source data units received from the host can be stored in a target data unit, which is one of a plurality of active storage data units. In this case, for the source data unit, a Bloom filter and Hamming similarity can be used to calculate the similarity with the plurality of active storage data units, and the target data unit can be selected from the plurality of active storage data units based on the similarity. Therefore, the fragmentation of source blocks can be mitigated or improved, and the number of times source blocks are read to output deduplicated source data units can be reduced or minimized.

[0226] Any of the elements and / or functional blocks disclosed above may include or be implemented in processing circuitry, such as hardware including logic circuitry, hardware / software combinations (e.g., a processor or combination thereof executing software). More specifically, processing circuitry may include, but is not limited to, a central processing unit (CPU), an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field-programmable gate array (FPGA), a system-on-a-chip (SoC), a programmable logic unit, a microprocessor, an application-specific integrated circuit (ASIC), etc. Processing circuitry may include electrical components such as transistors, resistors, capacitors, etc. Processing circuitry may include electronic components such as logic gates, including at least one of AND gates, OR gates, NAND gates, NOT gates, etc.

[0227] While some exemplary embodiments have been shown and described above, it will be apparent to those skilled in the art that modifications and variations can be made without departing from the scope of this disclosure as defined by the appended claims. Furthermore, the exemplary embodiments are not necessarily mutually exclusive. For example, some exemplary embodiments may include one or more features described with reference to one or more accompanying drawings, and may also include one or more other features described with reference to one or more other accompanying drawings.

Claims

1. A control method of a storage system, the control method comprising: dividing a source data unit among a plurality of source data units received from a host into a plurality of source chunks, the source data unit being a target of deduplication; generating a unique value for each of the plurality of source chunks; inserting the unique value into a source Bloom filter of the source data unit and updating the source Bloom filter; calculating a Hamming similarity with a storage Bloom filter of each of a plurality of active storage data units for the source Bloom filter; selecting an active storage data unit as a target data unit based on a highest Hamming similarity among the plurality of active storage data units; classifying each of the plurality of source chunks as one of a first deduplicated chunk, a deduplicated chunk, or a unique chunk using the unique value; and storing at least one of the first deduplicated chunk in the target data unit.

2. The control method of claim 1, further comprising: inserting the first deduplicated chunk into a target Bloom filter of the target data unit and updating bits of the target Bloom filter. each of the unique values has a fixed size, and each of the unique values is generated by a hash function.

3. The control method of claim 1, wherein, a size of the source Bloom filter and a size of the storage Bloom filter are the same.

4. The control method of claim 1, wherein, the size of the source Bloom filter and the size of the storage Bloom filter are greater than a number of chunks allowed per each of the plurality of active storage data units.

5. The control method of claim 4, wherein, allowable sizes of the plurality of active storage data units are the same.

6. The control method of claim 1, wherein, 7. The control method of claim 6, further comprising: when the first deduplicated chunk is added to the target data unit and a size of the target data unit becomes greater than the allowable size, changing the target data unit to a storage data unit and excluding the changed unit from the plurality of active storage data units. classifying each of the plurality of source chunks comprises:

8. The control method of claim 7, wherein, in response to the source data unit including a same unique value as a source data unit among the plurality of source data units that has been deduplicated, classifying a source chunk corresponding to the same unique value as the first deduplicated chunk; in response to the source data unit including a same unique value as the plurality of active storage data units or the storage data unit, classifying a source chunk corresponding to the same unique value as the deduplicated chunk; and classifying chunks among the plurality of source chunks other than the first deduplicated chunk and the deduplicated chunk as the unique chunk.

9. The control method of claim 8, further comprising: accessing the first deduplicated chunk and the deduplicated chunk by referring to a location of a stored chunk corresponding to the same unique value. the Hamming similarity indicates a number of locations in which each of bits in a same location in the source Bloom filter and the storage Bloom filter has a value of logical "1".

10. The control method of claim 1, wherein, the higher the Hamming similarity, the greater a number of chunks having a same unique value in the source data unit and the active storage data unit.

11. The control method of claim 10, wherein, 12. A control method of a storage system, the control method comprising: ​ dividing a source data unit, received from a host, that is a target of deduplication among a plurality of source data units into a plurality of source chunks; generating a unique value for each of the plurality of source chunks; inserting the unique value into a source Bloom filter of the source data unit and updating the source Bloom filter; calculating, for the source Bloom filter, a Hamming similarity to a storage Bloom filter of each of a plurality of active storage data units in which chunks of the target of deduplication are stored; selecting a target data unit among the plurality of active storage data units using the Hamming similarity; and adding at least one first deduplicated chunk among the plurality of source chunks to the target data unit.

13. The control method of claim 12, wherein, Selecting the target data unit includes selecting, responsive to one of the Hamming similarities being highest, an active storage data unit among the plurality of active storage data units based on the highest Hamming similarity as the target data unit.

14. The control method of claim 12, wherein, Selecting the target data unit includes selecting, responsive to a plurality of the Hamming similarities being highest or responsive to the Hamming similarities being below a threshold, the target data unit using a number of the plurality of active storage data units.

15. The control method of claim 14, wherein, Selecting the target data unit includes selecting, responsive to the number of the plurality of active storage data units being a maximum allowable number, an active storage data unit among the plurality of active storage data units based on a smallest size as the target data unit.

16. The control method of claim 14, wherein, Selecting the target data unit includes generating a new active storage data unit and selecting the new active storage data unit as the target data unit, responsive to the number of the plurality of active storage data units being equal to or less than a maximum allowable number.

17. The control method of claim 14, wherein, The threshold varies according to the number of the plurality of active storage data units.

18. The control method of claim 14, wherein, The threshold varies according to a distribution of Hamming similarities.

19. A control method of a storage system, the control method comprising: dividing a source data unit that is a target of deduplication into a plurality of source chunks and generating a unique value for each of the plurality of source chunks; inserting the unique value into a source Bloom filter of the source data unit and updating the source Bloom filter; calculating, for the source Bloom filter, a Hamming similarity to a storage Bloom filter of each of a plurality of active storage data units in which chunks of the target of deduplication are stored; selecting a target data unit from among the plurality of active storage data units using the Hamming similarity; classifying each of the plurality of source chunks as one of a first deduplicated chunk, a deduplicated chunk, or a unique chunk using the unique value; adding the first deduplicated chunk to the target data unit and referencing a location of the target data unit to access the first deduplicated chunk; and inserting the first deduplicated chunk into a target Bloom filter of the target data unit and updating a bit of the target Bloom filter.

20. The control method of claim 19, wherein, Selecting the target data unit includes: selecting, in response to one of the Hamming similarities being the highest, an active storage data unit among the plurality of active storage data units based on the highest Hamming similarity as the target data unit; selecting, in response to a number of the plurality of active storage data units being the maximum allowable number, an active storage data unit among the plurality of active storage data units based on a minimum size as the target data unit; and selecting, in response to a number of the plurality of active storage data units being equal to or less than the maximum allowable number, a new storage data unit and selecting the new storage data unit as the target data unit.

Images