Device and method for defragmentation in deduplication

Abstract

The present disclosure relates to a device and method for storing duplicate data blocks. Specifically, the present disclosure provides a device for eliminating the need for defragmentation in a deduplication backup system. The device is configured to store one or more compressed containers, wherein each container includes a plurality of fragments, wherein one or more duplicate data blocks are stored in the plurality of fragments of the one or more containers. The device is also configured to delete the first duplicate data block stored in the fragment by decompressing the fragment, replacing the first duplicate data block with zeros in the fragment, and recompressing the fragment. In this way, a solution is provided to delete obsolete data without metadata updates.

Description

Technical Field

[0001] The present disclosure relates to a device and method for storing data, in particular for storing duplicate data. In order to solve the fragmentation problem caused during the deduplication process, the present disclosure provides a solution to simplify the defragmentation process. To this end, the present disclosure proposes a device and method for allowing defragmentation in a backup storage to be eliminated. Background Art

[0002] Deduplication is the process of eliminating duplicate data in storage. Due to the nature of how data is saved in deduplication storage, deduplication often suffers from fragmentation issues. This is fragmentation at the software level, done on top of lower-level hardware that also does some defragmentation. Fragmentation is also a cause of further read amplification.

[0003] In deduplication, user data is stored as pointers to the actual data. The actual data is stored in some kind of container. When the user deletes data, the container's blocks become obsolete. Therefore, when deduplication is performed, there are often obsolete blocks of data in the repository that need to be collected. This usually requires a lot of work to be handled by offline / background processes, including a lot of calculations and sometimes complex procedures to allow the reference counts to be updated in a way that prevents errors.

[0004] Traditionally, to redeem space, data can be copied to another location in an attempt to maintain locality of reference, or the space can be reused for new data to prevent the complexity of copying. Handling obsolete data in the conventional way might involve waiting until a certain amount of data in a container or region is obsolete, then copying the remaining data to a new container and updating all references from the old location to the new location. This process requires locking the container's metadata, the container itself, and possibly other structures. The defragmentation process is difficult to perform in parallel, is therefore prone to error, and can also affect backup and restore speeds. Summary of the invention

[0005] In view of the above problems, the purpose of the embodiments of the present disclosure is to provide a device and method for eliminating the need for defragmentation in a deduplication backup system. In particular, one purpose is to avoid locking and / or updating metadata references when deleting obsolete data. Another purpose is to redeem space on the application layer. In addition, it is also desirable to be able to perform normal backup and restore operations simultaneously.

[0006] The embodiments of the present disclosure are achieved by means of the attached independent claims. Advantageous implementations of the embodiments of the present disclosure are further defined in the dependent claims.

[0007] A first aspect of the present disclosure provides a device for storing duplicate data blocks, the device being configured to: store one or more compressed containers, wherein each container includes a plurality of fragments, wherein one or more duplicate data blocks are stored in the plurality of fragments of the one or more containers; and delete the first duplicate data block stored in the fragment by decompressing the fragment, replacing the first duplicate data block with zeros in the fragment, and recompressing the fragment.

[0008] The first aspect of the device has the advantage of reducing or eliminating the need for defragmentation in a deduplication backup system. This is an advantage of zero replacement and fragment compression.

[0009] In an embodiment of the first aspect, the device is further configured to maintain metadata of one or more stored duplicate data chunks, wherein the metadata of each stored duplicate data chunk includes a reference to a location where the duplicate data chunk is stored.

[0010] Metadata can be used to map stored blocks of data to their physical addresses.

[0011] In an embodiment of the first aspect, each container further includes fragment metadata for each fragment, the fragment metadata including a location of the data fragment in the container.

[0012] Knowing the fragment metadata, the stored data fragment can be found.

[0013] In an embodiment of the first aspect, the device is further configured to: obtain one or more repeated data blocks; fill the one or more repeated data blocks into multiple fragments of one or more containers; and compress the multiple fragments of the one or more containers.

[0014] The backup system based on deduplication can divide the backup stream into data blocks of variable size. The data blocks to be written can be aggregated into a container, especially into fragments of the container. According to an embodiment of the present invention, each fragment is compressed in the container.

[0015] In an embodiment of the first aspect, the device is further configured to retrieve the stored first repeated data chunk according to metadata of the first repeated data chunk.

[0016] For example, during data recovery, data blocks are located and read based on the metadata.

[0017] In an embodiment of the first aspect, each repetitive data block is associated with a reference value indicating how often the repetitive data block is being updated.

[0018] Optionally, the reference value may be used to distinguish hot data from cold data in the memory.

[0019] In an embodiment of the first aspect, the device is further configured to store repeated data blocks having reference values ​​higher than a preset value and repeated data blocks having reference values ​​lower than a preset value in different containers.

[0020] Preferably, it may be desirable to keep cold data and hot data in different containers, respectively. The preset value may be configured based on a specific implementation or requirement.

[0021] In an embodiment of the first aspect, the device includes a disk, and the one or more containers are stored on the disk.

[0022] A second aspect of the present disclosure provides a method for storing duplicate data blocks, comprising: storing one or more compressed containers, wherein each container includes multiple fragments, wherein one or more duplicate data blocks are stored in the multiple fragments of the one or more containers; and decompressing the fragments, replacing the first duplicate data block with zeros in the fragments, and recompressing the fragments to delete the first duplicate data block stored in the fragments.

[0023] In an embodiment of the second aspect, the method further comprises maintaining metadata for one or more stored duplicate data chunks, wherein the metadata for each stored duplicate data chunk comprises a reference to a location where the duplicate data chunk is stored.

[0024] In an embodiment of the second aspect, each container further comprises fragment metadata for each fragment, the fragment metadata comprising a location of the data fragment in the container.

[0025] In an embodiment of the second aspect, the method further includes: obtaining one or more repeated data blocks; filling the one or more repeated data blocks into multiple fragments of one or more containers; and compressing the multiple fragments of the one or more containers.

[0026] In an embodiment of the second aspect, the method further comprises retrieving the stored first repeated data block according to metadata of the first repeated data block.

[0027] In an embodiment of the second aspect, each repetitive data block is associated with a reference value indicating how often the repetitive data block is being updated.

[0028] In an embodiment of the second aspect, the method includes storing repeated data blocks having reference values ​​higher than a preset value and repeated data blocks having reference values ​​lower than a preset value in different containers.

[0029] The method of the second aspect and its embodiments provide the same advantages and effects as the apparatus of the first aspect and its corresponding embodiments.

[0030] A third aspect of the present disclosure provides a computer program comprising program code, for executing the method according to the second aspect or any embodiment thereof when implemented on a processor.

[0031] A fourth aspect of the present disclosure provides a non-transitory storage medium storing executable program code, which, when executed by a processor, causes the method according to the second aspect or any embodiment thereof to be performed.

[0032] It should be noted that all devices, elements, units and devices described in this application can be implemented in software or hardware elements or any type of combination thereof. All steps performed by various entities described in this application and functions performed by various entities are intended to indicate that the corresponding entity is applicable to or used to perform the corresponding steps and functions. Even if in the description of the following specific embodiments, the specific functions or steps performed by the external entity are not reflected in the description of the specific detailed elements of the entity performing the specific steps or functions, the technician should be aware that these methods and functions can be implemented in the corresponding software or hardware elements or any type of combination thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The above aspects and implementations will be explained in the following description of specific embodiments in conjunction with the accompanying drawings, in which

[0034] Figure 1 A device according to an embodiment of the invention is shown.

[0035] Figure 2 A typical deduplication structure is shown.

[0036] Figure 3 shows an example of the data structure when copying data to a new container.

[0037] Figure 4 An example of a data structure when reusing redeemed space is shown.

[0038] Figure 5 The file structure according to an embodiment of the present invention is shown.

[0039] Figure 6 A method according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0040] With reference to the accompanying drawings, illustrative embodiments of methods, devices and program products for efficient packet transmission in a communication system are described. Although the description provides detailed examples of possible implementations, it should be noted that these details are only exemplary and in no way limit the scope of the present application.

[0041] In addition, one embodiment / example may indicate other embodiments / examples. For example, any description including but not limited to terms, elements, processes, instructions and / or technical advantages mentioned in one embodiment / example is applicable to other embodiments / examples.

[0042] Figure 1 A device 100 according to an embodiment of the present invention is shown. The device 100 is adapted to store duplicate data blocks. The device 100 is configured to store one or more compressed containers 101. In particular, each container 101 comprises a plurality of fragments, wherein the one or more duplicate data blocks are stored in a plurality of fragments 101 of the one or more containers. Furthermore, the device 100 is configured to delete the first duplicate data block stored in the fragment, in particular by decompressing the fragment, replacing the first duplicate data block with zeros in the fragment, and recompressing the fragment.

[0043] The device 100 may include a processing circuit (not shown) configured to perform, conduct or initiate various operations of the device 100 described herein. The processing circuit may include hardware and software. The hardware may include analog circuits or digital circuits, or analog and digital circuits. The digital circuit may include components such as an application-specific integrated circuit (ASIC), a field programmable array (FPGA), a digital signal processor (DSP) or a multi-function processor. The device 100 may further include a memory circuit that stores one or more instructions that can be executed by a processor or by a processing circuit, particularly under the control of software. For example, the memory circuit may include a non-transitory storage medium storing executable software code that, when executed by a processor or a processing circuit, causes various operations of the device 100 to be performed. In one embodiment, the processing circuit includes one or more processors and a non-transitory memory connected to the one or more processors. The non-transitory memory may carry an executable program code that, when executed by one or more processors, causes the device 100 to perform, conduct or initiate the operations or methods described herein.

[0044] It is worth noting that the deduplication process stores some form of reference, such as a pointer to the location where the duplicate data has been stored, as Figure 2 An example of a typical deduplication structure is shown. References to the storage locations of duplicate data chunks are often referred to as metadata.

[0045] In particular, according to an embodiment of the present invention, the device 100 is further configured to maintain metadata of one or more stored duplicate data chunks, wherein the metadata of each stored duplicate data chunk includes a reference to a location where the duplicate data chunk is stored.

[0046] Deduplication-based backup systems divide backups into variable-sized chunks or fragments. When users delete data, chunks of the container become obsolete. When obsolete data is handled with conventional solutions, the remaining data is copied or moved to new containers, such as Figure 3 As shown in the example of . In this way, all references from the old location need to be updated to the new location. When changing the data location, all metadata referencing the data location needs to be locked until the transaction is completed. Otherwise, pointers to the moved data can remain. This also affects the backup and restore speed.

[0047] For example, if a container initially has blocks from backup X: X1, X2, X3, X4, and X5, and then X2, X3, and X5 are deleted. Subsequently, if some data from another backup Y is written to the container, the data may already be in containers: X1 Y1 Y2 X4 Y3, and some of them are deleted. If the container is reused, you may end up with something like Figure 4 The situation shown.

[0048] like Figure 4 The example shown illustrates the effect of reusing containers over a long period of time in a system where the data in the reused containers comes from a variety of different and potentially unrelated backups, and each backup user data is also spread across a large number of containers.

[0049] It is worth noting that when performing data backups in this situation, both throughput performance and deduplication performance will be reduced. When performing data restores, random blocks need to be read from different locations, which is a known issue and bottleneck that slows down the restore of deduplicated data.

[0050] The disclosed embodiment designs a backup and delete deduplication system process to redeem space at the application level, which eliminates the need for defragmentation. It is worth noting that each container is divided into multiple parts. According to an embodiment of the present invention, each container may also include fragment metadata for each fragment, and the fragment metadata includes the location of the data fragment in the container.

[0051] Each fragment can be identified with an identifier. Fragment metadata can be used to map the fragment identifier to its physical address.

[0052] During the backup, the data blocks that need to be written are aggregated into containers to preserve the spatial locality of the backup stream. Optionally, according to an embodiment of the present invention, the device 100 can be further configured to obtain one or more duplicate data blocks. Therefore, the device 100 can be configured to fill the one or more duplicate data blocks into multiple fragments of one or more containers 101. In addition, the device 100 can be configured to compress multiple fragments of one or more containers 101.

[0053] During restoration, the backup stream is reconstructed based on the data. Optionally, according to an embodiment of the present invention, the device 100 may be further configured to retrieve the stored first duplicate data block based on metadata of the first duplicate data block.

[0054] Optionally, according to an embodiment of the present invention, the container stored in the device 100 may be a large container. For example, the size of the container may be greater than 50MB. Each fragment is compressed in the device 100. The header of the fragment may include the position of the fragment in the file (or object). With this scheme, data can be deleted from the physical storage by reading the fragment containing the exchanged block, decompressing the fragment, replacing the exchanged part with zero, compressing the old fragment and replacing it with the recompressed fragment. In particular, replacing the exchanged part with zero means replacing each bit of the data block that needs to be deleted with zero. As Figure 5 The file structure shown according to the embodiment of the present disclosure saves space in the compression container 101 after the deleted data is replaced by zeros and further compressed.

[0055] In this way, since the location of the fragment remains unchanged, there is no need to update the metadata reference. In addition, the locality of the reference is preserved. Since the areas belonging to the same backup are kept together, read amplification is improved. Read amplification refers to the number of disk reads per query, which is the ratio between the amount of data requested and the amount of data actually required to be read. It is worth noting that when the redeemed space is reused in a conventional manner, the locality of the reference will be lost when the container is reused. According to the embodiment provided by the present invention, the container is not reused, so the locality of the reference can be maintained.

[0056] This also helps with deduplication because the locality of data, especially cold data, is preserved. It is worth noting that cold data refers to data that is saved across multiple (e.g., hundreds) of backups, and hot data refers to data that is frequently replaced between backups. The process also implicitly stores cold data and hot data in different containers.

[0057] In particular, according to an embodiment of the present invention, each repeated data block is associated with a reference value indicating the frequency with which the repeated data block is being updated.

[0058] Preferably, according to an embodiment of the present invention, the device 100 may be further configured to store duplicate data blocks having reference values ​​higher than a preset value and duplicate data blocks having reference values ​​lower than a preset value in different containers.

[0059] As can be seen, the process requires no locking and no bad path handling. The process of reclaiming data can be done independently on any container. No locking means backups and data recovery can be done as usual. No metadata locks are required and there are no common areas to update, so it can be done in parallel and / or distributed among any number of threads, processes, or machines.

[0060] Additionally, even for some weird edge cases, e.g., two entities working on the same container, there is no risk of data corruption and no special case handling is required.

[0061] In the above embodiment, the cross-references are stored in the system and can be verified or reconstructed independently and in parallel. However, the scheme can also work in any other deduplication system.

[0062] Possibly, according to an embodiment of the present invention, the device 100 comprises a disk, and one or more containers are stored on the disk.

[0063] Figure 6 A method 600 for storing duplicate data blocks according to an embodiment of the present disclosure is shown. In a specific embodiment of the present disclosure, the method 300 is composed of Figure 1 The method 600 includes: storing one or more compressed containers, each container including a plurality of segments, wherein one or more duplicate data blocks are stored in the plurality of segments of the one or more containers in step 601; and decompressing the segments, replacing the first duplicate data block with zeros in the segments, and recompressing the segments to delete the first duplicate data block stored in the segments in step 602.

[0064] Optionally, according to an embodiment of the present disclosure, method 600 further includes maintaining metadata of one or more stored duplicate data chunks, wherein the metadata of each stored duplicate data chunk includes a reference to a location where the duplicate data chunk is stored.

[0065] Optionally, according to an embodiment of the present invention, each container further includes fragment metadata for each fragment, and the fragment metadata includes a location of the data fragment in the container.

[0066] Optionally, according to an embodiment of the present disclosure, method 600 further includes obtaining one or more duplicate data blocks. Therefore, the method may include filling the one or more duplicate data blocks into multiple fragments of one or more containers. Then, the method also includes compressing the multiple fragments of the one or more containers.

[0067] Optionally, according to an embodiment of the present disclosure, method 600 further includes retrieving the stored first duplicate data block according to metadata of the first duplicate data block.

[0068] Optionally, according to an embodiment of the present disclosure, each repeated data block is associated with a reference value indicating a frequency at which the repeated data block is being updated.

[0069] Optionally, according to an embodiment of the present disclosure, method 600 may include storing duplicate data blocks having reference values ​​higher than a preset value and duplicate data blocks having reference values ​​lower than a preset value in different containers.

[0070] Deduplication systems are specialized and require a lot of trade-offs to be considered when designing such systems. Based on the design proposed in the embodiments of the present disclosure, the following effects can be achieved:

[0071] 1. Eliminate the need for defragmentation.

[0072] 2. Delete obsolete data without metadata update.

[0073] 3. Reduce overall read amplification and increase data locality.

[0074] 4. Increase the lifecycle of containers with cold data and reduce the lifecycle of containers with hot data.

[0075] 5. No need for offline defragmentation.

[0076] 6. The process of each container is completely independent, so it can be fully distributed and parallelized without much effort.

[0077] The invention has been described in conjunction with various embodiments as examples and implementations. However, by studying the drawings, the present disclosure and the independent claims, a person skilled in the art can understand and implement other variations and implement the claimed invention. In the claims and in the specification, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality. A single element or other unit may perform the functions of several entities or items recited in the claims. The fact that certain means are recited in mutually different dependent claims does not indicate that a combination of these means cannot be used in an advantageous implementation.

[0078] In addition, any method according to the embodiments of the present disclosure can be implemented in a computer program having a code component, which, when run by a processing component, causes the processing component to perform the steps of the method. The computer program is included in a computer-readable medium of a computer program product. The computer-readable medium basically includes any memory, such as a ROM (read-only memory), a PROM (programmable read-only memory), an EPROM (erasable PROM), a flash memory, an EEPROM (electrically erasable PROM) or a hard disk drive.

[0079] In addition, those skilled in the art recognize that embodiments of the device 100 include necessary communication capabilities for executing the solution in the form of, for example, functions, components, units, elements, etc. Examples of other such components, units, elements, and functions are: processors, memories, buffers, control logic, encoders, decoders, rate matchers, down-rate matchers, mapping units, multipliers, decision units, selection units, switches, interleavers, deinterleavers, modulators, demodulators, inputs, outputs, antennas, amplifiers, receiver units, transmitter units, DSPs, trellis coded modulation (TCM) encoders, TCM decoders, power supply units, power supplies, communication interfaces, communication protocols, etc. are appropriately arranged together to execute the solution.

[0080] In particular, the processor of the device 100 may include, for example, one or more instances of a central processing unit (CPU), a processing unit, a processing circuit, a processor, an application specific integrated circuit (ASIC), a microprocessor, or other processing logic that can interpret and execute instructions. Thus, the expression "processor" may refer to a processing circuit that includes a plurality of processing circuits, for example, any, some, or all of the above. The processing circuit may further perform data processing functions for input, output, and processing of data, including data buffering and device control functions, such as call processing control, user interface control, etc.

Claims

1. A device (100) for storing duplicate data blocks, the device (100) being configured to: One or more compressed containers (101) are stored, wherein Each container (101) comprises a plurality of fragments, wherein one or more repeated data chunks are stored in the plurality of fragments of the one or more containers (101); and decompressing the segment, replacing the first duplicate data block with zeros in the segment, recompressing the segment, and deleting the first duplicate data block stored in the segment; The device (100) is also configured to: Metadata for the one or more stored duplicate data chunks is maintained, wherein the metadata for each stored duplicate data chunk includes a reference to a location where the duplicate data chunk is stored.

2. The device (100) according to claim 1, wherein Each container (101) further comprises fragment metadata for each fragment, the fragment metadata comprising the position of the fragment in the container (101).

3. The device (100) according to claim 1, configured to: obtaining the one or more repeated data blocks; filling the one or more repeated data blocks into a plurality of fragments of the one or more containers (101); and The plurality of fragments of the one or more containers (101) are compressed.

4. The device (100) according to claim 1, configured to: The stored first repeated data block is retrieved according to the metadata of the first repeated data block.

5. The device (100) according to claim 1, wherein Each repeated data block is associated with a reference value indicating how often the repeated data block is being updated.

6. The device (100) according to claim 5, configured to: Repeated data blocks with reference values ​​higher than a preset value and repeated data blocks with reference values ​​lower than the preset value are stored in different containers (101).

7. The device (100) according to claim 1, wherein The device (100) comprises a disk, and the one or more containers (101) are stored on the disk.

8. A method (600) for storing duplicate data blocks, comprising: storing one or more compressed containers (101), wherein each container comprises a plurality of fragments, wherein one or more duplicate data blocks are stored in the plurality of fragments of the one or more containers (101); and Recompressing the segment to delete (602) the first duplicate data block stored in the segment by decompressing the segment, replacing the first duplicate data block with zeros in the segment; The method further comprises: Metadata for the one or more stored duplicate data chunks is maintained, wherein the metadata for each stored duplicate data chunk includes a reference to a location where the duplicate data chunk is stored.

9. The method (600) of claim 8, wherein: Each container also includes fragment metadata for each fragment, the fragment metadata including the location of the fragment in the container.

10. The method (600) of claim 8, further comprising: obtaining the one or more repeated data blocks; filling the one or more repeated data blocks into a plurality of fragments of the one or more containers; and The plurality of fragments of the one or more containers are compressed.

11. The method (600) of claim 8, further comprising: The stored first repeated data block is retrieved according to the metadata of the first repeated data block.

12. The method (600) of claim 8, wherein: Each repeated data block is associated with a reference value indicating how often the repeated data block is being updated.

13. The method (600) of claim 12, further comprising: The repeated data blocks whose reference values ​​are higher than a preset value and the repeated data blocks whose reference values ​​are lower than the preset value are stored in different containers.

Images