A method for quickly merging backup points in a deduplication system

Abstract

A method for fast backup point merging in a deduplication system, step 1: a data backup application starts a backup point merging task module in the deduplication system; step 2: the backup point merging task module receives instructions and obtains the source backup and target backup that need to be merged; step 3: the backup point merging task module detects whether there are data blocks with the same content in the source backup and the target backup, or whether there are data blocks in the source backup that need to be updated and replaced by the target backup; if so, the index entries corresponding to the qualified data blocks in the source backup are exchanged with the index entries corresponding to the qualified data blocks in the target backup file; step 4: the backup point merging task module marks the exchanged indexes in the source backup, the references corresponding to the indexes, and the data blocks for deletion. This application solves the problem that data blocks must be transmitted from the deduplication system to the application and then returned to the deduplication system. At the same time, it can reduce bandwidth in cloud scenarios.

Description

Technical Field

[0001] The present invention relates to the technical field of data storage, and in particular to a method for fast backup point merging in a duplicate data deduplication system. Background Art

[0002] A typical data deduplication system saves the source data of a backup session into three types of files, namely index files, reference files, and data files. Data files are used to store data blocks, and reference files are used to record the metadata of these data blocks. Therefore, each data file must have one and only one corresponding reference file. On the other hand, index files are regarded as representations of source data. They do not contain real data, but they can determine where to find the real data by pointing to reference files. Obviously, entries in index files can point to multiple reference files, so subsequent backups can reference the backed-up data blocks, thereby achieving data deduplication (refer to Figure 1 ).

[0003] Assume that Backup 2 has 3 unique data blocks (D1', D2', D3') that have never existed in the deduplication system, so it must write new indexes, references, and data entries for them. Assume also that the last 2 data blocks (D4, D5) are identical to the blocks in Backup 1, it only needs to write index entries I4' and I5' and then have them point to the corresponding reference entries of Backup 1. Therefore, Backup 2 does not have to write the duplicate blocks again, it can read them from Backup 1 if necessary. This is the basic concept and typical layout of a deduplication system (refer to Figure 2 ).

[0004] The user's backup space is limited, but the data to be backed up is unlimited. Therefore, backup software usually saves user storage space by merging backup points that exceed the retention policy to avoid situations where completeness cannot be achieved and storage capacity is low. Backup point merging is a key function for reclaiming storage space in most block-based backup applications. Typically, the merge operation identifies the new and old versions of data between backups, then retains the new version and clears the old version to free up its storage space.

[0005] As shown above, the traditional merge procedure is to read the new version of the data from the source backup and write it back to the target backup to overwrite the old version of the data. Finally, the source backup is deleted to reclaim the storage space it occupies. Specifically, we need to read the new data in the source backup and write it back to the target backup. The following are the detailed steps:

[0006] 1) The application reads new data from the deduplication system based on its index and reference files from the source backup and writes it to the target backup.

[0007] 2) The deduplication system finds duplicate blocks, so it does not actually write them. Instead, it only writes index entries and points them to the application. At the same time, the old index, the corresponding references, and the corresponding data entries are cleared.

[0008] 3) Clear the index entries of the source backup. The target backup has saved all new data blocks so far.

[0009] This workflow of the merging process is intuitive and easy to implement, as the deduplication system only needs to focus on the write and read processes. Data merging is simply a combination of the two controlled by the application according to the application requirements. However, the disadvantage is also obvious, the data blocks must be transmitted from the deduplication system to the application and then returned to the deduplication system. It obviously increases the load on the deduplication system itself, as well as its communication bandwidth with the application. In a local environment where the application and the deduplication system run on the same server, the above overhead can be alleviated by running in the intranet. However, if you consider deploying the deduplication system to an MSP (managed service provider) or cloud where the data blocks must be sent over the WAN, the above problem is more prominent. Summary of the invention

[0010] The present invention aims at the deficiencies in the prior art and provides a method for fast backup point merging in a deduplication system, which solves the main drawback of backup point merging in the above-mentioned traditional deduplication system: data blocks must be transferred from the deduplication system to the application and then returned to the deduplication system. The present invention can effectively reduce bandwidth in cloud or MSP scenarios, and can reduce data outflow in cloud scenarios to save users' expenses.

[0011] To achieve the above object, the present invention adopts the following technical solutions:

[0012] A method for fast backup point merging in a deduplication system comprises the following steps:

[0013] Step 1: The backup application requests to start the backup point merging task module in the deduplication system;

[0014] Step 2: After receiving the request start instruction, the backup point merging task module obtains the source backup and target backup that need to be merged;

[0015] Step 3: The backup point merge task module detects data blocks with the same content in the source backup and the target backup, and exchanges the index entries corresponding to the data blocks with the same content in the source backup with the index entries corresponding to the data blocks with the same content in the target backup;

[0016] The backup point merging task module detects that the source backup contains data blocks that need to be updated and replaced by the target backup, and exchanges the index entries corresponding to the new version content data blocks in the source backup with the index entries corresponding to the old version content data blocks in the target backup file;

[0017] Step 4: The backup point merge task module marks and deletes the exchanged index entries in the source backup, the references and data blocks corresponding to the index entries.

[0018] To optimize the above technical solutions, the specific measures taken also include:

[0019] Furthermore, in step 3, the backup point merging task module also detects whether there are data blocks in the target backup that are not in the source backup. If so, the corresponding index entries of the qualified data blocks are retained so that the index entries still point to the corresponding data blocks.

[0020] The beneficial effects of the present invention are:

[0021] 1. This application decouples the deduplication system from the application, which only needs to send commands instead of processing data blocks. Therefore, the data transmission time between them will be significantly shortened, especially in the environment of WAN (such as MSP or cloud).

[0022] 2. Within the deduplication system, data blocks remain unchanged and only undersized index entries are moved, thus further increasing self and overall performance.

[0023] 3. The merge process does not have to run specific code to mark unreferenced data blocks for cleanup. These data blocks are still referenced by the source backup, but will be cleaned up smoothly after the source backup is deleted.

[0024] 4. Avoid unnecessary calculations of reference counts. For data blocks in traditional data merges, we must increase their reference counts when writing to the target backup, and then decrease their reference counts when deleting the source backup. However, in this new process and layout, we do not have to deal with this problem because their reference counts remain unchanged throughout the merge process (in reference counting, each object is responsible for maintaining the count value of all references to the object. When a new reference points to the object, the reference counter is incremented, and when a reference is removed, the reference count is decremented).

[0025] 5. Significantly improve the performance of data merging in deduplication systems.

[0026] 6. Decouple deduplication from the application during data consolidation, thereby moving existing deduplication from the WAN to the MSP or cloud.

[0027] 7. Simplify the data merging process to make it smoother and more intuitive. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 The present invention is a storage schematic diagram of a data deduplication system in the prior art.

[0029] Figure 2 It is the basic concept and typical layout diagram of the deduplication system in the prior art;.

[0030] Figure 3 It is a schematic diagram of the exchange index according to an embodiment of the present invention.

[0031] Figure 4 This is a schematic diagram after exchanging indexes according to an embodiment of the present invention.

[0032] Figure 5 It is a schematic diagram of data merging in an embodiment of the present invention. DETAILED DESCRIPTION

[0033] The present invention will now be described in further detail with reference to the accompanying drawings.

[0034] The data blocks in the source backup and the target backup will not change at all before and after the source backup and the target backup are merged. The most important difference here is that the index entries of the target backup have been updated to point to the new data blocks instead of the old data blocks. In order to achieve the same result by improving performance, the key idea of ​​this application is to exchange index entries between the source backup and the target backup in the deduplication system. The detailed process is described as follows:

[0035] 1. In the deduplication system, the data blocks are kept unchanged and only the index entries are exchanged between the two backups (refer to Figure Figure 3 ).

[0036] 2. After the swap, the new indexes are already in the target backup, but they still point to the new references and data blocks. In addition, the old indexes also point to the old references and data blocks so that they can be cleared later (refer to Figure 4 ).

[0037] When the index exchange is complete, the target backup already holds all the new data blocks. The only remaining work is to clean up the source backup. During this process, data with non-zero reference counts will not be cleared because they are still pointed to by indexes (that is, the target backup has data blocks that are not in the source backup, and the corresponding indexes will not be cleared).

[0038] Note that some applications use a deduplication system as a storage component, which may not know the internal details and therefore cannot know that some blocks have been merged into the target backup. Therefore, these can also be part of the blocks to be merged. In this case, it is easy to find that the idea of ​​index exchange also applies, because these blocks point to the same reference entries as the old blocks, so exchanging them is still much faster than the traditional merge method and leads to the same result. Therefore, we usually perform index entry exchanges in batches, regardless of whether they are merged or not. This operation can turn random reads and writes to the index into batch sequential reads and writes, effectively improving read and write performance.

[0039] The following is a specific example for explanation. Figure 3-Figure 5 .

[0040] The application starts the backup point merging task module;

[0041] The backup point merge task module obtains the source backup (backup 2) and the target backup (backup 1) to be merged;

[0042] The backup point merge task module detects the blocks D1, D2, D3 and D1', D2', D3' to be merged (the contents of D1, D2, D3 are the same as the corresponding D1', D2', D3', or D1', D2', D3' are new versions of D1, D2, D3, respectively);

[0043] The backup point merge task module exchanges index entries I1-I3 and I1'-I3' between two backups (refer to Figure 3 );

[0044] After the swap, I1', I2' and I3' are now in backup 1, but they still point to the new references and data blocks. In addition, I1, I2 and I3 also point to the old references and data blocks in order to clear them later (refer to Figure 4 );

[0045] When the index swap is complete, backup 1 already holds all the new data blocks (see Figure 5 );

[0046] The backup point merge task module marks the old indexes (I1-I3), corresponding references (R1-R3), and corresponding data blocks (D1, D2, D3) in backup 2 as deleted;

[0047] Recover invalid data in backup 2 to free up disk space.

[0048] The target backup has data blocks that are not in the source backup, namely D4 and D5; the corresponding references (R4 and R5) and indexes (I4 and I5) are not changed and still exist in the target backup.

[0049] It should be noted that the terms such as "upper", "lower", "left", "right", "front", "back", etc. cited in the invention are only for the convenience of description and are not used to limit the scope of implementation of the present invention. Changes or adjustments in their relative relationships should be regarded as the scope of implementation of the present invention without substantially changing the technical content.

[0050] The above are only preferred embodiments of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should be regarded as the protection scope of the present invention.

Claims

1. A method for fast backup point merging in a deduplication system, characterized in that: The following steps are involved: Step 1: The backup application requests to start the backup point merging task module in the deduplication system; Step 2: After receiving the request start instruction, the backup point merging task module obtains the source backup and target backup that need to be merged; Step 3: The backup point merge task module detects data blocks with the same content in the source backup and the target backup, and exchanges the index entries corresponding to the data blocks with the same content in the source backup with the index entries corresponding to the data blocks with the same content in the target backup; The backup point merging task module detects that the source backup contains data blocks that need to be updated and replaced by the target backup, and exchanges the index entries corresponding to the new version content data blocks in the source backup with the index entries corresponding to the old version content data blocks in the target backup file; Step 4: The backup point merge task module marks and deletes the exchanged index entries in the source backup, the references and data blocks corresponding to the index entries.

2. The method for fast backup point merging in a deduplication system according to claim 1, characterized in that: In step 3, the backup point merging task module also detects whether there are data blocks in the target backup that are not in the source backup. If so, the corresponding index entry of the qualified data block is retained so that the index entry still points to the corresponding data block.

Images