Fragment rewriting method and controller in a deduplication system based on differential compression

Abstract

This invention discloses a fragment rewriting method and controller in a deduplication system based on differential compression, belonging to the field of data storage technology. The method includes: for each memory segment of a stored data block, counting the number of times each container is referenced by duplicate and differential data blocks within it, filtering out the T containers with the highest reference counts; marking duplicate and differential data blocks referencing the selected containers in the current memory segment as non-fragmented data blocks, and marking self-repeating and differential data blocks referencing other containers as fragmented data blocks; further, combining container sparsity, simulated recovery caching, and sliding window analysis to identify whether a data block is fragmented; traversing the data blocks, if it is a non-fragmented duplicate data block, performing deduplication; if it is a non-fragmented differential data block, performing differential compression; if it is a fragmented or non-repeating data block, rewriting. This invention can improve system recovery performance while ensuring deduplication rate.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of data storage, and more particularly relates to a method for rewriting fragments in a deduplication system based on differential compression and a controller. BACKGROUND

[0002] Data deduplication technology and differential compression technology are both common redundant data elimination technologies. Compared with traditional data compression technology, data deduplication technology identifies and eliminates redundant data at a larger granularity (such as files, 4KB data blocks). Therefore, data deduplication technology is faster and has higher redundant data elimination performance on a larger scale storage system. In the context of explosive expansion of data size, data deduplication technology can effectively meet the performance requirements of redundant data elimination under massive data and is widely used in existing storage systems. Differential compression technology is a technology that eliminates redundant data between two different data, so that only one difference data needs to be stored additionally. Data deduplication can only eliminate duplicate data with the same content, while differential compression can eliminate redundant data for two data with different content. Therefore, existing related researches regard differential compression as a supplementary compression technology after data deduplication, and perform differential compression on the deduplicated storage system to save more storage space after the application of data deduplication technology.

[0003] Data deduplication technology generally includes four stages: data blocking, fingerprint calculation, fingerprint indexing, and data storage. First, in the data blocking stage, a blocking algorithm is used to calculate the data stream to determine the cutting point position, and the data stream is divided into data blocks according to the cutting point position. Second, after obtaining the blocks, a hash function (such as SHA-1, SHA-256) is used to generate a hash value for each block, which is called a fingerprint. Thereafter, the deduplication system indexes the existing fingerprints to determine whether there is a duplicate fingerprint. If the fingerprint is duplicated, it can be considered that the data block is also duplicated. Finally, the storage system directly writes the non-duplicated data block, and the duplicated data block is not written but a pointer to the previously existing duplicated data block is saved, so that the redundant data in the storage system is eliminated.

[0004] The differential compression introduced in the data deduplication technology is generally divided into two stages, namely the similarity detection stage and the differential calculation stage. First, in the similarity detection stage, the differential compression module of the data deduplication system will calculate the corresponding feature value for each input data block. The feature (Super Feature) here is a specific value used to represent whether two data blocks in the data deduplication system are similar. Then the data deduplication system will index the feature values saved in the previous deduplication process, find the same feature values, and obtain the metadata information of the data blocks with the feature values. Finally, the differential compression module will take the data blocks with the same feature values found as the base data block (Base Chunk), find the data different from the base data block in the existing data block and write it to a new data block Delta, generate a data block (Delta Chunk) that only retains the difference data between the two data blocks, and save the data block to the data deduplication storage system, while retaining the metadata information corresponding to the data block.

[0005] The data deduplication system combined with the differential compression technology can effectively identify and eliminate more redundant data, saving more storage space for the data storage system. However, the backup and restoration process of the data deduplication technology and the differential compression technology will generate a large amount of computational overhead, affecting the throughput of the system, and the fragmentation problem caused by the data deduplication technology will seriously affect the performance of the storage system, while the introduction of the differential compression technology will bring new fragmentation problems.

[0006] Accessing a file in the data deduplication system first requires a data deduplication restoration process. In the data deduplication system restoration process, the data blocks needed will be accessed in the order of the file spectrum in the data deduplication backup process, and the original data stream will be restored accordingly. Repetitive data blocks will refer to previously stored non-repetitive data blocks in the data deduplication process, and the restoration of a repetitive data block may require access to a container stored a long time ago. Similarly, the base block used in the differential compression process can also cause similar data block fragmentation problems. In the restoration process, the recovery module will read the entire container into memory, but the presence of fragmentation problems can make the restoration process only need to use a single or a few data blocks in the entire container, which causes the read amplification (the size of the data block read is larger than the size of the data block needed) problem, thereby reducing the restoration performance of the data deduplication system. Therefore, the restoration performance of the deduplication system with the introduction of differential compression will be significantly reduced.

[0007] Currently, there are some works dedicated to improving the restoration performance of the deduplication system. However, these works still have the following shortcomings:

[0008] 1) The part of the work only considers how to rewrite the data deduplication technology to improve the recovery performance of the data deduplication system, and ignores the problem of new fragmentation caused by introducing the differential compression.

[0009] 2) The self-repeating fragment, that is, the repeated block fragment that appears multiple times in the same data stream and is combined with the basic block fragment to cause the recovery performance to decrease, will obviously affect the recovery performance of the data deduplication system, and the existing work often does not consider the influence of the self-repeating fragment.

[0010] 3) The existing work identifies a part of the data block that does not need to be rewritten as a fragment and rewrites it, thereby reducing the redundancy elimination rate of the data deduplication system.

[0011] Overall, in the differential compression-based deduplication system, how to improve the system recovery performance while ensuring the redundancy deduplication rate is a problem to be studied. SUMMARY

[0012] In view of the defects of the prior art and the improvement needs, the present application provides a fragment rewriting method and a controller in a differential compression-based deduplication system, which aims to improve the system recovery performance while ensuring the redundancy deduplication rate in the differential compression-based deduplication system.

[0013] To achieve the above-mentioned purpose, according to one aspect of the present application, a fragment rewriting method in a differential compression-based deduplication system is provided, comprising the following steps:

[0014] (S1) For the data stream to be written, divide it into data blocks, identify the repeated data blocks and the differential data blocks therein, and for each memory segment storing the data blocks, count the number of times each container is referenced by the repeated data blocks and the differential data blocks in the memory segment to screen out the first T containers with the largest number of references, mark the repeated data blocks and the differential data blocks in the current memory segment that reference the screened containers as non-fragmented data blocks, and mark the self-repeating data blocks and the differential data blocks in the current memory segment that reference the remaining containers as fragmented data blocks; T is a positive integer;

[0015] (S2) Traverse the data blocks, for each data block traversed, judge its type, if it is a repeated data block in a non-fragmented data block, perform deduplication, if it is a differential data block in a non-fragmented data block, perform differential compression, and if it is a fragmented data block or a non-repeated data block, reassign a container and write it.

[0016] The application is based on the screening of containers by the number of references, the relative most repeated data blocks and differential data blocks in the same memory segment, and after the de-duplication or differential compression of these data blocks, when recovering these data blocks subsequently, the container is read once, and the data blocks referenced by the container can be recovered, and the read amplification is small; on the contrary, for the containers that are not screened, since they are referenced by less repeated data blocks and differential data blocks in the same data stream, if the repeated data blocks or differential data blocks referenced by these containers are de-duplicated or differentially compressed, when recovering these data blocks subsequently, reading the container will cause reading a large number of useless data blocks, causing a serious read amplification problem; therefore, after identifying the repeated data blocks and differential data blocks in the data stream, the application counts the number of times that each container is referenced by the repeated data blocks and differential data blocks in the same memory segment, and screens the containers with more reference times, marks the repeated data blocks and differential data blocks in the containers as non-fragmented data blocks, and performs de-duplication or differential compression, and for the containers with less reference times, the repeated data blocks and differential data blocks in the containers are identified as fragmented data blocks and are re-written, which can effectively alleviate the fragmentation problem in the data de-duplication process, reduce the read amplification in the data recovery process, and improve the data recovery performance. The application considers both repeated data blocks and differential data blocks when counting the reference times of the containers, and is a joint rewriting method based on differential compression sensing.

[0017] The application screens the containers by counting the reference counts of the containers, can identify the fragments in the repeated data blocks, and rewrites the fragments, therefore, the application can fully consider the influence of the fragmented repeated data blocks on the recovery performance of the data de-duplication system, weaken the influence, and be beneficial to improving the recovery performance of the subsequent system.

[0018] Further, the application provides a fragment rewriting method in a differential compression-based deduplication system, and step (S1) further includes:

[0019] For the non-repeated data blocks in the un-screened containers, it is judged whether the container referenced by the non-repeated data blocks is currently marked as a sparse container, if yes, the non-repeated data blocks are marked as fragmented data blocks; otherwise, the non-repeated data blocks are marked as non-fragmented data blocks.

[0020] The sparse container is a container with a utilization rate lower than a preset utilization rate threshold; the utilization rate of the container is the ratio of the sum of the sizes of the repeated data blocks and the differential data blocks referencing the container to the capacity of the container.

[0021] Research shows that a container, if in the previous data stream is sparse container, namely the size of the repeated data block and the difference data block referenced by the container is small compared with the capacity of the container, then the container is still sparse container in the current data stream, that is, the sparsity of the container has inheritability. Because the sparse container is referenced by less repeated data block and difference data block, reading the container will cause large read amplification in subsequent data recovery process, which affects the data recovery performance. Based on this consideration, the present application uses the inherited sparse container to mark the non-self-repeated data block in the container not selected in step (S1) for fragmentation, which can accurately identify the repeated data block that will cause serious fragmentation problem during deduplication writing, and rewrite it, thereby further alleviating the fragmentation problem in the data deduplication process, reducing the read amplification in the data recovery process, and improving the data recovery performance.

[0022] Further, the deduplication system based on difference compression provided by the present application further comprises:

[0023] After the backup of the current data stream ends, the utilization rate of each container is calculated, and the container with a utilization rate lower than the utilization rate threshold is marked as a sparse container.

[0024] The sparsity of the container will dynamically change with the load change. The present application calculates the utilization rate of each container after the backup of the current data stream ends, and updates the sparsity of the container, which can accurately mark the sparsity of the container without affecting the normal writing of the data stream, and ensure the accuracy of the subsequent data block fragmentation marking.

[0025] Further, the deduplication system based on difference compression provided by the present application further comprises: maintaining a simulation recovery cache in the memory for caching a specified number of containers;

[0026] And in step (S2), for the data block currently traversed, if it is identified as a difference data block in step (S1), before judging its type, it further comprises: searching for the container where the basic data block referenced by the difference data block is located in the simulation recovery cache, if not hit, marking the difference data block as a fragmented data block; if hit, marking the difference data block as a non-fragmented data block.

[0027] In the data stream recovery process, the recovery cache is used to cache the recently accessed containers, and the recovery cache is a scheme for reducing the number of additional container read times in the data deduplication recovery process, which utilizes the locality of the data stream, that is, a referenced container is likely to be referenced together with other data blocks in the subsequent recovery process, which can effectively avoid most fragmented data blocks. For the data block written using differential compression, if the container where the referenced basic data block is located is cached by the recovery cache during the recovery process, the recovery of the data block will not cause additional container reading and disk access, so differential compression is performed on such data block during writing, which will not affect the performance of subsequent data recovery. In addition, for a data stream, the order flow during backup writing is the same as that during recovery, based on this consideration, the application maintains a simulated recovery cache during data stream writing, caches the recently accessed containers during data stream writing, and for the differential data block that needs to be compressed differentially, the container where the corresponding basic data block is located is searched in the simulated recovery cache, if it is hit, it means that the container will also hit the recovery cache in the subsequent recovery process, for such differential data block, differential compression is performed, and for other differential data blocks, re-writing is performed, thereby the historical information can be fully utilized, the data deduplication efficiency is effectively improved without affecting the data recovery performance.

[0028] Further, the application provides a method for rewriting fragments in a deduplication system based on differential compression, and step (S2) further comprises: after writing the differential data block in the fragmented data block, updating the simulated recovery cache by using the container where the data block is located; and after writing the differential data block in the non-fragmented data block, updating the simulated recovery cache by using the container where the referenced basic data block is located and the container where the differential encoding is located.

[0029] Further, the application provides a method for rewriting fragments in a deduplication system based on differential compression, and in step (S2), the data blocks are traversed by using the sliding window to slide on the data stream, and in the sliding process of the sliding window, the data block removed from the sliding window is the currently traversed data block.

[0030] Further, step (S2) further comprises: maintaining a hash table for indexing the container where the data block identified as the repeated data block in step (S1) and marked as the non-fragmented data block in the current sliding window is located.

[0031] Further, for the currently traversed data block, if it is identified as the differential data block in step (S1), or identified as the repeated data block and marked as the fragmented data block in step (S1), before judging its type, it further comprises: searching for the container where the referenced basic data block is located in the hash table, and if it is hit, marking it as the non-fragmented data block.

[0032] On the basis of simulating the recovery cache, the sliding window and the hash table are introduced, the final rewrite decision is determined only when the data block is removed from the sliding window, and the determination of the rewrite decision also considers the data block information after the current data block, specifically, the data block that has been identified as the repeated data block and marked as the non-fragmented data block in step (S1) is deduplicated in the subsequent process, the rewrite decision of the data block is determined, the container referenced by the data block is indexed by the hash table, when the data block is removed from the sliding window, if the referenced container can be found in the hash table, it is indicated that the data block is deduplicated or differentially compressed when written, and the data block is not caused to cause additional container access when recovered in the subsequent recovery process, the data block is marked as the non-fragmented data block and deduplicated or differentially compressed by the application, the future information can be fully utilized, the data deduplication efficiency is further improved without affecting the data recovery performance.

[0033] Further, the simulated recovery cache only caches the container ID.

[0034] Since only the container hit in the simulated recovery cache needs to be determined in the writing process, and no other information is required, the application only caches the container ID in the simulated recovery cache, only a small amount of cache space is occupied, and the historical information can be used to accurately mark the fragmentation state of the data block.

[0035] According to another aspect of the application, a controller applied to a deduplication system based on differential compression is provided, comprising: a computer readable storage medium and a processor; the computer readable storage medium stores a computer program, and the processor is used to read the computer program in the computer readable storage medium and execute the fragmentation rewrite method in the deduplication system based on differential compression provided by the application.

[0036] Overall, the above technical solutions conceived by the application can achieve the following beneficial effects:

[0037] (1) After the repeated data block and the differential data block in the data stream are identified, the number of times that each container is referenced by the repeated data block and the differential data block in the same memory segment is counted, and the containers with a large number of references are screened out, for the screened containers, the repeated data block and the differential data block therein are marked as non-fragmented data blocks and written after deduplication or differential compression, and for the containers with a small number of references, the repeated data block and the differential data block therein are identified as fragmented data blocks and rewritten, which can effectively alleviate the fragmentation problem in the data deduplication process, reduce the read amplification in the data recovery process, and improve the data recovery performance.

[0038] (2) The application can accurately identify the repeated data blocks which will cause serious fragmentation problem when rewritten, and rewrite them, so as to further alleviate the fragmentation problem in the data deduplication process, reduce the read amplification in the data recovery process, and improve the data recovery performance.

[0039] (3) In the data stream writing process, a simulation recovery cache is maintained, and the recently accessed containers in the data stream writing process are cached, and for the differential data blocks which need to be differentially compressed, the container where the corresponding basic data block is located is searched in the simulation recovery cache, if hit, it means that the container will also hit the recovery cache in the subsequent recovery process, and for such differential data blocks, differential compression is performed, and for other differential data blocks, re-writing is performed, so that the historical information can be fully utilized, and the data deduplication efficiency can be effectively improved without affecting the data recovery performance.

[0040] (4) On the basis of the simulation recovery cache, the rewriting decision of the data block is further determined by using the sliding window and the hash table, when the data block is removed from the sliding window, if the container referenced by the data block can be found in the hash table, the data block is marked as a non-fragmented data block, and deduplication or differential compression is performed, so that the future information can be fully utilized, and the data deduplication efficiency can be further improved without affecting the data recovery performance. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 The flowchart of the data stream backup process of the existing differential compression-based re-deletion system;

[0042] Figure 2 The flowchart of the fragmentation rewriting method in the differential compression-based re-deletion system provided by the embodiment of the application;

[0043] Figure 3 The method schematic diagram for identifying the self-repeated data block fragmentation in the data deduplication stage provided by the embodiment of the application;

[0044] Figure 4 The method schematic diagram for identifying the non-self-repeated data block fragmentation in the data deduplication stage provided by the embodiment of the application;

[0045] Figure 5 The method schematic diagram for identifying the basic data block fragmentation based on the historical information provided by the embodiment of the application;

[0046] Figure 6 The method schematic diagram for identifying the basic data block fragmentation based on the future information provided by the embodiment of the application. DETAILED DESCRIPTION

[0047] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application. In addition, the technical features involved in the various embodiments of the present application described below can be combined with each other as long as they do not conflict with each other.

[0048] In the present application, the terms "first", "second", etc. (if any) in the present application and the accompanying drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.

[0049] Reference is made to Figure 1 When a conventional re-deletion system based on differential compression performs data stream backup, the entire process is divided into the following seven stages:

[0050] (1) Data blocking: dividing the data stream into data blocks of equal size;

[0051] (2) Fingerprint calculation: calculating the fingerprint of each data block;

[0052] (3) Fingerprint index: through fingerprint comparison, finding out the data blocks with repeated fingerprints, which are referred to as repeated data blocks; the data blocks in the current data stream may be repeated with the data blocks in the previously backed up data stream, or may be repeated with other data blocks in the current data stream; among the repeated data blocks, the data blocks that appear multiple times in the same data stream are referred to as self-repeated data blocks, and the repeated data blocks in the self-repeated data blocks are referred to as non-self-repeated data blocks; since the probability of fingerprint conflict is very low, relevant research shows that the probability of fingerprint conflict is much lower than the probability of hardware error in less than 1PB of data, so the present application does not consider the case of fingerprint conflict;

[0053] (4) Similarity detection: calculating the features of the data blocks in the current data block that are not identified as repeated data blocks;

[0054] (5) Basic block matching: through feature comparison, finding out the data blocks with the same features; among the data blocks with the same features, the data block that already exists in the system is referred to as a basic data block, and the other data block is referred to as a differential data block that references the basic data block;

[0055] (6) Differential encoding: calculating and encoding the data difference between the differential data block and the basic data block referenced by the differential data block;

[0056] (7) Data storage: for the duplicated data block which needs to be deduplicated, save a pointer to the previous duplicated data block; for the differential data block which needs to be differentially compressed, save a pointer to the referenced base data block and write its differential encoding into the container; for the rest data block, allocate a container and write it in.

[0057] In the process of data deduplication and differential compression, new fragmented data blocks are introduced, which causes the read amplification problem in the subsequent data recovery process of the deduplication system based on differential compression, and affects the data recovery performance. In order to solve this problem, the present application provides a method for rewriting fragmented data in a deduplication system based on differential compression and a controller, the overall idea of which is to accurately identify the duplicated data block and the differential data block which may introduce fragmented data blocks during the process of writing the data stream backup, and allocate new containers to these data blocks for rewriting, so as to effectively reduce the fragmented data blocks, so that the read amplification problem is effectively alleviated in the subsequent data recovery process, and the data recovery performance is improved.

[0058] The following is an example.

[0059] Example 1

[0060] A method for rewriting fragmented data in a deduplication system based on differential compression, as shown in Figure 2 , comprising the following steps:

[0061] (S1) For the data stream to be written, divide it into data blocks, identify the duplicated data blocks and the differential data blocks in them, and for each memory segment storing the data blocks, count the number of times each container is referenced by the duplicated data blocks and the differential data blocks in the memory segment to select the top T containers with the maximum number of references, mark the duplicated data blocks and the differential data blocks in the current memory segment which reference the selected containers as non-fragmented data blocks, and mark the self-duplicated data blocks and the differential data blocks in the current memory segment which reference the remaining containers as fragmented data blocks; T is a positive integer, and in this embodiment, T = 1;

[0062] (S2) Traverse the data blocks, for each data block traversed, judge its type, if it is a duplicated data block in the non-fragmented data block, write it by deduplication; if it is a differential data block in the non-fragmented data block, write it after differential compression; if it is a fragmented data block or a non-duplicated data block, allocate a container and write it in;

[0063] The step (S1) of the present embodiment further comprises:

[0064] For non-self-repeated data blocks in the un-screened container, it is judged whether the referenced container is currently marked as a sparse container, if yes, the non-self-repeated data block is marked as a fragmented data block; otherwise, the non-self-repeated data block is marked as a non-fragmented data block.

[0065] Among them, the sparse container is a container with a utilization rate lower than a preset utilization rate threshold; the utilization rate of the container is the ratio of the sum of the sizes of the repeated data blocks and the differential data blocks referencing the container to the container capacity.

[0066] The step (S1) of the embodiment implements a joint rewriting method of differential compressed sensing, which first identifies self-repeated data blocks and differential data blocks that will cause fragmentation problems by counting the number of times the repeated data blocks and the differential data blocks in the segment reference the containers, so as to limit the number of times the containers are referenced in the recovery process; further, non-self-repeated data blocks that will cause fragmentation problems will be identified based on the sparsity of the containers. For the identified fragments, rewriting will be performed without deduplication.

[0067] The joint rewriting method of differential compressed sensing is further explained in combination with a specific example below.

[0068] Figure 3 As shown, after the fragmentation identification based on the above step (S1) and rewriting, the distribution of data in the containers is shown; backup 1 and backup 2 are two data streams, backup 2 is the current data stream, and backup 1 is the data stream that has been written into the backup before, the data blocks in backup 1 are stored in containers 1-4, and the capacity of each container is 3 data blocks;

[0069] The data blocks in backup 2 are divided into segments and further identified, and the size of each memory segment is 5 data blocks, the data blocks in backup 2 are stored in 2 segments accordingly, A, B, D, and E in segment 1 are repeated data blocks, C' is a differential data block, F' in segment 2 is a differential data block, H and I are repeated data blocks, A is a self-repeated data block, and K is a non-repeated data block;

[0070] For segment 1, the number of times each container is referenced by the repeated data blocks and the differential data blocks in the memory segment is counted, container 1 is referenced by repeated data block A, repeated data block B, and differential data block C' in segment 1, and the corresponding reference times are 3; container 2 is referenced by repeated data block D and repeated data block E in segment 1, and the reference times are 2; the reference times of the remaining containers are 0;

[0071] After container 1 is filtered according to T, the duplicate data blocks A, B and the differential data block C' in segment 1 that reference container 1 will be marked as non-fragmented data blocks; in segment 1, the duplicate data blocks D and E that reference container 2 need to be further identified because they are not self-duplicate data blocks.

[0072] For segment 2, the number of times each container is referenced by the duplicate data blocks and the differential data blocks in the memory segment is counted, container 3 is referenced by the duplicate data blocks H and I in segment 2, and the number of times of reference is 2; container 1 is referenced by the self-duplicate data block A in segment 2, and the number of times of reference is 1; container 2 is referenced by the differential data block F' in segment 2, and the number of times of reference is 1; the number of times of reference of the remaining containers is 0.

[0073] After container 3 is filtered according to T, the duplicate data blocks H and I in segment 2 that reference container 3 will be marked as non-fragmented data blocks; in segment 2, the self-duplicate data block A that references container 1 will be marked as a fragmented data block and will be overwritten later; in segment 2, the differential data block F' that references container 2 needs to be further identified.

[0074] Based on the data blocks that cannot be directly identified based on the number of times of reference of the containers, the fragmented data blocks will be identified by the historical information backed up previously; specifically, in the present embodiment, the non-self-duplicate data blocks stored in the sparse containers in the current data stream will be marked as fragmented data blocks, and will be overwritten later; in order not to affect the backup writing of the data stream, the present embodiment directly uses the inherited sparse containers for identification; in the present embodiment, the sparse containers specifically refer to the containers with a low ratio (utilization) of the sum of the sizes of the duplicate data blocks and the differential data blocks that reference the containers to the capacity of the containers; specifically, in the present embodiment, a sparse container table is maintained for recording the container IDs of the containers that are currently marked as sparse containers; for the non-self-duplicate data blocks that need to be identified, the container IDs referenced by the non-self-duplicate data blocks are searched in the sparse container table, and if a hit is obtained, the non-self-duplicate data block is marked as a fragmented data block.

[0075] Considering that the sparseness of the containers will dynamically change with the load, the present embodiment further includes:

[0076] After the backup of the current data stream ends, the utilization of each container is calculated, and the containers with a utilization lower than a utilization threshold are marked as sparse containers, and the sparse container table is updated, and the related process is as shown in Figure 4 Thereby, the sparseness of the containers can be accurately marked adaptively to the load change without affecting the normal writing of the data stream, and the accuracy of the subsequent data block fragmentation marking is ensured.

[0077] To further identify the difference data block that may cause fragmentation problem in the backup writing process of the data stream, the embodiment further includes, on the basis of the joint rewriting method of the difference compressive sensing, maintaining a simulation recovery cache for caching a specified number of containers in the memory; since no other information is needed in the simulation recovery process, only the judgment of whether the basic data block of the current data block is in the recovery cache is needed, therefore, in the embodiment, only the container ID (8 bytes per container ID) is cached, and optionally, in the embodiment, the simulation recovery cache is as large as 2 containers.

[0078] And for the difference data block, it is checked whether the container where the referenced basic data block is located is in the current recovery cache, if not, the data block will be identified as a difference data block that may cause fragmentation problem and be rewritten; if yes, the data block will be subjected to difference compression; to achieve the identification process, in step (S2), for the data block currently traversed, if it is identified as a difference data block in step (S1), before judging the type, it further includes: searching for the container where the referenced basic data block of the difference data block is located in the simulation recovery cache, if not hit, marking the difference data block as a fragmentation data block that will be rewritten later; if hit, marking the difference data block as a non-fragmentation data block that will be subjected to difference compression later.

[0079] The recovery cache is a scheme for reducing the number of additional container read times in the data deduplication recovery process, which utilizes the locality of the data stream, that is, a referenced container is likely to be referenced together with other data blocks in the subsequent recovery process, which can effectively avoid most fragmentation data blocks, in addition, the order flow in the backup writing process is the same as that in the recovery process, in the data stream writing process, the embodiment maintains a simulation recovery cache, caches the recently accessed containers in the data stream writing process, and selects the simulation recovery cache scheme that does not produce fragments among the multiple features of the data block that needs to be subjected to difference compression, so as to fully utilize the historical information, effectively improve the data deduplication efficiency, and avoid affecting the data recovery performance.

[0080] The embodiment updates the simulation recovery cache after the difference compression or rewriting of each difference data block, accordingly, step (S2) further includes: after writing the difference data block in the fragmentation data block, updating the simulation recovery cache by using the container where the data block is located; after writing the difference data block in the non-fragmentation data block, updating the simulation recovery cache by using the container where the referenced basic data block is located and the container where the difference encoding is located.

[0081] Optionally, in the process of updating the simulation recovery cache, if replacement occurs, the LRU (Least Recently Used) algorithm is used.

[0082] The method of identifying the fragmented data block in the differential data block is a simulation recovery cache rewriting method based on historical information.

[0083] The following will be combined Figure 5 The specific process of the simulation recovery cache rewriting method based on historical information will be further explained.

[0084] The size of the simulation recovery cache is 2 containers; the upper container pool shows the data layout after the previous data deduplication, and the size of each container is 4 data blocks; the brackets behind the characteristics 1, characteristics 2 and characteristics 3 of the data block represent the matching basic data block, and if NULL is in the bracket, it means that no matching basic data block is found; for a data block that needs differential compression, this embodiment will first calculate the characteristics (here, 3) of the data block, and then find the corresponding basic data block; as shown in Figure 5 The differential data block B' has 3 characteristics, which correspond to the basic data block I, the basic data block B and the basic data block K, and the container 1 where the basic data block B is located is in the current simulation recovery cache, if the differential data block B' is compressed according to the basic data block B, since the backup order flow and the recovery order flow are the same, the simulation recovery cache and the recovery cache are the same, and in the recovery process, accessing the basic data block B and the differential encoding d(B) to reconstruct the recovery differential data block B' will not cause additional container reading and disk access, so the differential data block B' can be compressed, for such a differential data block, this embodiment will mark it as a non-fragmented data block; in the subsequent process, the differential data block F' has 3 characteristics, which correspond to the basic data block C, the basic data block F and no matching basic data block is found, the basic data block C pointed by the characteristic 1 and the basic data block F pointed by the characteristic 2 are stored in the container 1 and the container 5 which are not in the current simulation recovery cache, then whether the basic data block C or the basic data block F is selected to compress the differential data block F', in the subsequent recovery process, the basic data block C or the basic data block F accessed by the recovery differential data block F' will cause an additional container reading and disk reading operation, which will affect the restoration performance of the recovery process; for such a differential data block, this embodiment will mark it as a fragmented data block, which will be rewritten in the subsequent process without differential compression.

[0085] Embodiment 2:

[0086] A method for rewriting fragmented data in a deduplication system based on differential compression, similar to embodiment 1, the difference is that this embodiment introduces a sliding window and a hash table based on the simulation recovery cache, and realizes a delayed rewriting window rewriting method based on future information. After step (S1), the marking results of the fragmented data blocks are further corrected.

[0087] Specifically, this embodiment introduces a sliding window to traverse the data blocks. Specifically, during the sliding process of the sliding window on the data stream, the data blocks removed from the sliding window will be regarded as the currently traversed data blocks. The sliding window collects the data block information after the current data block to make rewriting decisions. The data block information after the current data block comes from the identification and marking of data block fragments in the deduplication stage by the joint rewriting method of differential compression sensing.

[0088] In step (S2) of this embodiment, in addition to maintaining a simulation recovery cache, a hash table is also maintained to index the container where the data block identified as a duplicate data block in step (S1) and marked as a non-fragmented data block in the current sliding window is located.

[0089] When each data block is removed from the sliding window, its type will be determined and the corresponding rewriting decision will be made, as follows.

[0090] If the currently traversed data block is identified as a duplicate data block in step (S1) and marked as a non-fragmented data block, its type is a non-fragmented data block and it will be deduplicated in the future.

[0091] If the currently traversed data block is identified as a differential data block in step (S1), the determination of its type includes: searching for the container where the referenced basic data block is located in the simulation recovery cache and the hash table. If neither hits, it will be marked as a fragmented data block and will be rewritten in the future. Otherwise, it will be marked as a non-fragmented data block and will be subjected to differential compression in the future.

[0092] If the currently traversed data block is identified as a duplicate data block in step (S2) and marked as a fragmented data block, the determination of its type is: searching for the container referenced by it in the hash table. If it hits, it will be marked as a non-fragmented data block and will be deduplicated in the future. Otherwise, it will be marked as a fragmented data block and will be rewritten in the future.

[0093] On the basis of the simulation recovery cache, the sliding window and the hash table are introduced, the final rewrite decision is determined only when the data block is removed from the sliding window, and the determination of the rewrite decision also considers the data block information after the current data block, so that the future information can be fully utilized, the data deduplication efficiency is further improved without affecting the data recovery performance.

[0094] The future information based delayed rewrite window rewriting method in the embodiment is further explained in combination with the examples shown in the following Figure 6 The future information based delayed rewrite window rewriting method in the embodiment is further explained in combination with the examples shown in the following

[0095] As Figure 6 For example, the size of the simulation recovery cache is 2 containers, the size of the sliding window is 8 data blocks, and the hash table represents the container ID that will be referenced in the subsequent rewriting process. In the embodiment, the container ID is the container ID referenced by the data block identified as the duplicate data block and marked as the non-fragmented data block in step (S1) in the current sliding window. During the sliding of the sliding window, the data block popped out needs to make a final rewrite decision. The basic data block G matched in the differential compression stage in the figure is a data block referencing container 2. Since container 2 is not in the simulation recovery cache, if the marking method in embodiment 1 is used, the differential data block G' will be marked as a fragmented data block. In the embodiment, the differential data block G' will be further marked according to the subsequent data stream locality information. In the subsequent data stream, since the duplicate data block J, the duplicate data block K and the duplicate data block L are non-fragmented duplicate data blocks marked by the joint rewriting method in the first step of rewriting, the container 2 where these data blocks are located will be accessed in the subsequent rewriting process. Therefore, the container 2 referenced by the basic data block G will not bring an additional container access, and therefore, in the embodiment, the differential data block G' is marked as a non-fragmented data block, and will be subjected to differential compression in the future. The marking method of the differential data block H' is the same as that of the differential data block G', which will not be described here. For the duplicate data block I, it has been identified as a self-duplicate data block by the joint rewriting method in the first step of rewriting, and is marked as a fragmented data block. If the marking method in embodiment 1 is used, the duplicate data block I will be marked as a fragmented data block. In the embodiment, the container 2 referenced by the data block I will be referenced in the subsequent process, so that the deduplication of the duplicate data block I will not bring an additional container access. Therefore, in the embodiment, the duplicate data block I is marked as a non-fragmented data block and will be deduplicated in the future.

[0096] Embodiment 3:

[0097] A controller applied to a differential compression based deduplication system, comprising: a computer readable storage medium and a processor; the computer readable storage medium has stored therein a computer program, and the processor is configured to read the computer program in the computer readable storage medium and execute the fragment rewriting method in the differential compression based deduplication system provided in the above-mentioned embodiments 1 or 2.

[0098] Those skilled in the art can easily understand that the above description is only the preferred embodiment of the present application, and is not intended to limit the present application. Any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for rewriting of fragmentation in a deduplication system based on delta compression, characterized in that, Comprising the following steps: (S1) For the data stream to be written, divide it into data blocks, identify the repeated data blocks and the differential data blocks therein, and for each memory segment storing the data blocks, count the number of times each container is referenced by the repeated data blocks and the differential data blocks in the memory segment to screen out the top T containers with the largest number of references, mark the repeated data blocks and the differential data blocks in the current memory segment that reference the screened containers as non-fragmented data blocks, and mark the repeated data blocks and the differential data blocks in the current memory segment that reference the remaining containers as fragmented data blocks; T is a positive integer; (S2) Traverse the data blocks, for each data block traversed, determine its type, if it is a repeated data block in the non-fragmented data blocks, perform deduplication; if it is a differential data block in the non-fragmented data blocks, perform differential compression; if it is a fragmented data block or a non-repeated data block, reassign containers and write; The method for rewriting fragments in the deduplication system based on differential compression further comprises: maintaining a simulation recovery cache in memory for caching a specified number of containers; And in step (S2), for the current data block traversed, if it is identified as a differential data block in step (S1), before determining its type, it further comprises: searching for the container where the basic data block referenced by the differential data block is located in the simulation recovery cache, if it does not hit, marking the differential data block as a fragmented data block; if it hits, marking the differential data block as a non-fragmented data block; In step (S2), the traversal of data blocks is achieved by sliding a sliding window over the data stream, and during the sliding of the sliding window, the data blocks removed from the sliding window are the currently traversed data blocks; And step (S2) further comprises: maintaining a hash table for indexing the containers where the data blocks identified as repeated data blocks and marked as non-fragmented data blocks in step (S1) in the current sliding window are located; And for the current traversed data block, if it is identified as a differential data block in step (S1), or is identified as a repeated data block and marked as a fragmented data block in step (S1), before determining its type, it further comprises: searching for the container where its referenced basic data block is located in the hash table, if it hits, marking it as a non-fragmented data block.

2. The method of claim 1, wherein, The step (S1) further comprises: For non-self-repeated data blocks in the un-screened containers, determine whether the container referenced by the non-self-repeated data block is currently marked as a sparse container, if so, mark the non-self-repeated data block as a fragmented data block; otherwise, mark the non-self-repeated data block as a non-fragmented data block; Wherein, the sparse container is a container with a utilization rate lower than a preset utilization rate threshold; the utilization rate of the container is the ratio of the sum of the sizes of the repeated data blocks and the differential data blocks referencing the container to the capacity of the container.

3. The method of claim 2, wherein the method further comprises: determining if the data block is a valid data block; and if the data block is a valid data block, then determining if the data block is a valid data block by comparing the data block with the data block stored in the data block location in the data block storage device. Further comprising: After the backup of the current data stream ends, calculate the utilization rate of each container, and mark the containers with a utilization rate lower than the utilization rate threshold as sparse containers.

4. The method of claim 1, wherein the method further comprises: determining if the data block is a valid data block; and if the data block is a valid data block, then determining if the data block is a valid data block by comparing the data block with the data block stored in the data block location in the data block storage device. The step (S2) further comprises: after writing the differential data block in the fragmented data block, updating the simulation recovery cache by using the container where the data block is located; and after writing the differential data block in the non-fragmented data block, updating the simulation recovery cache by using the container where the basic data block referenced by the differential data block is located and the container where the differential encoding is located.

5. The method of claim 1, wherein, The simulation recovery cache only caches container IDs.

6. A controller for use in a deduplication system based on delta compression, the controller comprising: The method comprises: a computer readable storage medium and a processor; The computer readable storage medium stores a computer program, and the processor is configured to read the computer program in the computer readable storage medium and execute the method for rewriting fragments in a differential compression based deduplication system according to any one of claims 1-5.

Images