Method, computer readable storage medium and file system for deduplication

Abstract

The present application relates to a method for deduplicating a file, a computer readable storage medium and a file system, the file having been divided into a plurality of data blocks and the plurality of data blocks having been deduplicated, the method comprising: defining a calculation range in the file according to a type of data block in the file, wherein the calculation range comprises a plurality of continuous data blocks in the file; generating an evaluation value according to the type of data block in the calculation range to determine whether to mark the data block in the calculation range; and re-dividing and deduplicating the marked data block in the file.

Description

Technical Field

[0001] This invention relates to a technique for deduplication, and more particularly to an optimization of deduplication. Background Technology

[0002] Existing file systems typically divide files into multiple data chunks and then perform deduplication on these chunks. Deduplication involves storing only one of the identical data chunks from different files, while the remaining identical chunks are not stored; instead, a pointer is used to indicate the stored chunk. For example, an email sent by a company manager to all employees might contain many files with identical content; this deduplication process can save a significant amount of storage space.

[0003] After the data block splitting described above, each file may have a residual data block at the end. Furthermore, if the content of some data blocks in a file is modified, the file system may re-split the modified data blocks, and new residual data blocks may also be generated at the end of these re-split data blocks. The content of these residual data blocks is usually unlikely to overlap with other data blocks, thus hindering deduplication. Therefore, how to reduce residual data blocks to improve the effectiveness of deduplication and save storage space has become an urgent technical problem to be solved in this field. Summary of the Invention

[0004] To address the aforementioned problems, the present invention provides a deduplication method, a computer-readable storage medium, and a file system for a file that has been divided into multiple data blocks and whose data blocks have undergone deduplication. The deduplication method includes: defining a calculation range in the file based on the types of data blocks in the file, wherein the calculation range includes multiple consecutive data blocks in the file; generating an evaluation value based on the types of data blocks in the calculation range to determine whether to mark the data blocks in the calculation range; and re-segmenting and deduplicating the marked data blocks in the file.

[0005] The present invention also provides a computer-readable storage medium for use in electronic devices, computing devices, data storage devices, computers or servers, storing instructions to perform the above-described deduplication method.

[0006] The present invention also provides a file system applied in a computing device or computer, wherein the above-described deduplication method is executed by a processor.

[0007] The deduplication method, computer-readable storage medium, and file system provided by this invention generate evaluation values ​​based on the types of data blocks in the file, and then determine whether to re-segment the data blocks and deduplicate them based on the evaluation values. Therefore, it can avoid inefficient re-segmentation, reduce fragmented data blocks, improve the efficiency of deduplication, and save storage space. Attached Figure Description

[0008] Figure 1 This is a schematic diagram illustrating an application scenario of a deduplication method according to an embodiment of the present invention.

[0009] Figure 2 This is an example diagram illustrating deduplication according to an embodiment of the present invention.

[0010] Figure 3 This is an example diagram of files and temporary storage areas according to an embodiment of the present invention.

[0011] Figure 4 and Figure 5 These are example diagrams illustrating the resegmentation of modified data blocks and the deletion of duplicate data according to different embodiments of the present invention.

[0012] Figure 6 This is a flowchart of a deduplication method according to an embodiment of the present invention.

[0013] Figure 7 This is an example diagram of a deduplication method according to an embodiment of the present invention.

[0014] Explanation of reference numerals in the attached figures

[0015] 1-9: Data blocks

[0016] 3',3”,5': Data block

[0017] 110: Electronic devices

[0018] 120: Storage device

[0019] 200: File

[0020] 710~760: Documents

[0021] 300: Temporary Storage Area

[0022] 501: Data Block Section

[0023] S61~S69: Methods and Procedures Detailed Implementation

[0024] The following describes the implementation of the present invention through specific embodiments. Those skilled in the art can easily understand other advantages and technical effects of the present invention from the content disclosed in this specification.

[0025] Figure 1 This is a schematic diagram illustrating an application scenario of a deduplication method according to an embodiment of the present invention.

[0026] The deduplication method of this embodiment can be applied to electronic device 110. Electronic device 110 can be a smartphone, computer, server, or other electronic or computing device with data processing capabilities. Furthermore, electronic device 110 can also be a data storage device or data storage server for near-end or cloud-based use. Electronic device 110 can connect to storage device 120 via various networks (such as wired or wireless networks). Storage device 120 is another electronic device located remotely or in the cloud, used to store files in a file system. Electronic device 110 can read files from the file system from storage device 120 and then write the modified files back to storage device 120.

[0027] Figure 2 This is an example diagram of deduplication according to an embodiment of the present invention, wherein the electronic device 110 divides the file 200 into six data blocks, which may have various different lengths. How to divide the file into multiple data blocks of different lengths is prior art. For example, a rolling hash (also known as recursive hashing or rolling checksum) or other splitting algorithms can be performed based on the data content of the file to determine the position of the split data blocks.

[0028] The first data block of file 200 is located on the far left of the diagram, and the sixth data block of file 200 is located on the far right. The number in each data block is its identifier; identical identifiers indicate identical data content. File 200 has a corresponding file descriptor, which records the identifier of each data block. For example, as shown in Table 1 below, the identifier for the first data block of file 200 is 1; the second and fourth data blocks of file 200 have the same content and are both numbered 2; the third and fifth data blocks of file 200 also have the same content and are both numbered 3; and the identifier for the sixth data block of file 200 is 4. Electronic device 110 performs deduplication on file 200; therefore, storage device 120 only stores data blocks 1, 2, 3, and 4. The file descriptor can be stored in storage device 120. When electronic device 110 accesses file 200, its file descriptor can be temporarily stored in electronic device 110. When needed, electronic device 110 can record and reconstruct file 200 according to the file description.

[0029] Table 1

[0030] Position order one II.4 Three or five six Data block code 1 2 3 4

[0031] When electronic device 110 accesses a file, a temporary storage area can be set up in electronic device 110 to temporarily store the data blocks needed by electronic device 110 and to temporarily store the data blocks modified by electronic device 110, so as to avoid frequent transfer of data blocks between electronic device 110 and storage device 120, which would reduce processing efficiency. For example Figure 3 As shown, the electronic device 110 sets up a temporary storage area 300 to temporarily store data blocks of the file 200. Two different application programs running on the electronic device 110 need to read the second data block 2 of the file 200. Therefore, the electronic device 110 retrieves data block 2 from the storage device 120 and stores it in the temporary storage area 300 for the application programs to read. Additionally, the fifth data block 3 of the file 200 is modified twice, resulting in partial data blocks 3' and 3''. The partial data block 3'' is generated later, thus overwriting part of data block 3' and being written to the temporary storage area 300.

[0032] Figure 4 This is an example diagram of the resegmentation and deduplication of modified data blocks according to an embodiment of the present invention. The left diagram shows the file 200 before resegmentation and deduplication, and the right diagram shows the file 200 after resegmentation and deduplication.

[0033] In this embodiment, the data blocks in the file can be divided into three categories: unmodified data blocks (not specifically marked in the figure), modified data blocks (marked with halftone dots in the figure), and fragmented data blocks (marked with diagonal lines in the figure). Unmodified data blocks refer to data blocks that have not been modified since being obtained from the storage device 120 and are not located at the end of the file, for example... Figure 4 Data blocks 1, 2, and 3 in the left diagram are considered modified data blocks, meaning data blocks that have been modified since being retrieved from storage device 120, or newly added data blocks, for example... Figure 4 Data blocks 3" and 5" in the left figure refer to residual data blocks, which are the end data blocks that do not conform to the segmentation criteria of the segmentation algorithm resulting from file splitting or re-segmentation, for example... Figure 4 Data blocks 4, 7, and 9 in the data.

[0034] exist Figure 4In the left figure, a portion of the fifth data block 3 of file 200 is modified into a portion of data block 3” in the temporary storage area 300, and a new data block 5’ is added to the temporary storage area 300. The electronic device 110 determines whether the temporary storage area 300 is dirty based on preset conditions. For example, if the total of the modified and newly added portions of all data blocks in file 200 reaches or exceeds a preset percentage of the file size of file 200 (different percentages can be set according to actual needs; for example, in this embodiment, it is set to the modified and newly added data blocks being greater than or equal to the file size of file 200). If the file size of file 200 is 25% (or less than 25% of the file size), then the electronic device 110 determines that the temporary storage area 300 is contaminated. Alternatively, if the cumulative number of modifications and additions to data blocks of file 200 has reached a preset value (different preset values ​​can be set according to actual needs), then the electronic device 110 determines that the temporary storage area 300 is contaminated. When the electronic device 110 determines that the temporary storage area 300 is contaminated, it will merge the latest version of the modified data block portion 3” in the temporary storage area 300 and the remaining portion of data block 3 obtained from the storage device 120, and then re-segment and delete duplicate data on the merged result. In addition, the electronic device 110 will also segment and delete duplicate data on the newly added data block 5’. The results of the above segmentation and deduplication are as follows: Figure 4 As shown in the diagram on the right, the fifth data block 3 of file 200 has been re-splitted into data blocks 6 and 7, and the newly added seventh data block 5' has been split into data blocks 8 and 9. Before the re-splitting, file 200 contained only one fragment data block 4, while after the re-splitting, file 200 has two additional fragment data blocks 7 and 9. Figure 4 As the example shows, each modification to a file may generate at least one fragmented data block. The content of these fragmented data blocks is usually unlikely to duplicate other data blocks, so the increased fragmented data blocks reduce the effectiveness of deduplication and waste storage space.

[0035] Figure 5 This is an example diagram illustrating the resegmentation of modified data blocks and the deletion of duplicate data according to another embodiment of the present invention.

[0036] First, like Figure 4 As shown in the left figure, a portion of the fifth data block 3 of file 200 has been modified to become part of data block 3” in the temporary storage area 300, and a new data block 5’ has been added to the temporary storage area 300. When the electronic device 110 determines that the temporary storage area 300 is contaminated, then as follows... Figure 5As shown in the left figure, the latest version 501 of data block 3, excluding the modified portion 3”, is obtained from the storage device 120 and stored in the temporary storage area 300. The residual data block 4 between the fifth data block 3 and the newly added data block 5’ is also read into the temporary storage area 300. Then, the electronic device 110 merges data block portions 501 and 3”, as well as data blocks 4 and 5’, into a continuous data segment. The merged continuous data is then re-segmented and duplicate data is deleted. The result is as follows: Figure 5 As shown in the right figure, the continuous data has been re-divided into data blocks 6, 7, and 8.

[0037] In this embodiment, as Figure 5 As shown in the left figure, file 200 before re-splitting only has one residual data block 4, and as... Figure 5 As shown in the right figure, the re-segmented file 200 still contains only one fragment of data block 8. Therefore, Figure 5 The method is more Figure 4 This method is more effective at deduplication and saves storage space, while also avoiding the problem of continuously adding fragmented data blocks. The process is as follows: Figure 6 As shown.

[0038] Figure 6 This is a flowchart of a deduplication method according to an embodiment of the present invention, which can be executed by an electronic device 110. As described above, the electronic device 110 can execute the deduplication method on a file after the file has been divided into multiple data blocks, the multiple data blocks have undergone deduplication, and the corresponding temporary storage area has become contaminated. The following refers to... Figure 6 This section describes the process of the deduplication method.

[0039] In step S61, based on the type of data blocks in the file, the beginning and end of the calculation range are defined in the file to delineate the calculation range. Specifically, the beginning of the calculation range is set to the first modified data block or fragment data block in the file, and the end of the calculation range is set to the next modified data block or fragment data block in the file after the beginning. The calculation range includes all data blocks in the file from the beginning to the end, and then proceeds to step S62.

[0040] In step S62, an evaluation value for the computational scope is generated based on the types of data blocks within the scope. This evaluation value is generated based on the number and distribution of modified data blocks within the computational scope, and also based on the number of fragmented data blocks within the computational scope. Specifically, the evaluation value is the benefit value of the computational scope divided by the cost value. The benefit value equals the number of modified and fragmented data blocks within the computational scope plus the number of consecutive segments of modified data blocks, and the cost value is the total number of data blocks within the computational scope. The aforementioned consecutive segments of modified data blocks are defined as segments containing at least one continuous sequence of modified data blocks.

[0041] For example, Figure 7 If the calculation range of file 710 is from the first data block to the third data block, then there are 3 modified data blocks and fragmented data blocks in the calculation range, there are 2 consecutive segments of modified data blocks in the calculation range (the first data block and the third data block respectively), and there are 3 data blocks in the calculation range, so the evaluation value is (2+3) / 3=5 / 3.

[0042] For example, Figure 7 If the calculation range of file 740 is from the first data block to the eighth data block, then there are 4 modified data blocks in the calculation range, no residual data blocks, and 2 consecutive segments of modified data blocks in the calculation range (the first to the third data block and the eighth data block respectively). In addition, there are 8 data blocks in the calculation range. Therefore, the evaluation value is (4+2) / 8 = 6 / 8 = 3 / 4.

[0043] For example, Figure 7 If the calculation range of file 760 is from the first data block to the sixth data block, then there are 3 modified data blocks and fragmented data blocks in the calculation range, there are 2 consecutive segments of modified data blocks in the calculation range (the first data block and the sixth data block respectively), and there are 6 data blocks in the calculation range. Therefore, the evaluation value is (3+2) / 6 = 5 / 6.

[0044] Next, in step S63, it is checked whether the evaluated value of the calculated range is greater than or equal to a preset threshold. In one embodiment, the threshold can be selected from a range greater than 0 and less than or equal to 2. In another embodiment, the threshold can be selected from a range greater than or equal to 0 and less than or equal to 2. If the evaluated value is greater than or equal to the threshold, the process proceeds to step S66; otherwise, the process proceeds to step S64.

[0045] In step S64, check if the end of the calculation range is the end of the file. If the end is the end of the file, proceed to step S69; otherwise, proceed to step S65.

[0046] In step S65, the start and end settings of the calculation range are changed. Specifically, the start end is changed to the first modified data block or fragment data block in the file starting from the end end (if the end end is a modified data block or fragment data block, the start end is changed to the end end), and then the end end is changed to the next modified data block or fragment data block in the file after the start end. Then the process returns to step S62.

[0047] In step S66, the data blocks in the calculation range are marked, and then proceed to step S67.

[0048] In step S67, check if the end of the calculation range is the end of the file. If the end is the end of the file, the process proceeds to step S69; otherwise, the process proceeds to step S68.

[0049] In step S68, the end setting of the calculation range is changed. Specifically, the end is set to the next modified data block or fragment data block in the file after the end, and then the process returns to step S62.

[0050] Finally, in step S69, the marked data blocks in the file are re-segmented and duplicate data is deleted. Specifically, the electronic device 110 first checks whether each modified data block has been completely modified. If any data block has not been completely modified, the electronic device 110 retrieves the latest version of the remaining portion of that data block from the storage device 120, excluding the modified portion, and stores it in a temporary storage area. Furthermore, if any marked data blocks have not yet been stored in the temporary storage area, the electronic device 110 retrieves the latest version of that data block from the storage device 120 and stores it in the temporary storage area. Then, the electronic device 110 merges the marked data blocks in the temporary storage area into a continuous data segment, re-segments and deletes duplicate data from the merged continuous data, and updates the file description record of the file and the data blocks stored in the storage device 120 accordingly.

[0051] It should be noted that the choice of the threshold in step S63 affects the computational cost and the number of residual data blocks in this deduplication method. For example, if the threshold is set to 0, the scope of file resegmentation and deduplication will be maximized, minimizing the number of residual data blocks, but maximizing the computational cost. If the threshold is set to other values, the number of residual data blocks will gradually increase as the threshold increases, while the computational cost will gradually decrease.

[0052] Figure 6 Examples of deduplication methods are as follows: Figure 7 As shown, for Figure 7 In the example, the threshold is set to 1.

[0053] First, for file 710, the initial calculation range is set to the first and second data blocks. At this point, the evaluation value is 3 / 2 > 1, so the first and second data blocks are marked. Next, the calculation range is changed to the first and third data blocks. At this point, the evaluation value is 5 / 3 > 1, so the first and third data blocks are marked. Finally, the marked first and third data blocks are re-segmented and duplicate data is removed.

[0054] For file 720, the initial calculation range was set to the first to third data blocks. At this point, the evaluation value was 4 / 3 > 1, so the first to third data blocks were marked. Finally, the marked first to third data blocks were re-segmented and duplicate data was removed.

[0055] For file 730, the initial calculation range was set to the first to sixth data blocks. At this point, the evaluation value was 4 / 6 < 1, so the data blocks were not marked. Finally, instead of re-splitting the data blocks of the entire calculation range and deleting duplicate data, the modified first and sixth data blocks were processed separately in the traditional way.

[0056] For file 740, the initial calculation scope was set to the first and second data blocks. At this point, the evaluation value was 3 / 2 > 1, so the first and second data blocks were marked. Next, the calculation scope was changed to the first and third data blocks. At this point, the evaluation value was 4 / 3 > 1, so the first and third data blocks were marked. Then, the calculation scope was changed to the first and eighth data blocks. At this point, the evaluation value was 6 / 8 < 1, so no data blocks were marked. Finally, the marked first and third data blocks were re-segmented and duplicate data was removed. The modified eighth data block was processed separately using the traditional method.

[0057] For file 750, the initial calculation range was set to the first to third data blocks. The evaluation value at this point was 3 / 3 = 1, so the first to third data blocks were marked. Next, the calculation range was changed to the first to fifth data blocks. The evaluation value at this point was 5 / 5 = 1, so the first to fifth data blocks were marked. Finally, the marked first to fifth data blocks were re-segmented and duplicate data was removed.

[0058] For file 760, the initial calculation range was set to the first to fourth data blocks. At this time, the evaluation value was 3 / 4 < 1, so no data block was marked. Next, the calculation range was changed to the fourth to sixth data blocks. At this time, the evaluation value was 3 / 3 = 1, so the fourth to sixth data blocks were marked. Finally, the marked fourth to sixth data blocks were re-segmented and duplicate data was deleted, while the modified first data block was processed separately in the traditional way.

[0059] Furthermore, the present invention also provides a computer-readable storage medium, such as a memory, magnetic tape, magnetic disk, optical disk, or hard disk. This computer-readable storage medium stores instructions for performing the deduplication method described above. This computer-readable storage medium can be used in electronic device 110.

[0060] Furthermore, the present invention also provides a file system that executes the above-described deduplication method via a processor. This file system can be applied to electronic device 110.

[0061] In summary, the deduplication method, computer-readable storage medium, and file system provided by this invention generate evaluation values ​​based on the types of data blocks in the file, and then determine whether to re-segment the data blocks and deduplicate them based on the evaluation values. Therefore, it can avoid inefficient re-segmentation, reduce fragmented data blocks, improve the efficiency of deduplication, and save storage space.

[0062] The above embodiments are merely illustrative of the principles and technical effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify and alter the above embodiments without departing from the spirit and scope of the invention. Therefore, the scope of protection of the present invention should be as set forth in the claims.

Claims

1. A deduplication method applied to an electronic device, specifically for a file that has been divided into multiple data blocks and whose data blocks have undergone deduplication, wherein the electronic device is connected to a storage device via a network, and the storage device stores the file, the deduplication method comprising the following steps: A temporary storage area is provided in the electronic device to temporarily store the data blocks of the file; The calculation scope is defined in the file according to the type of data blocks in the file, wherein the calculation scope includes multiple consecutive data blocks in the file; Based on the type of data blocks within the computational scope, an evaluation value is generated to determine whether to label the data blocks within the computational scope. If there are data blocks in the modified data blocks of the file that have not been completely modified, then the latest version of the remaining part of the data block excluding the modified part is obtained from the storage device, and the latest version of the remaining part is stored in the temporary storage area. If there are data blocks in the marked data blocks of the file that have not yet been stored in the temporary storage area, then the latest version of the data block is obtained from the storage device and stored in the temporary storage area. Merge the marked data blocks of this file in the temporary storage area into a continuous data block; The continuous data segment was re-segmented and duplicate data was removed; and Based on the results of re-segmenting and deleting duplicate data of the continuous data segment, the file description record of the file is updated, and the data block of the file stored in the storage device is also updated.

2. The deduplication method as described in claim 1, wherein, The file contains multiple data blocks of varying lengths, and these data blocks can be categorized as unmodified data blocks, modified data blocks, and fragmented data blocks.

3. The deduplication method as described in claim 2, wherein, The step of defining the scope of the calculation includes the following sub-steps: Set the beginning of this calculation range to the first modified or residual data block in the file; and The end of the calculation range is set as the next modified data block or fragment data block in the file after the beginning, wherein the calculation range includes multiple consecutive data blocks in the file from the beginning to the end.

4. The deduplication method as described in claim 3, wherein, The method also includes the following steps: If the evaluation value is greater than or equal to a threshold, the end is set to the next modified data block or fragment data block in the file after the end, and the evaluation value is recalculated to determine whether to mark the data block in the calculation range.

5. The deduplication method as described in claim 3, wherein, The method also includes the following steps: If the evaluation value is less than a threshold, the beginning is set to the first modified data block or fragment data block in the file starting from the end, and the end is set to the next modified data block or fragment data block in the file after the beginning. The evaluation value is then recalculated to determine whether to mark the data block in the calculation range.

6. The deduplication method as described in claim 1, wherein, The step of deciding whether to tag a data block within the computational scope includes the following sub-steps: If the evaluation value is greater than or equal to a threshold, the data block in the calculation range is marked; if the evaluation value is less than the threshold, the data block in the calculation range is not marked.

7. The deduplication method as described in claim 6, wherein, The threshold is greater than zero and the threshold is less than or equal to two.

8. The deduplication method as described in claim 6, wherein, The threshold is greater than or equal to zero and less than or equal to two.

9. The deduplication method as described in claim 1, wherein, The evaluation value is generated based on the number and distribution of modified data blocks in the calculation range, and also based on the number of residual data blocks in the calculation range.

10. The deduplication method as described in claim 9, wherein, The evaluation value is equal to the benefit value divided by the cost value. The benefit value is equal to the number of modified data blocks and residual data blocks in the calculation range plus the number of consecutive segments of modified data blocks. The cost value is the total number of data blocks in the calculation range.

11. A computer-readable storage medium applied to an electronic device, storing instructions for performing the deduplication method as described in any one of claims 1 to 10.

12. A file system applied to an electronic device, wherein a processor executes the deduplication method as described in any one of claims 1 to 10.

Images