A data fingerprint detection method, device and storage medium
By dynamically merging infrequently changing data blocks in large files, the problem of excessive resource consumption in comparing large file blocks in existing technologies is solved, and more efficient data fingerprint processing is achieved.
Patent Information
- Application Number
- CN202010576124.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-06-22
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2040-06-22
AI Technical Summary
When dividing large files into blocks, existing technologies still require comparing the data fingerprints of each data block one by one, even though the files are large, because the changes are concentrated in one part. This results in excessive time and resource consumption.
By obtaining a change record table of data blocks, and using wear factors and aggregation segment factors to dynamically merge data blocks that do not change frequently, the number of data blocks is reduced, thus reducing the resource consumption for data fingerprint comparison.
It effectively shortens the data fingerprint comparison time, saves resource consumption, and improves system processing capacity and throughput.
Smart Images

Figure CN111949652B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data processing, and in particular to a data fingerprint detection method, device and storage medium. Background Art
[0002] With the continuous development of data processing technology and network transmission capabilities, the storage and transmission of large and even super-large files are becoming more and more common. For large files, especially super-large files, differential backup and transmission by comparing data fingerprints is particularly important, which can not only greatly reduce the transmission bandwidth, but also improve the processing capacity of the file storage system.
[0003] In existing solutions, regardless of the file size, large files are usually divided into blocks of a fixed size, and a data fingerprint is established for each data block.
[0004] The inventors have found that for some large files, although the files are large, the parts that change frequently are concentrated. In this case, if the large files are still divided into blocks of fixed size, then during each backup or transmission, even if most of the data blocks are unchanged, the data fingerprints of each data block still need to be compared one by one, which not only takes a long time, but also consumes a lot of computer resources.
[0005] It can be seen that how to improve the above-mentioned large file segmentation strategy and improve the processing efficiency of fingerprint data is a technical problem that needs to be solved. Summary of the invention
[0006] In response to the above problems, the inventors have creatively come up with the idea that in this case, if the frequently changing data can be accurately located and the infrequently changing data blocks can be merged, the number of blocks can be greatly reduced, thereby shortening the time for data fingerprint comparison and saving the resources required for comparing data fingerprints.
[0007] Based on the above invention ideas, the inventor provides a data fingerprint detection method, device and storage medium.
[0008] According to a first aspect of an embodiment of the present invention, a data fingerprint detection method is provided, the method comprising performing the following operations when a first condition is met: obtaining a first change record table for all data blocks, each entry in the first change record table recording historical information of changes in each data block within a period of time, and each data block corresponding to a data fingerprint; performing wear processing on the first change record table according to a preset wear factor to obtain a second change record table; detecting whether the second change record table has a mergeable entry corresponding to a preset aggregation segment factor and satisfying an aggregation condition, and if so, merging at least two data blocks corresponding to the mergeable entry into one data block.
[0009] According to an implementation manner of an embodiment of the present invention, before the first condition is met, the method also includes: obtaining the file to be differentially processed; dividing the file into blocks to obtain data blocks; creating a first change record table, wherein each table entry in the first transformation record table corresponds to each data block, and is used to record historical information of changes in each data block within a period of time.
[0010] According to an implementation manner of an embodiment of the present invention, after creating the first change record table, the method also includes performing the following operations when performing differential processing on the file: obtaining a first data fingerprint of all data blocks, the first data fingerprint is the latest data fingerprint corresponding to each data block; obtaining a second data fingerprint of all data blocks, the second data fingerprint is the last stored data fingerprint corresponding to each data block; detecting whether the first data fingerprint and the second data fingerprint of each data block are the same, and if different, updating the table entry corresponding to the corresponding data block in the first change record table.
[0011] According to an implementation of an embodiment of the present invention, updating the table entry corresponding to the corresponding data block in the first change record table includes: obtaining all table entries corresponding to the corresponding data block; and updating each table entry in all the table entries in turn.
[0012] According to an implementation manner of an embodiment of the present invention, each entry in the first transformation record table is used to record historical information of changes in each data block within a period of time, including: each entry in the first transformation record table is used to record the number of times each data block changes within a period of time; accordingly, the entry corresponding to the data block is updated, including: adding one to the number recorded in the entry.
[0013] According to an implementation of the embodiment of the present invention, the first condition includes reaching a predetermined time.
[0014] According to an implementation manner of an embodiment of the present invention, the first change record table includes a change record table using a bitmap as a storage structure.
[0015] According to an implementation of an embodiment of the present invention, before performing wear processing on the first change record table according to a preset wear factor to obtain the second change record table, the method further includes: determining the wear factor and the aggregation segment factor.
[0016] According to a second aspect of an embodiment of the present invention, a data fingerprint detection device includes: a first condition detection module, used to detect whether a first condition is met; a change record table acquisition module, used to obtain a first change record table of all data blocks; a wear module, used to perform wear processing on the first change record table according to a preset wear factor to obtain a second change record table; an aggregation segment factor detection module, used to detect whether there is a mergeable table item in the second change record table that corresponds to a preset aggregation segment factor and meets the aggregation condition; an aggregation module, used to merge at least two data blocks corresponding to the mergeable table item into one data block.
[0017] According to a third aspect of an embodiment of the present invention, a storage medium is provided, on which program instructions are stored, wherein the program instructions are used to execute any of the above-mentioned data fingerprint detection methods when running.
[0018] The embodiment of the present invention provides a data fingerprint detection method, device and storage medium. The method uses a change record table to record the historical information of each data block that has changed within a period of time. When a certain condition is met, the change record table is subjected to wear processing according to a preset wear factor; the change record table after the wear processing is detected to see whether there is a mergeable table item corresponding to the preset aggregation segment factor and meeting the aggregation condition. If so, at least two data blocks corresponding to the mergeable table item are merged into one data block. In this way, data blocks that do not change frequently can be dynamically merged based on information such as the number of times or frequency of each data transformation, thereby reducing the number of data blocks and correspondingly reducing the number of data fingerprints to be processed, thereby further reducing resource consumption and greatly improving the system's processing power and throughput.
[0019] It should be understood that the teachings of the embodiments of the present invention are not required to achieve all the beneficial effects described above, but specific technical solutions can achieve specific technical effects, and other implementations of the embodiments of the present invention can also achieve beneficial effects not mentioned above. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] The above and other objects, features and advantages of the exemplary embodiments of the present invention will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present invention are shown in an exemplary and non-limiting manner, in which:
[0021] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts.
[0022] Figure 1 The following is a schematic diagram of the implementation process of the data fingerprint detection method according to an embodiment of the present invention;
[0023] Figure 2This is a schematic diagram of a specific implementation flow of the application data fingerprint detection method according to the first embodiment of the present invention;
[0024] Figure 3 The figure is a schematic diagram of the structure of a data fingerprint detection device according to an embodiment of the present invention. DETAILED DESCRIPTION
[0025] In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.
[0026] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples, unless they are contradictory.
[0027] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.
[0028] According to a first aspect of an embodiment of the present invention, a data fingerprint detection method is provided. Figure 1 As shown, the method includes performing the following operations when the first condition is met: operation 110, obtaining a first change record table for all data blocks, each table item in the first change record table records the historical information of changes in each data block within a period of time, and each data block corresponds to a data fingerprint; operation 120, performing wear processing on the first change record table according to a preset wear factor to obtain a second change record table; operation 130, detecting whether there is a mergeable table item in the second change record table that corresponds to a preset aggregation segment factor and meets the aggregation condition, and if so, merging at least two data blocks corresponding to the mergeable table item into one data block.
[0029] In operation 110, in order to facilitate differential processing of a large file, such as differential backup, the large file is usually divided into blocks to obtain multiple data blocks, and then a data fingerprint is created for each data block. In this way, in the subsequent differential processing, the data fingerprint of each data block can be compared to determine whether the data of the data block has changed.
[0030] In order to record the historical information of the changes of these data blocks within a period of time, the data fingerprint detection method of the embodiment of the present invention uses a first change record table to record the changes of each data block. Each entry in the first change record table corresponds to a data block and records the historical information of the changes of the data block within a period of time. With the historical information of the changes of each data block within a period of time, the law and trend of the changes of each data can be grasped, and the data blocks that have not changed for a long time or have a low change frequency can be dynamically merged, thereby reducing the number of data blocks and fingerprint data corresponding to the data blocks, and further improving the processing capacity and throughput of the system.
[0031] In this implementation, the specific form and data structure of the first change record table are not limited, and the implementer may select any applicable specific form and data structure according to specific implementation conditions.
[0032] Among them, a period of time refers to a period of time in which differential processing can occur more than twice, otherwise it is not enough to obtain the rules related to the number or frequency of data changes, and it is difficult to reflect the beneficial effects of the data fingerprint detection method of the embodiment of the present invention. Theoretically, the more data accumulated during this period of time, the easier it is to accurately locate the parts of the file that do not change frequently, so that the effect of data merging is better. However, this period of time cannot be too long, because the data merging is carried out after this period of time. If this period of time is too long, the infrequently changing data blocks cannot be merged in time, and the outstanding effects of the data fingerprint detection method of the embodiment of the present invention cannot be repeatedly reflected. Therefore, the setting of this period of time can be based on experimental data that is closer to the actual data, and adjusted according to the experimental results to achieve better practical application effects.
[0033] The historical information that has changed mainly refers to the number or frequency of data changes in each data block within a period of time.
[0034] A data fingerprint refers to a sequence value that can be used to determine the integrity and identity of data. For example, it is an identifiable summary value obtained by performing a certain operation on the data content using a certain algorithm. Commonly used algorithms include the message digest algorithm (MD5).
[0035] In operation 120, the wear factor refers to a value based on which uniform wear processing is performed on each item in the first change record table. Wear processing mainly refers to subtracting the wear factor from the value of each item, dividing it by the wear factor, or using a specific function with the wear factor as a parameter to calculate the value of each item to obtain a new value. The second change record table is a table composed of the new values obtained after these wear processing. Wear processing can highlight the differences in data on the one hand, and on the other hand, it can make it easier to find data blocks with less changes, reducing the complexity of subsequent calculations.
[0036] In operation 130, the aggregation segment factor mainly refers to the number of data blocks that are compared and merged at one time when merging data blocks. For example, if the aggregation segment factor is 2, it means that two data blocks are compared each time, and if the two data blocks meet the aggregation conditions, the two data blocks are merged into one data block; if the aggregation segment factor is 3, it means that three data blocks are compared each time, and if the three data blocks meet the aggregation conditions, the three data blocks are merged into one data block; and so on. Here, the aggregation segment factor should be set to a value greater than 2, otherwise it is meaningless. The aggregation condition refers to the condition that the data blocks corresponding to the aggregation segment factor must meet to be merged. Usually, this condition requires that the change history information recorded by the data blocks corresponding to the aggregation segment factor is the same or similar, and is lower than a certain value, so that adjacent and infrequently changing mergeable data blocks can be found. The aggregation conditions in the specific implementation process can also be determined by the implementer according to the implementation conditions and the goals or effects that the implementer wants to achieve.
[0037] This operation is a substantial operation to achieve data block merging. It will merge the historical information that has changed, especially the data blocks that have not changed frequently within a period of time. This will help improve the efficiency of differential processing, reduce the calculation and comparison of data fingerprints, greatly reduce resource consumption, and improve system throughput.
[0038] According to an implementation manner of an embodiment of the present invention, before the first condition is met, the method also includes: obtaining the file to be differentially processed; dividing the file into blocks to obtain data blocks; creating a first change record table, wherein each table entry in the first transformation record table corresponds to each data block, and is used to record historical information of changes in each data block within a period of time.
[0039] Among them, differential processing refers to processing only the newly added and modified data after the last full processing each time, such as differential backup, etc. The files to be differentially processed are usually large files or even super large files. In order to perform differential processing, the file must first be divided into blocks. When there is no historical information on data changes, the file can be divided into data blocks of equal size based on an empirical value, and a change record table can be created, and each entry in the change record table is used to record the changes of each data block over a period of time. This change record table provides the data fingerprint detection method of the embodiment of the present invention with historical information on the changes of data blocks over a period of time, and is the data basis for merging data blocks.
[0040] According to an implementation manner of an embodiment of the present invention, after creating the first change record table, the method also includes performing the following operations when performing differential processing on the file: obtaining a first data fingerprint of all data blocks, the first data fingerprint is the latest data fingerprint corresponding to each data block; obtaining a second data fingerprint of all data blocks, the second data fingerprint is the last stored data fingerprint corresponding to each data block; detecting whether the first data fingerprint and the second data fingerprint of each data block are the same, and if different, updating the table entry corresponding to the corresponding data block in the first change record table.
[0041] In this embodiment, each time a file is differentially processed, the changed data blocks are recorded at the same time, and the corresponding table entries of the changed data blocks in the first change record table are updated. In this way, historical information such as whether each data block has changed within a period of time, the number of changes or the frequency of changes, etc. can be recorded.
[0042] According to an implementation of an embodiment of the present invention, updating the table entry corresponding to the corresponding data block in the first change record table includes: obtaining all table entries corresponding to the corresponding data block; and updating each table entry in all the table entries in turn.
[0043] When multiple data blocks are merged after being worn out because they meet the mergeability conditions corresponding to the aggregation segment factor, the merged data blocks will correspond to multiple table entries. At this time, all table entries corresponding to the merged data blocks need to be updated in sequence.
[0044] According to an implementation manner of an embodiment of the present invention, each entry in the first transformation record table is used to record historical information of changes in each data block within a period of time, including: each entry in the first transformation record table is used to record the number of times each data block changes within a period of time; accordingly, the entry corresponding to the data block is updated, including: adding one to the number recorded in the entry.
[0045] In this embodiment, each entry in the first change record table records the number of times each data block changes within a period of time. This recording method is simple and easy to implement, and can intuitively reflect the number of changes or frequencies of each data block within a period of time, as well as the different frequencies of changes in different contents of the entire file, so that some infrequently changing data blocks can be dynamically merged according to the frequency of data block changes, thereby greatly reducing the number of data blocks, as well as the calculation and comparison operations of data fingerprint detection.
[0046] According to an implementation of the embodiment of the present invention, the first condition includes reaching a predetermined time.
[0047] In this embodiment, the wear processing and the merging of data blocks are performed regularly. That is, the wear processing and the merging of data blocks are performed once every preset time. This embodiment is simple and easy to implement, and has been proven to be effective in practice. As mentioned above, whether the preset time is reasonable will affect the implementation effect of the data fingerprint detection method of the embodiment of the present invention. Therefore, experiments can be conducted using experimental data similar to actual data, and a time for better implementation effect can be determined based on the experimental results.
[0048] According to an implementation of an embodiment of the present invention, the first change record table includes a change record table using a bitmap as a storage structure.
[0049] In this embodiment, a bitmap is used as the storage structure of the first change record table. The so-called bitmap is to use a bit to mark the value (Value) corresponding to a certain element, and the key (Key) is the element. The use of a bitmap can also reduce each unit of the statistical array used for counting and sorting to a bit-level Boolean array. Since the data is stored in bits, it can greatly save storage space and simplify the complexity of calculation.
[0050] According to an embodiment of the present invention, before performing wear processing on the first change record table according to a preset wear factor to obtain the second change record table, the method further includes: determining the wear factor and the aggregation segment factor.
[0051] In this embodiment, the wear factor and the aggregation segment factor are both preset values. To determine the appropriate values of the wear factor and the aggregation segment factor, several candidate values can be specified based on experience, and then experiments can be conducted using experimental data similar to actual data, and then values with better implementation effects can be selected from these candidate values based on the experimental results.
[0052] The following is combined Figure 2The specific implementation process of the data fingerprint detection method applied in the first embodiment of the present invention is introduced. In this application, the data fingerprint detection method of the embodiment of the present invention is mainly used for differential backup of large files. In order to facilitate the description of the process and reflect the effect, it is assumed that the size of the file is 64M, but in fact the size of the file may be much larger than 64M. First, it is assumed that before the data fingerprint detection method of the embodiment of the present invention is implemented, a full backup of the file has been performed, and when a differential backup of the file is performed subsequently, only the transformed data blocks need to be transmitted. Among them, the process of performing differential backup and transmitting differential data mainly includes the following steps:
[0053] Step 2010, setting the minimum data block unit and creating a corresponding change record table;
[0054] Assume that at the beginning of differential backup, the minimum data block unit is set to 4M, and the file is divided into equal parts, resulting in 16 data blocks. A change record table using a bitmap as a storage structure is created for the 16 data blocks, and the initial value of the bit corresponding to each data block is set to 1. At this time, the data status of the change record table is shown in Table 1:
[0055] 1(1) 1(2) 1(3) 1(4) 1(5) 1(6) 1(7) 1(8) 1(9) 1(10) 1(11) 1(12) 1(13) 1(14) 1(15) 1(16)
[0056] Table 1
[0057] In Table 1, the value in each table cell represents the value stored in the table item corresponding to each data block, and the number in the bracket after the value represents the number of the data block. The data state represented by Table 1 is that the value of the change record table item corresponding to each data block from the 1st data block to the 16th data block is 1.
[0058] Step 220, reading the file by data blocks;
[0059] Step 2030, reading the data fingerprint of the data block;
[0060] Step 2040, transmitting differential data according to the comparison result of the data fingerprint, and updating the transformation record table;
[0061] Assume that, according to the comparison result of the data fingerprint this time, it is known that the 2nd, 6th, 7th, 10th, and 13th data blocks have changed, then add 1 to the table entries of the 2nd, 6th, 7th, 10th, and 13th data blocks respectively, and obtain the data status change record table shown in Table 2:
[0062] 1(1) 2(2) 1(3) 1(4) 1(5) 2(6) 2(7) 1(8) 1(9) 2(10) 1(11) 1(12) 2(13) 1(14) 1(15) 1(16)
[0063] Table 2
[0064] Step 2050, determining whether the preset wear and merge time has been reached, if so, proceeding to step 2060, if not, returning to step 2020, and continuing to wait for the next differential processing;
[0065] During this differential backup, the predetermined wear and merge time has not been reached, so the process returns to step 2020 and continues to wait for the second differential backup.
[0066] Assuming that according to the fingerprint detection comparison results of step 2020 to step 2040, it is known that the 3rd, 7th, 8th, 11th, and 14th data blocks have changed during the second differential backup, then the table entries of the 3rd, 7th, 8th, 11th, and 14th data blocks are added by 1 respectively, and the data status is obtained as shown in Table 3. Change record table:
[0067] 1(1) 2(2) 2(3) 1(4) 1(5) 2(6) 3(7) 1(8) 1(9) 2(10) 2(11) 1(12) 2(13) 2(14) 1(15) 1(16)
[0068] Table 3
[0069] Assuming that the predetermined wear and merge time has been reached, continue with step 2060.
[0070] Step 2060: Perform wear aggregation on the transformation record table and update the segmentation information. Then return to step 2020 to continue waiting for the next differential processing, and in the next differential processing, use the data blocks specified by the new segmentation information to read the file.
[0071] Assuming that the wear factor in this application is 1, the change record table shown in Table 3 is first subjected to wear processing. In this application, the wear processing is performed by subtracting the wear factor from each table item, and the change record table shown in Table 4 is obtained:
[0072] 0(1) 1(2) 1(3) 0(4) 0(5) 1(6) 2(7) 0(8) 0(9) 1(10) 1(11) 0(12) 1(13) 1(14) 0(15) 0(16)
[0073] Table 4
[0074] Assume that in this application, the aggregation segment factor is 2. When merging the aggregation segments, first, read the values of the data blocks corresponding to the aggregation segment factor, for example, the values corresponding to the 1st and 2nd data blocks are (0, 1); the values corresponding to the 3rd and 4th data blocks are (1, 0); the values corresponding to the 5th and 6th data blocks are (0, 1); the values corresponding to the 7th and 8th data blocks are (2, 0); the values corresponding to the 9th and 10th data blocks are (0, 1); the values corresponding to the 11th and 12th data blocks are (1, 0); the values corresponding to the 13th and 14th data blocks are (1, 1); and the values corresponding to the 15th and 16th data blocks are (0, 0).
[0075] Assume that the aggregation condition in this application is that the aggregation segment values corresponding to the aggregation segment factor are all less than or equal to zero. The actual meaning of this aggregation condition can be understood as that within a period of time, the number of changes in the data blocks in the aggregation segment corresponding to the aggregation segment factor is less than or equal to zero after wear processing; in other words, the number of changes in the data blocks in the aggregation segment corresponding to the aggregation segment factor is less than or equal to the value set by the wear factor. According to this condition, it can be known that in the data blocks in the above-mentioned aggregation segment, the values of the 15th and 16th data blocks meet this aggregation condition, so the 15th and 16th data blocks can be merged into one data block.
[0076] After merging the data blocks in Table 4 according to the above aggregation segment factors and aggregation conditions, a change record table of data status as shown in Table 5 is obtained:
[0077] 0(1) 1(2) 1(3) 0(4) 0(5) 1(6) 2(7) 0(8) 0(9) 1(10) 1(11) 0(12) 1(13) 1(14) 0(15) 0(15)
[0078] Table 5
[0079] Among them, the original 15th and 16th data blocks are merged into data block 15, and the number of the entire data blocks is reduced from 16 to 15. Accordingly, in the subsequent differential backup, only 15 data fingerprints need to be calculated, maintained and compared.
[0080] After that, you can go back to step 2020 and continue to wait for the next differential backup, and use the data blocks specified by the new segmentation information to read the file, and repeat the above process to continuously dynamically update the change record table, handle wear and merge data blocks.
[0081] It should be noted that, in the change record table shown in Table 5, since the 15th data block is the merger of the previous 15th and 16th data blocks, the 15th data block will correspond to two table entries. When updating the change record table according to the change of the 15th data block, it is necessary to obtain all the table entries corresponding to the 15th data block (that is, table entry 15 and table entry 16), and update all the table entries corresponding to the 15th data block in turn.
[0082] It should be noted that the specific implementation process of the application data fingerprint detection method of the first embodiment of the present invention is only an exemplary description, and the implementer can adopt any applicable implementation method according to the implementation conditions to implement the data fingerprint detection method of the embodiment of the present invention.
[0083] According to a second aspect of an embodiment of the present invention, a data fingerprint detection device is provided. Figure 3As shown, the device 30 includes: a first condition detection module 301, used to detect whether the first condition is met; a change record table acquisition module 302, used to obtain a first change record table of all data blocks; a wear module 303, used to perform wear processing on the first change record table according to a preset wear factor to obtain a second change record table; an aggregation segment factor detection module 304, used to detect whether there is a mergeable table item in the second change record table that corresponds to a preset aggregation segment factor and meets the aggregation condition; an aggregation module 305, used to merge at least two data blocks corresponding to the mergeable table item into one data block.
[0084] According to an implementation mode of an embodiment of the present invention, the device 30 also includes: a differential file acquisition module, used to acquire the file to be differentially processed; a block module, used to block the file to obtain data blocks; a change record table creation module, used to create a first change record table, wherein each table entry in the first change record table corresponds to each data block, and is used to record the historical information of changes in each data block within a period of time.
[0085] According to an implementation mode of an embodiment of the present invention, the device 30 also includes: a first data fingerprint acquisition module, used to obtain the first data fingerprint of all data blocks, the first data fingerprint is the latest data fingerprint corresponding to each data block; a second data fingerprint acquisition module, used to obtain the second data fingerprint of all data blocks, the second data fingerprint is the last stored data fingerprint corresponding to each data block; a data fingerprint detection module, used to detect whether the first data fingerprint and the second data fingerprint of each data block are the same; a change record table update module, used to update the table entry corresponding to the corresponding data block in the first change record table.
[0086] According to an implementation manner of an embodiment of the present invention, the change record table update module includes: an entry acquisition submodule, used to acquire all entries corresponding to the corresponding data block; and an entry update submodule, used to update each of all the entries in sequence.
[0087] According to an implementation manner of an embodiment of the present invention, each entry in the first transformation record table is used to record historical information of changes in each data block within a period of time, including: each entry in the first transformation record table is used to record the number of times each data block changes within a period of time; accordingly, the change record table update module is specifically used to increase the number recorded by the entry by one.
[0088] According to an implementation of an embodiment of the present invention, the first condition detection module 301 is specifically used to detect whether conduction is performed for a predetermined time.
[0089] According to an implementation of an embodiment of the present invention, the device 30 further includes a wear factor and aggregation segment factor determination module, which is used to determine the wear factor and the aggregation segment factor.
[0090] According to a third aspect of an embodiment of the present invention, a storage medium is provided, on which program instructions are stored, wherein the program instructions are used to execute any of the above-mentioned data fingerprint detection methods when running.
[0091] It should be pointed out here that the above description of the data fingerprint detection device embodiment and the above description of the computer storage medium embodiment are similar to the description of the aforementioned method embodiment and have similar beneficial effects as the aforementioned method embodiment, so they will not be described in detail. For the technical details that have not been disclosed in the description of the data fingerprint detection device embodiment and the description of the computer storage medium embodiment of the present invention, please refer to the description of the aforementioned method embodiment of the present invention for understanding, and in order to save space, they will not be described in detail.
[0092] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.
[0093] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another device, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.
[0094] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0095] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.
[0096] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage medium, read-only memory (ROM), disk or optical disk, etc. Various media that can store program codes.
[0097] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods of each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage media, ROM, magnetic disks or optical disks.
[0098] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.
Claims
1. A data fingerprint detection method, the method comprising: Get the file to be differentially processed; Dividing the file into blocks to obtain data blocks; Creating a first change record table, wherein each entry of the first change record table corresponds to each data block; When the first condition is met, perform the following operations: Obtaining a first change record table of all data blocks, wherein each entry in the first change record table records historical information of changes of each data block within a period of time, and each data block corresponds to a data fingerprint; Performing wear processing on the first change record table according to a preset wear factor to obtain a second change record table; Detecting whether there is a mergeable table entry corresponding to a preset aggregation segment factor and satisfying an aggregation condition in the second change record table, and if so, merging at least two adjacent data blocks corresponding to the mergeable table entry into one data block; the aggregation condition includes requiring that the change history information recorded in the data block corresponding to the aggregation segment factor is lower than a certain value; After the first change record table is created, when performing differential processing on the file, the following operations are performed: Obtaining first data fingerprints of all data blocks, where the first data fingerprint is the latest data fingerprint corresponding to each data block; Obtaining a second data fingerprint of all data blocks, where the second data fingerprint is a last stored data fingerprint corresponding to each data block; Detect whether the first data fingerprint and the second data fingerprint of each data block are the same. If they are different, update the table entry corresponding to the corresponding data block in the first change record table.
2. According to the method of claim 1, the updating of the table entry corresponding to the corresponding data block in the first change record table comprises: Get all table entries corresponding to the corresponding data block; Each entry in all the tables is updated in turn.
3. According to the method according to any one of claims 1 or 2, each entry in the first transformation record table is used to record the historical information of each data block changed within a period of time, including: Each entry in the first transformation record table is used to record the number of times each data block changes within a period of time; Accordingly, the table entries corresponding to the update data blocks include: The number of times recorded in the entry is increased by one. The method according to claim 1 , wherein the first condition comprises reaching a predetermined time. 5 . The method according to claim 1 , wherein the first change record table comprises a change record table using a bitmap as a storage structure.
6. The method according to claim 1, before performing wear processing on the first change record table according to a preset wear factor to obtain a second change record table, the method further comprises: The wear factor and the aggregate segment factor are determined.
7. A data fingerprint detection device, the device comprising: A differential file acquisition module is used to acquire files to be differentially processed; A block module, used to block files to obtain data blocks; A change record table creation module, used to create a first change record table, wherein each table entry of the first change record table corresponds to each data block, and is used to record the historical information of changes of each data block within a period of time; A first condition detection module, used to detect whether the first condition is met; A change record table acquisition module, used to acquire the first change record table of all data blocks; A wear module, used for performing wear processing on the first change record table according to a preset wear factor to obtain a second change record table; an aggregation segment factor detection module, used to detect whether the second change record table has a mergeable table item corresponding to a preset aggregation segment factor and satisfying an aggregation condition; the aggregation condition includes requiring that the change history information recorded in the data block corresponding to the aggregation segment factor is lower than a certain value; An aggregation module, used to merge at least two adjacent data blocks corresponding to the mergeable table entries into one data block; A first data fingerprint acquisition module, used to acquire first data fingerprints of all data blocks, where the first data fingerprint is the latest data fingerprint corresponding to each data block; A second data fingerprint acquisition module, used to acquire second data fingerprints of all data blocks, where the second data fingerprint is the last stored data fingerprint corresponding to each data block; A data fingerprint detection module, used to detect whether the first data fingerprint and the second data fingerprint of each data block are the same; The change record table updating module is used to update the table entries corresponding to the corresponding data blocks in the first change record table.
8. A storage medium having program instructions stored thereon, wherein: The program instructions are used to execute the data fingerprint detection method according to any one of claims 1 to 6 when running.
