Data processing method and related device
By optimizing the type migration and access order of data blocks, the problems of fragmented arrangement and poor temporal locality during data recovery were solved, achieving efficient data recovery.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HARBIN INST OF TECH SHENZHEN GRADUATE SCHOOL
- Filing Date
- 2022-06-29
- Publication Date
- 2026-05-29
AI Technical Summary
In existing data compression methods, the fragmented arrangement of data blocks leads to low reading efficiency during data recovery, and the poor temporal locality between differential data blocks and basic data blocks results in a decline in the performance of the recovery process.
By determining the type of the target data block, similar data blocks are migrated to a dedicated set of containers, and the data blocks are accessed and restored in a specific order, ensuring that each target container set contains only the data blocks required for restoration, thus optimizing the acquisition order and storage location of the data blocks.
It improves data recovery efficiency, reduces disk access to read unnecessary data blocks, and enhances overall data recovery speed and throughput performance.
Smart Images
Figure CN115145763B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of storage technology, and in particular to data processing methods and related devices. Background Technology
[0002] Existing large-scale data compression systems often combine block-level deduplication and differential compression techniques to achieve better compression results. Both block-level deduplication and differential compression reduce the data size by saving duplicate data only once and referencing it multiple times.
[0003] Existing data compression methods typically involve storing the data blocks to be processed (including basic data blocks and differential data blocks) sequentially into containers. Once a container is full, the remaining data blocks are stored in a new container. When a container is full, the entire container is compressed, and the compressed container is stored, ultimately achieving data compression. To recover specific data, different containers need to be decompressed, and the required data blocks retrieved.
[0004] However, according to the existing data compression method, the arrangement of the required data blocks in each container is fragmented. That is, the required data blocks and the unnecessary data blocks in the container to be decompressed are arranged in an alternating manner. On this basis, decompressing the container and reading all the data blocks in the container in sequence will greatly reduce the data reading efficiency during data recovery. Summary of the Invention
[0005] This application provides a data processing method and related equipment to improve data recovery efficiency.
[0006] The first aspect of this application provides a data processing method, including:
[0007] Determine the target container set where each target data block used to recover the target data is located, wherein the target container set contains only the target data blocks of the target data, and the target blocks are differential data blocks or basic data blocks;
[0008] The access order of the multiple target container sets is determined, and the target data blocks stored in each target container set are obtained sequentially according to the access order; wherein the acquisition order of each differential data block is earlier than the acquisition order of the basic data blocks corresponding to the differential data block;
[0009] The target data is restored according to the acquisition order of each target data block.
[0010] In one specific implementation, each of the target container sets has a data identifier interval, which includes at least one data identifier. The data identifier interval includes an identifier that is used by each target data block in the target container set for data participating in the recovery.
[0011] The access order is as follows: first access the target container set with the smaller right-end value of the data identifier interval, and then access the target container set with the larger right-end value of the data identifier interval; for multiple target container sets with the same right-end value of the data identifier interval, first access the target container set with the larger left-end value of the data identifier interval, and then access the target container set with the smaller left-end value of the data identifier interval.
[0012] In one specific implementation, restoring the target data according to the acquisition order of each target data block includes:
[0013] If the target data block is a differential data block, then the target data block is written into the recovery space corresponding to the target data block;
[0014] If the target data block is a base data block and the recovery space contains a differential data block corresponding to the target data block, then the target data block and the differential data block are differentially decompressed, and the differential data block in the recovery space is updated to the differentially decompressed data block.
[0015] If the target data block is a base data block and the recovery space does not contain a differential data block corresponding to the target data block, then the target data block is written into the recovery space corresponding to the target data block.
[0016] In one specific implementation, restoring the target data according to the acquisition order of each target data block includes:
[0017] If the target data block is a differential data block, then the target data block is written to the cache;
[0018] If the target data block is a base data block and a differential data block corresponding to the target data block is cached, then the target data block and the differential data block are differentially decompressed, and the decompressed data block is written into the recovery space corresponding to the target data block.
[0019] If the target data block is a base data block and the cache does not contain the differential data block corresponding to the target data block, then the target data block is written to the recovery space corresponding to the target data block.
[0020] Once each target data block is written into the recovery space corresponding to the target data block, the target data blocks in each recovery space are combined into target data.
[0021] A second aspect of this application provides a data processing method, including:
[0022] The target data is divided into multiple target data blocks arranged in sequence according to the preset block division conditions;
[0023] The data block type of each target data block is determined based on the data blocks in at least one set of undetermined containers;
[0024] If the target data block is the first data block, then the similar data blocks that have been stored corresponding to the target data block are migrated from the undetermined container set to the first target container set, and the difference data blocks corresponding to the target data are stored in the second target container set, wherein the difference data blocks are determined according to the target data block and the similar data blocks;
[0025] Construct a correspondence between the target data and the corresponding container set, wherein the corresponding container set is the first target container set and the second target container set.
[0026] In some specific implementations, the method further includes:
[0027] If the target data block is the second data block, then the existing identical data blocks corresponding to the target data block will be migrated from the pending container set to the third target container set;
[0028] If the target data block is a remaining data block, then the target data block is stored in the second target container set, and the remaining data block is the target data block other than the first data block and the second data block among the plurality of target data blocks;
[0029] The corresponding container set is either the third target container set or the second target container set.
[0030] In some specific implementations, the method further includes:
[0031] The plurality of target data blocks are sequentially divided into at least one data segment according to preset segmentation conditions;
[0032] Determine the target undetermined container set containing the similar data blocks that have already been stored for each target data block in each data segment whose data block type is the first data block;
[0033] If the number of times the target pending container set is named within a data segment does not meet the preset naming threshold, then the data block type of each target data block corresponding to the target pending container set within the data segment will be modified to a remaining data block.
[0034] A third aspect of this application provides a data processing apparatus, comprising:
[0035] A determining unit is configured to determine the target container set in which each target data block for recovering target data is located, wherein the target container set contains only the target data blocks of the target data, and the target data blocks are differential data blocks or basic data blocks;
[0036] An acquisition unit is configured to determine the access order of multiple target container sets, and sequentially acquire the target data blocks stored in each target container set according to the access order; wherein the acquisition order of each differential data block is earlier than the acquisition order of the basic data blocks corresponding to the differential data block;
[0037] The recovery unit is used to recover the target data according to the acquisition order of each target data block.
[0038] In one specific implementation, each of the target container sets has a data identifier interval, which includes at least one data identifier. The data identifier interval includes an identifier that is used by each target data block in the target container set for data participating in the recovery.
[0039] The access order is as follows: first access the target container set with the smaller right-end value of the data identifier interval, and then access the target container set with the larger right-end value of the data identifier interval; for multiple target container sets with the same right-end value of the data identifier interval, first access the target container set with the larger left-end value of the data identifier interval, and then access the target container set with the smaller left-end value of the data identifier interval.
[0040] In one specific implementation, the recovery unit is specifically used to write the target data block into the recovery space corresponding to the target data block if the target data block is a differential data block;
[0041] If the target data block is a base data block and the recovery space contains a differential data block corresponding to the target data block, then the target data block and the differential data block are differentially decompressed, and the differential data block in the recovery space is updated to the differentially decompressed data block.
[0042] If the target data block is a base data block and the recovery space does not contain a differential data block corresponding to the target data block, then the target data block is written into the recovery space corresponding to the target data block.
[0043] In one specific implementation, the recovery unit is specifically used to write the target data block into the cache if the target data block is a differential data block;
[0044] If the target data block is a base data block and a differential data block corresponding to the target data block is cached, then the target data block and the differential data block are differentially decompressed, and the decompressed data block is written into the recovery space corresponding to the target data block.
[0045] If the target data block is a base data block and the cache does not contain the differential data block corresponding to the target data block, then the target data block is written to the recovery space corresponding to the target data block.
[0046] Once each target data block is written into the recovery space corresponding to the target data block, the target data blocks in each recovery space are combined into target data.
[0047] A fourth aspect of this application provides a data processing apparatus, comprising:
[0048] A partitioning unit is used to divide the target data into multiple target data blocks arranged in sequence according to preset partitioning conditions;
[0049] A determining unit is configured to determine the data block type of each target data block based on data blocks in at least one set of undetermined containers;
[0050] A storage unit is configured to, if the target data block is a first data block, migrate the already stored similar data blocks corresponding to the target data block from the undetermined container set to the first target container set, and store the difference data block corresponding to the target data block into the second target container set, wherein the difference data block is determined according to the target data block and the similar data blocks;
[0051] A construction unit is used to construct a correspondence between the target data and the corresponding container set, wherein the corresponding container set is the first target container set and the second target container set.
[0052] In some specific implementations, the storage unit is further configured to migrate the existing identical data blocks corresponding to the target data block from the undetermined container set to the third target container set if the target data block is the second data block.
[0053] If the target data block is a remaining data block, then the target data block is stored in the second target container set, and the remaining data block is the target data block other than the first data block and the second data block among the plurality of target data blocks;
[0054] The corresponding container set is either the third target container set or the second target container set.
[0055] In some specific implementations, the device further includes:
[0056] The partitioning unit is also used to divide the plurality of target data blocks into at least one data segment according to preset segmentation conditions.
[0057] The determining unit is further configured to determine the target undetermined container set where the similar data blocks that have been stored are located for each target data block whose data block type is the first data block in each data segment;
[0058] The determining unit is further configured to modify the data block type of each target data block corresponding to the target undetermined container set in the data segment to a remaining data block if the number of times the target undetermined container set is named in a data segment does not meet the preset naming threshold.
[0059] A fifth aspect of this application provides a data processing apparatus, comprising:
[0060] Central processing unit, memory, and input / output interfaces;
[0061] The memory is either a short-term storage memory or a persistent storage memory;
[0062] The central processing unit is configured to communicate with the memory and execute instructions in the memory to perform the method described in the first or second aspect.
[0063] A sixth aspect of this application provides a computer program product containing instructions that, when run on a computer, cause the computer to perform the method described in the first or second aspect.
[0064] A seventh aspect of this application provides a computer storage medium storing instructions that, when executed on a computer, cause the computer to perform the method described in the first or second aspect.
[0065] As can be seen from the above technical solutions, the embodiments of this application have the following advantages: the target container set storing each target data block of the target data to be recovered only contains the target data block required to recover the target data, so that when recovering the target data, there is no need to access data blocks that are not used to recover the target data, thus improving the data recovery efficiency. Attached Figure Description
[0066] Figure 1 This is a schematic flowchart of a data processing method disclosed in an embodiment of this application;
[0067] Figure 2 This is an example diagram of a data block type modification method disclosed in an embodiment of this application;
[0068] Figure 3 This is a schematic flowchart of a data compression method disclosed in an embodiment of this application;
[0069] Figure 4 This is a compression effect diagram of the data compression method disclosed in the embodiments of this application;
[0070] Figure 5 This is another schematic flowchart of the data processing method disclosed in the embodiments of this application;
[0071] Figure 6 This is an example diagram of a data recovery method disclosed in an embodiment of this application;
[0072] Figure 7 This is a schematic diagram of the data processing apparatus disclosed in an embodiment of this application;
[0073] Figure 8 This is another schematic diagram of the data processing apparatus disclosed in the embodiments of this application;
[0074] Figure 9 This is another schematic diagram of the data processing apparatus disclosed in the embodiments of this application. Detailed Implementation
[0075] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0076] Currently, compression methods for large-scale data mainly achieve this through block-level deduplication. Block-level deduplication involves dividing the data into blocks, and then, for multiple identical data blocks, retaining only one identical block and replacing the other identical data blocks with an index, thereby compressing the data and reducing its size. Furthermore, besides identical data blocks, there are also data blocks that are not completely identical but similar. For two data blocks with similar parts, differential compression can be introduced for further compression, ultimately achieving fine-grained (compared to block-level deduplication) elimination of redundant data. Differential compression refers to the process of calculating the differences between similar data blocks (block A and block B) using one block A as the base data block (the base data block can also be block B, but we'll use block A as the base data block here). These differences are then used as replacements for block B, thus saving storage overhead. The resulting ordinary and differential data blocks are placed into containers in the order they appear. The container capacity can be a fixed value; once a container is full, it is compressed as a whole. Subsequent data blocks are then placed into the next container. Therefore, the data after block-level deduplication and differential compression contains two types of blocks: ordinary blocks and differential blocks. The differential block is the product of differential compression, replacing a regular block before differential compression; while a regular block may serve as the base block for a differential block.
[0077] As data compression becomes increasingly sophisticated, it also brings certain challenges to data compression and recovery. The current data compression process mainly faces the following three problems:
[0078] First, when performing block-level deduplication and differential compression on data, the locality of the basic data blocks is poor.
[0079] Specifically, differential compression requires both a base data block and a data block to be compressed. In a pipelined program, data blocks are processed in the order they appear. When processing a data block A, if it is found to be not a completely duplicated data block, but has a similar data block B (which must have been previously processed and written to disk), then similar data block B needs to be read first as the base data block for differential compression of data block A. To achieve a higher compression ratio, processed data blocks are stored in a container, and compression is performed on the entire container. Therefore, when retrieving similar data block B, the entire container storing similar data block B needs to be read and decompressed, resulting in significant read amplification (the container storing similar data block B only contains similar data block B, which is required for differential compression of data block A, but other data blocks in the container besides similar data block B are still read), leading to poor overall throughput. In fact, the significant read amplification caused when retrieving similar block B is mainly due to the randomness of the location of the base data block (within any container). Therefore, even without using a container design, reading fragmented data piecemeal will still result in low read efficiency.
[0080] Secondly, when restoring data after differential compression, the spatial locality of the required data blocks is poor.
[0081] Specifically, both block-level deduplication and differential compression reduce data size by saving duplicate data blocks only once and then referencing them multiple times. However, this referencing also disrupts the spatial locality of data. For example, consider two datasets, D1 and D2, where D1 includes sequentially arranged data blocks C1, C2, C3, and C4; and D2 includes sequentially arranged data blocks C5, C3, C6, and C1' (where C1' is a similar block to C1). Block-level deduplication and differential compression are performed on both datasets D1 and D2, and the resulting data is placed in a container S. Container S contains sequentially arranged data blocks C1, C2, C3, C4, C5, C6, and Δ(C1'), where data block Δ(C1') represents the difference between data block C1' (calculated using data block C1 as the base) and data block C1. If data D2 needs to be recovered from container S, data blocks C1, C3, C5, C6, and Δ(C1') need to be read sequentially from container S. However, the arrangement of these data blocks in container S is fragmented (data blocks needed to recover D2 and those not needed are interleaved). This scattered distribution of data blocks leads to significant read amplification, resulting in poor overall throughput. Even without using a design scheme to bypass containers, reading these fragmented data individually still results in low read efficiency.
[0082] Third, when restoring data after differential compression, the temporal locality between the differential data block (the product of differential compression) and the corresponding basic data block is poor, that is, the data block reading order is disordered.
[0083] Specifically, based on the example in point two, it can be seen that not only are the required data blocks fragmented in location (poor spatial locality, the second problem with existing solutions), but there is also a contradiction between the order in which the required data blocks are used and the order in which they are stored (i.e., poor temporal locality). Specifically, assuming the second problem is solved, and C1, C3, C5, C6, and Δ(C1') required to recover D2 are stored tightly and consecutively together, we will find that C1 and Δ(C1') must be used simultaneously for differential decompression to obtain data block C1' (the reverse process of differential compression). However, these two blocks are in very different order (the first and last data blocks in container S, respectively). This means that when reading sequentially, if data block C1 is read first, it cannot be decoded; other data blocks (C2, C3, C4, C5, and C6) must be skipped, and data block Δ(C1') must be read first. This leads to discontinuous access to data, causing performance degradation. On the other hand, storing data block C6 and data block Δ(C1') in the same data container will cause the data container to be accessed again when data block C6 is needed later, resulting in unnecessary multiple accesses to the same data container and causing performance degradation.
[0084] To address the aforementioned problems in data compression and data recovery processes in existing technologies, embodiments of this application provide a data processing method and related equipment to improve data recovery efficiency.
[0085] Please see Figure 1 This application provides a data processing method for deduplicating and compressing large-scale data, including the following steps:
[0086] 101. Divide the target data into multiple target data blocks arranged in sequence according to the preset block division conditions.
[0087] When deduplication and compression of target data are required, the target data should first be divided into multiple target data blocks according to pre-set block division conditions. Target data refers to any data that needs to be deduplicated and compressed; it can be newly added data or backup data, but this embodiment does not limit the scope.
[0088] Specifically, if the preset segmentation condition is to divide the target data into data blocks of a specified size sequentially, then the target data is segmented from beginning to end, starting from the beginning of the target data. Each time the specified size is met, a data block is created. It is understood that the preset segmentation condition can also be to split the target data according to the character type of the target data content, resulting in multiple target data blocks; this application does not limit this.
[0089] 102. Determine the data block type of each target data block based on the data blocks in at least one set of undetermined containers.
[0090] To process each target data block obtained in step 101 based on its data block type, it is first necessary to classify each target data block and apply different data processing methods to different types of target data blocks. The data block type includes a first data block, which is defined as a target data block if a similar data block exists in the existing data blocks of the pending container set.
[0091] Specifically, a similar data block refers to a data block that is not completely identical to the target data block but has similar parts that has already been stored in any pending container set. In order to avoid the waste of storage space due to the repeated storage of similar parts, step 103 is executed for the target data block (i.e., the data block type is the first data block) that has already been stored in at least one pending container set.
[0092] For determining similar data blocks, some implementations compare the data content of the target data block with at least one data block already stored in the pending container set. If a similar data block exists in the data blocks already stored in the pending container set, the target data block is determined as the first data block. Alternatively, a fingerprint mechanism and index are introduced. The fingerprint of each data block in the container set is stored in the index. When determining whether a similar data block exists for the target data block, since each data block in the container set stores a corresponding local sensitive hash value in the index, the local sensitive hash value of the target data block can be compared with each local sensitive hash value in the index, and the unique data block with the same local sensitive hash value is determined as the similar data block of the target data block. In practical applications, a target data block may have multiple locality-sensitive hash values. In this case, the data block with the most similar locality-sensitive hash values to the target data block is identified as the similar data block. If multiple data blocks in the pending container set have the most similar locality-sensitive hash values to the target data block, then any data block with the most similar locality-sensitive hash values to the target data block is selected as the similar data block.
[0093] It is understood that at least one pending container set refers to each container set to which the data blocks obtained from the previous deduplication compression data corresponding to the current deduplication compression target data (i.e., the target data divided in step 101) are stored. If the current deduplication compression target data (i.e., the target data divided in step 101) is the first data to be deduplicated, then there are no already stored data blocks and no pending container set, meaning deduplication compression cannot be performed. The first data to be deduplicated refers to the first data to be compressed in the specified storage space or in all storage spaces of the deduplication compression system; that is, there are no already stored data blocks in the specified storage space or in all storage spaces of the deduplication compression system. This embodiment does not specifically limit the storage space corresponding to the first deduplication compression.
[0094] 103. If the target data block is the first data block, then the similar data blocks that have been stored corresponding to the target data block are migrated from the pending container set to the first target container set, and the differential data blocks corresponding to the target data are stored in the second target container set, wherein the differential data blocks are determined according to the target data block and the similar data blocks.
[0095] When it is determined that similar data blocks exist for the target data block, existing technologies typically use the corresponding similar data blocks as base data blocks to perform differential compression on the target data block, retaining only the differential data blocks corresponding to the target data block (i.e., the differential data blocks calculated between the target data block and the corresponding similar data blocks based on the similar data blocks) and the reference relationships between the differential data blocks and the similar data blocks. However, while this method achieves the effect of differential compression, since data reading is performed on a set of data block containers, when recovering (i.e., reading) the target data, it is necessary to simultaneously read the data block container sets storing non-first data blocks (i.e., target data blocks whose data block type is not the first data block) and the data block container sets storing the first data block (i.e., target data blocks whose data block type is the first data block). The data block container sets storing the first data block usually also store data that is not present in the target data (i.e., data blocks that are not needed for recovering the target data). In this case, a large amount of data that is not needed by the target data needs to be read during the recovery process, causing read amplification problems and preventing the full throughput of the system from being utilized for target data recovery, ultimately resulting in slow target data recovery speed.
[0096] To address the aforementioned issues, this embodiment, upon determining the existence of a first data block (i.e., a target data block of type first data block), migrates each first data block (i.e., a target data block of type first data block) from its original storage in the pending container set to the first target container set, and stores the differential data block corresponding to each first data block (i.e., a target data block of type first data block) in the second target container set. By migrating the similar data blocks corresponding to the first data blocks, the lifecycle of the similar data blocks is updated. That is, the lifecycle of the similar data blocks is different from that of the non-target data blocks originally stored together. Therefore, the similar data blocks are no longer stored together with the non-target data blocks originally stored together, but are stored independently. The lifecycle is defined as follows: a storage system including block-level deduplication and differential compression stores data F1, F2, F3... etc., if there is one and only data F... n F m and F k The partitioning result contains the same data block C. k And / or contains the need to include C k As the base data block, then data block C k The lifecycle is {F n F m F k} (i.e., recover data F) n F m and F k Data block C is required. k). Here, the container set mentioned above and the container set mentioned below are collective terms for a collection of containers with the same lifecycle, and a container set includes at least one container.
[0097] Based on the above definition of the data block lifecycle and the method described in this embodiment, it can be ensured that each target data block of the target data is stored in a data block container set with the same lifecycle, and that the data blocks stored in each data block container set are data blocks with the same lifecycle. This ensures that when restoring the target data, the containers in the container set read only store the data blocks required for restoring the target data (including but not limited to the target data block, similar data blocks corresponding to the target data block, and differential data blocks corresponding to the target data block).
[0098] 104. Construct the correspondence between the target data and the corresponding container set, which is the first target container set and the second target container set.
[0099] In constructing the correspondence between target data and corresponding container sets, if the data block type of the target data block is the first data block, then the corresponding container set for each target data block is the first target container set storing the similar data blocks corresponding to the target data block and the second target container set storing the differential data blocks corresponding to the target data block.
[0100] In this embodiment, the target container set storing each target data block of the target data to be recovered only contains the target data blocks required to recover the target data. This way, when recovering the target data, there is no need to access data blocks that are not used to recover the target data, thus improving the data recovery efficiency.
[0101] Furthermore, if the target data block is the second data block, then the existing stored identical data blocks corresponding to the target data block are migrated from the pending container set to the third target container set; if the target data block is the remaining data block, then the target data block is stored in the second target container set, and the remaining data block is the target data block other than the first data block and the second data block among multiple target data blocks; the corresponding container set is either the third target container set or the second target container set.
[0102] Specifically, if a data block identical to the target data block exists in the pending container set, then the target data block is determined to be the second data block, and the identical data block is the same data block corresponding to the target data block. If no data block identical to or similar to the target data block exists in the pending container set, then the target data block is determined to be a remaining data block. If the target data block is the second data block, deduplication compression is performed on the target data block; if the target data block is a remaining data block, normal compression is performed on the target data block. In this embodiment, deduplication compression refers to migrating the same data block corresponding to the target data block from the original pending container set to the third target container set, and normal compression refers to storing the target data block of type remaining data block in the second target container set.
[0103] It is understood that, based on the data processing method for the second data block and the remaining data blocks in this embodiment, the data compression method provided in this application embodiment can store data blocks with the same life cycle (specific definition can be referred to the foregoing embodiment) in the storage space (or storage system) in the same data block container set, ensuring that the data blocks in the same data block container set are all target data blocks used only to recover the same data, thus avoiding a large amount of read amplification.
[0104] Furthermore, considering that during data processing, if there is more than one target data block of type 1 data block among the multiple target data blocks obtained in step 101, it is necessary to read the similar data blocks corresponding to each data block. These similar data blocks are usually not stored in the same pending container set, leading to low efficiency in differential compression. After step 102, i.e., after determining the data type of each target data block, in this embodiment, several target data blocks can also be considered as a data segment (the specific segmentation depends on the preset segmentation conditions). Within the same data segment, for target data blocks of type 1 data block, based on the target pending container set where the similar data blocks (i.e., the corresponding basic data blocks) corresponding to each target data block of type 1 data block are located, the number of times each target pending container set is designated (i.e., how many target data blocks require differential compression from the data blocks in that target pending container set) is counted. The target pending container set will only be read if the number of times it is named meets a preset naming threshold. The corresponding similar data blocks within this set will then be used as base data blocks for differential compression of the corresponding target data block. For other data blocks within this data segment that can be differentially compressed (i.e., target data blocks whose corresponding similar data blocks have not been read and whose data block type is the first data block), their data block type will be changed to the remaining data block (i.e., no differential compression will be performed). Figure 2Taking the data segment shown as an example, the data segment includes four data blocks of type 1 data block, namely data blocks 1, 2, 3, and 4. The correspondence between data blocks 1, 2, 3, and 4 and their corresponding similar data blocks (i.e., data blocks i, j, and q) is as follows: Figure 2 The dashed arrows shown are for reference. Data blocks i and j are located in container n, and data block q is located in container m. Therefore, container n is assigned three times, and container m is assigned once. If the preset assignment threshold is greater than 2, the data block type of data block 4 will be changed from the first data block to the remaining data block.
[0105] The preset segmentation condition can be a preset data segment size (such as 12 MB, meaning that the data size of all data blocks contained in a data segment is as close as possible to and less than or equal to 12 MB), or it can be a specified number of data blocks (such as eight data blocks), which is not limited here.
[0106] In practical applications, the data block type for each target data block can be determined by referring to... Figure 3 The process is shown below.
[0107] Specifically, for target data divided into multiple blocks, firstly, a strong hash value (i.e., a hash value that can identify the uniqueness of a data block, as shown in the figure FP1, FP2, FP3, FP4, FP5, FP6, ...) of each target data block is calculated. This is used to determine from the fingerprint index (an index containing the strong hash values of each data block that have been stored) whether there is an identical data block in the pending container that is completely the same as any target data block. If so, the data type of the corresponding target data block is determined as the second data block.
[0108] Next, the data block types of the target data blocks other than the second data block are determined. Local sensitive hash values (which can be used to determine the similarity between two data blocks, as shown in the figure: SF1, SF2, SF3, SF4, SF5, SF6, ...) of the target data blocks other than the second data block are calculated. These hash values are used to determine whether there are similar data blocks in the pending container that are similar to any target data block from the similarity fingerprint index (an index containing the locally sensitive hash values of each data block that have already been stored). If so, the data type of the corresponding target data block is determined as the first data block.
[0109] Then, the target pending container for storing similar data blocks corresponding to each first data block is determined, and the naming count of each target pending container is determined. The data block type of the target data block whose naming count of the target pending container for the corresponding similar data block does not meet the preset naming threshold is changed from the first data block to the remaining data block (i.e., data block type update).
[0110] Finally, the data block type of the data block whose data block type was not determined among the multiple data blocks was determined as the remaining data block.
[0111] In this embodiment, after further processing, the data block type of the target data block in the target pending container set that does not meet the preset naming threshold is changed from the first data type to the remaining data type, which reduces disk access and effectively improves the throughput of the differential compression process.
[0112] According to the data compression process described in the foregoing embodiments, the following can be achieved: Figure 4 In the figure, the classification effect of the data processing scheme of this application is shown. Cat.(x, y) represents the data interval identifier of different target datasets (i.e., different categories, also referring to the set of data blocks with the same lifecycle). Figure 4 It is evident that, according to the data processing method of this application embodiment, data blocks with the same lifecycle can be placed in a target container set of the same type. For example, the lifecycle of the data blocks in Cat.(1,3) is the first data, the second data, and the third data. When n data have already been stored in the storage space, and we need to recover the kth data, the required data blocks are contained in exactly a few Cats (i.e., the target dataset), and the required target container set can be expressed by the formula: For example, recovering the second set of data requires, and only requires, all the data blocks in Cat.(1,2), Cat.(2,2), Cat.(1,3), and Cat.(2,3). This way, the required data blocks are tightly packed into several target container sets, alleviating the problem of scattered data blocks needed for data recovery.
[0113] Based on the data processing method for data compression described above, please refer to the following section. Figure 5 Another data processing method of this application, applied to data recovery, includes the following steps:
[0114] 501. Determine the target container set where each target data block used to recover the target data is located, wherein the target container set contains only the target data blocks of the target data, and the target data blocks are differential data blocks or base data blocks;
[0115] After compressing the target data using the aforementioned data processing method applied to data compression, efficient data recovery can be achieved through the data processing method applied to data recovery in the embodiments of this application.
[0116] First, to recover the target data, it is necessary to determine the target data blocks required for recovery and the target container set containing each target data block. The target container set obtained by the aforementioned data processing method applied to data compression contains only the target data blocks required for recovery, and target data blocks with the same lifecycle (the lifecycle definition is similar to that in the aforementioned embodiments and will not be repeated here) are stored in the same target dataset.
[0117] In some specific implementations, if the storage information and reference information of the data block corresponding to each data are stored in the index, then each target container set can be directly determined from the index.
[0118] 502. Determine the access order of multiple target container sets, and retrieve the target data blocks stored in each target container set in the order of access; wherein the retrieval order of each differential data block is earlier than the retrieval order of the corresponding basic data blocks;
[0119] After determining the multiple target container sets required to restore the target data in step 501, the target data blocks are read from each target container set in the order of access. The order of access must be selected to ensure that the differential data block in the target data block is obtained first, and then the basic data block corresponding to the differential data block is obtained.
[0120] 503. Restore the target data according to the acquisition order of each target data block.
[0121] Access the target data blocks in each target container set in the access order, and restore the target data according to the acquisition order of each target data block.
[0122] In this embodiment, the acquisition order of each differential data block is guaranteed to be earlier than the acquisition order of the corresponding basic data block. This solves the problem in the prior art where the basic data block is acquired first but cannot be used, and the basic data block needs to be acquired again when the corresponding differential data block is acquired. This greatly improves data recovery efficiency. In addition, the target container set only contains the target data blocks required to recover the target data, which also greatly reduces disk access and read amplification issues.
[0123] Furthermore, each target container set has a data identifier range, which includes at least one data identifier. The data identifier range includes identifiers that are used by each target data block in the target container set for data participating in the recovery. The access order is as follows: first access the target container set with the smaller right-hand value of the data identifier range, and then access the target container set with the larger right-hand value of the data identifier range. For multiple target container sets with the same right-hand value of the data identifier range, first access the target container set with the larger left-hand value of the data identifier range, and then access the target container set with the smaller left-hand value of the data identifier range.
[0124] Specifically, when establishing each target container set, a corresponding data identifier range is configured for that target container set (used to identify the data corresponding to the data identifier indicated by the data identifier range for each target data block in the target container set). This is because each data identifier range contains a left-hand value and a right-hand value, where the left-hand value of each data identifier range is less than or equal to the right-hand value. Furthermore, for at least one target container in the target container set, access is performed in reverse order (here, reverse refers to the order in which data blocks are stored, i.e., the data blocks in the target container whose last data block was stored are retrieved first). Within a single target container, reading is performed in reverse order (here, reverse refers to the order in which data blocks are stored, i.e., the last data block stored in the target container is retrieved first).
[0125] In some specific implementations, if it is necessary to restore the target data processed according to the aforementioned data compression process (the arrangement of data blocks in each target container after compression can be found in [reference]), Figure 4 The data processing scheme for this application can determine the access order in the following ways.
[0126] Specifically, when recovery is needed Figure 4 When referring to the second data, firstly, for the multiple target container sets required for data recovery (such as...) Figure 4 The data is arranged in rows and columns. First, the first column is accessed (the target container set with the smaller y value in the corresponding data interval identifier Cat.(x,y) of the multiple target container sets). Next, in the multiple target container sets in the same column, the larger row number is accessed first (the x value in Cat.(x,y) is larger). Finally, within the same target container set, access is performed from back to front. For example, to recover the second data, only the four target container sets Cat.(1,2), Cat.(2,2), Cat.(1,3), and Cat.(2,3) are needed. The access order is: first access Cat.(2,2), then access Cat.(1,2), then access Cat.(2,3), and finally access Cat.(1,3).
[0127] Furthermore, the problem of disordered data block order can be solved by writing the pre-acquired differential blocks into the storage space through the differential write-ahead mechanism. The storage space can be a load or a cache, which is not limited here.
[0128] Specifically, data recovery can be viewed as the process of reassembling individual data blocks as parts to reconstruct a complete data load. The recovered data load W r Often it's a single file, and the data recovery process involves transferring it to W. rThis file is continuously populated with data. The steps of the differential block write-ahead mechanism are as follows: Because of the support for access order for different target container sets, it can be guaranteed that during data recovery, when differential block D is accessed... n At that time, its basic block B n It has not been accessed yet. Therefore, when the differential block D is accessed... n D n Pre-write to W r In the middle, the location to be written is C n In W r The position that should be in the middle (C) n It is D n (The block obtained after differential decompression). When subsequently accessing D... n Basic block B n At that time, from W r D n Read back D from the pre-write position n and using D n and B n Differential decompression yields C n Ultimately, C n Write to W r The middle should be in the position that covers the previous D. n The location for pre-writing. Backup space can be located on the hard drive, while user space is the load to be restored (W). r The storage space can be a solid-state drive (SSD). In practical applications, if recovery is required... Figure 4 The third data in the sequence, where the data block type is the first data block, the recovery steps for data block F' can be found in the following reference. Figure 6 The process is shown below.
[0129] It is understood that in this embodiment, D may not be included. n Pre-write to W r Instead, it is stored in a cache in memory, and at the same time, when accessing B... n D is retrieved from the cache at that time. n Differential decompression is performed in this embodiment, but D is not included. n The location for writing ahead or caching is specifically defined.
[0130] In this embodiment, the problem of the differential data block and its corresponding basic data block being unable to be accessed continuously is further solved, and the problems of repeated access and skipped access during the data recovery process are resolved.
[0131] Based on the aforementioned data processing method, the data processing apparatus provided in the embodiments of this application will be described below.
[0132] Please see Figure 7 This application provides a data processing apparatus, including:
[0133] The determining unit 701 is used to determine the target container set where each target data block for recovering target data is located, wherein the target container set contains only the target data blocks of the target data, and the target data blocks are differential data blocks or basic data blocks;
[0134] The acquisition unit 702 is used to determine the access order of the multiple target container sets, and to acquire the target data blocks stored in each target container set in the order of access; wherein the acquisition order of each differential data block is earlier than the acquisition order of the basic data blocks corresponding to the differential data block;
[0135] The recovery unit 703 is used to recover the target data according to the acquisition order of each target data block.
[0136] In one specific implementation, each of the target container sets has a data identifier interval, which includes at least one data identifier. The data identifier interval includes an identifier that is used by each target data block in the target container set for data participating in the recovery.
[0137] The access order is as follows: first access the target container set with the smaller right-end value of the data identifier interval, and then access the target container set with the larger right-end value of the data identifier interval; for multiple target container sets with the same right-end value of the data identifier interval, first access the target container set with the larger left-end value of the data identifier interval, and then access the target container set with the smaller left-end value of the data identifier interval.
[0138] In one specific implementation, the recovery unit 703 is specifically used to write the target data block into the recovery space corresponding to the target data block if the target data block is a differential data block;
[0139] If the target data block is a base data block and the recovery space contains a differential data block corresponding to the target data block, then the target data block and the differential data block are differentially decompressed, and the differential data block in the recovery space is updated to the differentially decompressed data block.
[0140] If the target data block is a base data block and the recovery space does not contain a differential data block corresponding to the target data block, then the target data block is written into the recovery space corresponding to the target data block.
[0141] In one specific implementation, the recovery unit 703 is specifically used to write the target data block into the cache if the target data block is a differential data block;
[0142] If the target data block is a base data block and a differential data block corresponding to the target data block is cached, then the target data block and the differential data block are differentially decompressed, and the decompressed data block is written into the recovery space corresponding to the target data block.
[0143] If the target data block is a base data block and the cache does not contain the differential data block corresponding to the target data block, then the target data block is written to the recovery space corresponding to the target data block.
[0144] Once each target data block is written into the recovery space corresponding to the target data block, the target data blocks in each recovery space are combined into target data.
[0145] Please see Figure 8 This application provides a data processing apparatus, including:
[0146] The partitioning unit 801 is used to divide the target data into multiple target data blocks arranged in sequence according to preset partitioning conditions;
[0147] The determining unit 802 is configured to determine the data block type of each target data block based on data blocks in at least one set of undetermined containers;
[0148] Storage unit 803 is configured to, if the target data block is a first data block, migrate the already stored similar data blocks corresponding to the target data block from the undetermined container set to the first target container set, and store the difference data block corresponding to the target data block into the second target container set, wherein the difference data block is determined according to the target data block and the similar data blocks;
[0149] The construction unit 804 is used to construct the correspondence between the target data and the corresponding container set, wherein the corresponding container set is the first target container set and the second target container set.
[0150] In some specific implementations, the storage unit 803 is further configured to migrate the existing identical data block corresponding to the target data block from the undetermined container set to the third target container set if the target data block is the second data block.
[0151] If the target data block is a remaining data block, then the target data block is stored in the second target container set, and the remaining data block is the target data block other than the first data block and the second data block among the plurality of target data blocks;
[0152] The corresponding container set is either the third target container set or the second target container set.
[0153] In some specific implementations, the device further includes:
[0154] The partitioning unit 801 is also used to divide the plurality of target data blocks into at least one data segment according to preset segmentation conditions.
[0155] The determining unit 802 is further configured to determine the target undetermined container set where the similar data blocks that have been stored are located for each target data block whose data block type is the first data block in each data segment;
[0156] The determining unit 802 is further configured to modify the data block type of each target data block corresponding to the target undetermined container set in the data segment to a remaining data block if the number of times the target undetermined container set is named in a data segment does not meet the preset naming threshold.
[0157] Figure 9 This is a schematic diagram of a data processing device structure provided in an embodiment of this application. The data processing device 900 may include one or more central processing units (CPUs) 901 and a memory 905, in which one or more application programs or data are stored.
[0158] The memory 905 can be volatile or persistent storage. The program stored in the memory 905 can include one or more modules, each module including a series of instruction operations on the data processing device. Furthermore, the central processing unit 901 can be configured to communicate with the memory 905 and execute the series of instruction operations in the memory 905 on the data processing device 900.
[0159] The data processing device 900 may also include one or more power supplies 902, one or more wired or wireless network interfaces 903, one or more input / output interfaces 904, and / or one or more operating systems, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.
[0160] The central processing unit 901 can perform the aforementioned... Figures 1 to 8 The specific operations performed by the data processing device in the illustrated embodiment will not be described in detail here.
[0161] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.
[0162] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.
[0163] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0164] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0165] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0166] This application also provides a computer program product containing instructions that, when run on a computer, cause the computer to perform the data processing method described above.
Claims
1. A data processing method, characterized in that, include: The target data is divided into multiple target data blocks arranged in sequence according to the preset block division conditions; The data block type of each target data block is determined based on the data blocks in at least one set of undetermined containers; If the target data block is the first data block, then the similar data blocks that have been stored corresponding to the target data block are migrated from the pending container set to the first target container set, and the difference data block corresponding to the target data is stored in the second target container set, wherein the difference data block is determined according to the target data block and the similar data block, and the first data block is a data block that includes the similar data block of the first data block in the data blocks already stored in the pending container set. If the target data block is the second data block, then the existing identical data blocks corresponding to the target data block will be migrated from the set of containers to the third target container set; wherein, the second data block is a data block in the set of containers to the target that contains the same data block as the second data block; If the target data block is a remaining data block, then the target data block is stored in the second target container set, and the remaining data block is the target data block other than the first data block and the second data block among the plurality of target data blocks; A correspondence is established between the target data and the corresponding container set, wherein the corresponding container set is at least one of the first target container set, the second target container set, and the third target container set; wherein each of the corresponding container sets has a data identifier range, the data identifier range includes at least one data identifier, and the data identifier included in the data identifier range is an identifier that each target data block in the corresponding container set is used for and only used for data participating in the recovery. The access order of the corresponding container sets is as follows: first access the corresponding container set with the smaller right-end value of the data identifier interval, and then access the corresponding container set with the larger right-end value of the data identifier interval; for multiple corresponding container sets with the same right-end value of the data identifier interval, first access the corresponding container set with the larger left-end value of the data identifier interval, and then access the corresponding container set with the smaller left-end value of the data identifier interval.
2. The method according to claim 1, characterized in that, The method further includes: The plurality of target data blocks are sequentially divided into at least one data segment according to preset segmentation conditions; Determine the target undetermined container set containing the similar data blocks that have already been stored for each target data block in each data segment whose data block type is the first data block; If the number of times the target pending container set is named within a data segment does not meet the preset naming threshold, then the data block type of each target data block corresponding to the target pending container set within the data segment will be modified to a remaining data block.
3. A data processing method, characterized in that, include: A target container set is determined for each target data block used to recover target data, wherein the target container set contains only the target data blocks of the target data, and the target data blocks are differential data blocks or basic data blocks; the target container set is the first target container set, the second target container set, or the third target container set obtained according to claim 1 or 2; each target container set has a data identifier range, the data identifier range includes at least one data identifier, and the data identifier included in the data identifier range is an identifier used by each target data block in the target container set for and only for the data participating in the recovery; Determine the access order of the multiple target container sets, and sequentially obtain the target data blocks stored in each target container set according to the access order; The acquisition order of each differential data block is earlier than the acquisition order of the corresponding basic data block; the access order is as follows: first access the target container set with the smaller right-end value of the data identifier interval, and then access the target container set with the larger right-end value of the data identifier interval; for multiple target container sets with the same right-end value of the data identifier interval, first access the target container set with the larger left-end value of the data identifier interval, and then access the target container set with the smaller left-end value of the data identifier interval. The target data is restored according to the acquisition order of each target data block.
4. The method according to claim 3, characterized in that, The step of restoring the target data according to the acquisition order of each target data block includes: If the target data block is a differential data block, then the target data block is written into the recovery space corresponding to the target data block; If the target data block is a base data block and the recovery space contains a differential data block corresponding to the target data block, then the target data block and the differential data block are differentially decompressed, and the differential data block in the recovery space is updated to the differentially decompressed data block. If the target data block is a base data block and the recovery space does not contain a differential data block corresponding to the target data block, then the target data block is written into the recovery space corresponding to the target data block.
5. The method according to claim 3, characterized in that, The step of restoring the target data according to the acquisition order of each target data block includes: If the target data block is a differential data block, then the target data block is written to the cache; If the target data block is a base data block and a differential data block corresponding to the target data block is cached, then the target data block and the differential data block are differentially decompressed, and the decompressed data block is written into the recovery space corresponding to the target data block. If the target data block is a base data block and the cache does not contain the differential data block corresponding to the target data block, then the target data block is written to the recovery space corresponding to the target data block. Once each target data block is written into the recovery space corresponding to the target data block, the target data blocks in each recovery space are combined into target data.
6. A data processing apparatus, characterized in that, include: A partitioning unit is used to divide the target data into multiple target data blocks arranged in sequence according to preset partitioning conditions; A determining unit is configured to determine the data block type of each target data block based on data blocks in at least one set of undetermined containers; A storage unit is configured to, if the target data block is a first data block, migrate the already stored similar data blocks corresponding to the target data block from the undetermined container set to the first target container set, and store the difference data block corresponding to the target data block into a second target container set, wherein the difference data block is determined based on the target data block and the similar data blocks; the first data block is a data block that includes similar data blocks of the first data block among the data blocks already stored in the undetermined container set. If the target data block is the second data block, then the existing identical data blocks corresponding to the target data block are migrated from the pending container set to the third target container set; wherein, the second data block is a data block in the pending container set that contains the same data block as the second data block; if the target data block is a remaining data block, then the target data block is stored in the second target container set, wherein the remaining data block is a target data block other than the first data block and the second data block among the plurality of target data blocks; A construction unit is used to construct a correspondence between the target data and the corresponding container set, wherein the corresponding container set is at least one of the first target container set, the second target container set, and the third target container set; A determining unit is configured to determine the target container set containing each target data block used to recover target data, wherein the target container set contains only the target data blocks of the target data, and the target data blocks are differential data blocks or basic data blocks; the target container set is a first target container set, a second target container set, or a third target container set; each target container set has a data identifier range, the data identifier range including at least one data identifier, and the data identifiers included in the data identifier range are identifiers used by each target data block in the target container set for and only for data participating in the recovery; An acquisition unit is configured to determine the access order of multiple target container sets, and sequentially acquire the target data stored in each target container set according to the access order; wherein the acquisition order of each differential data block is earlier than the acquisition order of the corresponding basic data block; the access order is as follows: first access the target container set with the smaller right-end value of the data identifier interval, then access the target container set with the larger right-end value of the data identifier interval; for multiple target container sets with the same right-end value of the data identifier interval, first access the target container set with the larger left-end value of the data identifier interval, then access the target container set with the smaller left-end value of the data identifier interval. The recovery unit is used to recover the target data according to the acquisition order of each target data block.
7. A data processing apparatus, comprising: Central processing unit, memory, and input / output interfaces; The memory is either a short-term storage memory or a persistent storage memory; The central processing unit is configured to communicate with the memory and execute instructions in the memory to perform the data processing method according to any one of claims 1 to 2 or any one of claims 3 to 5.
8. A computer storage medium, characterized in that, The computer storage medium stores instructions that, when executed on the computer, cause the computer to perform the data processing method as described in any one of claims 1 to 2 or any one of claims 3 to 5.