Data compression method, data compression device and related equipment

Abstract

The invention discloses a data compression method, a data compression device and related equipment. N similar data sets are obtained, each similar data set comprises two similar data blocks, one is a reference block, and the other is a similar block. The N similar data groups do not comprise the same data feature, and N is an integer greater than or equal to 2. Aggregating the N reference blocks in the N similar data groups to obtain a first aggregated block; and aggregating the N similar blocks in the N similar data groups to obtain a second aggregated block. And performing differential compression on the first polymerized block and the second polymerized block. In the application, since the N similar data groups do not include the same data feature, the reference blocks in the first aggregation block do not include the same data feature, and the similar blocks in the second aggregation block do not include the same data feature. Therefore, by performing differential compression on the first data block and the second aggregation block, repeated data from the plurality of data blocks can be calculated, and the data compression ratio is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data compression method, a data compression device and related equipment. Background Art

[0002] With the rapid development of big data, massive data is growing explosively, which brings huge challenges to data storage management. In order to improve the utilization of storage space, data compression technology is an indispensable key technology in the storage system. Data compression is the process of encoding the original data with less space. Data compression refers to a technical method to reduce the amount of data to reduce storage space, improve its transmission, storage and processing efficiency, or reorganize data according to a certain algorithm to reduce data redundancy and storage space without losing useful information.

[0003] Delta Compression technology is used to eliminate redundant data between two different similar data, so that only one copy of the difference data needs to be stored additionally on the basis of preserving the same data between the two data. Delta compression technology includes two links: similarity detection and differential encoding. Similarity detection is mainly to find a pair of similar data blocks, which are called reference blocks and similar blocks respectively. After the differential encoding link, the similar blocks can be encoded into copy instructions and insert instructions based on the reference blocks. Among them, for the data content contained in the similar block that is the same as the reference block, the copy instruction is encoded, and for the data content contained in the similar block that is different from the reference block, the insert instruction is encoded.

[0004] In actual applications, in a storage space, a data block often contains multiple data blocks similar to it. After differential compression of the data block and one of the similar data blocks, the resulting reference block or differential block will also contain the same data content as the other similar data blocks, so the data compression ratio of this differential compression is relatively low. Summary of the invention

[0005] The present application provides a data compression method, a data compression device and related equipment for improving the data compression ratio.

[0006] In a first aspect, the present application provides a method for data compression. The compression object of differential compression is two similar data blocks in a storage space. Therefore, it is necessary to first perform a similarity test on the data blocks in the storage space to screen out two similar data blocks. Among the two similar data blocks, any one of them can be selected as a reference block, and the other can be selected as a similar block. In the similarity detection technology, the two similar data blocks detected include at least one identical data feature, and the data content between the two similar data blocks is not completely the same. In other words, between the two similar data blocks, the reference block and the similar block include at least one identical data feature, and the data content between the reference block and the similar block is not completely the same.

[0007] Generally speaking, there are multiple groups of such two similar data blocks (regarded as a group) detected in the storage space. In the present application, it is necessary to obtain N similar data groups from multiple groups of two similar data blocks, where N is an integer greater than or equal to 2. Among them, the N similar data groups do not include the same data features, but a similar data group in the present application is a group of two similar data blocks, so the two data blocks in a similar data group must include the same data features.

[0008] It should be understood that the data features mentioned in this application are equivalent to the fingerprints of data blocks. Exemplarily, the data features may be feature values ​​obtained by performing a hash operation on part of the data content of a data block, or may be a super-feature value obtained by combining multiple feature values, or may be other features used to represent part of the data content of a data block, which is not limited in this application.

[0009] After aggregating N data blocks, the aggregated blocks in the present application (such as the first aggregated block and the second aggregated block) will be obtained, and the same data features will not be included in a single aggregated block in the present application. Specifically, the N reference blocks in the N similar data groups are aggregated to obtain the first aggregated block; the N similar blocks in the N similar data groups are aggregated to obtain the second aggregated block. It can be seen from the above that the N similar data groups do not include the same data features. Therefore, the reference blocks in the first aggregated block should not include the same data features. Similarly, the similar blocks in the second aggregated block should not include the same data features.

[0010] In the present application, the process of aggregating multiple data blocks into an aggregate block itself is equivalent to a logical division, and does not change the physical properties of the data blocks, such as the data content, data structure or storage location of each data block. Aggregating multiple data blocks into an aggregate block refers to treating multiple data blocks that meet the aggregation conditions as a whole data block for subsequent processing (such as differential compression). Multiple data blocks within a single aggregate block come from different storage addresses, and the storage addresses of these data blocks may be adjacent or non-adjacent, and this application does not limit this.

[0011] Since the two data blocks (reference block and similar block) in each similar data group will be respectively allocated to the first aggregate block and the second aggregate block, the first aggregate block and the second aggregate block must be similar. Therefore, the first aggregate block and the second aggregate block can also be regarded as two similar data blocks. In the present application, the first aggregate block is used as the reference block, and the second aggregate block is used as the similar block, and differential compression is performed to obtain an aggregate differential block. Among them, the aggregate differential block includes metadata for pointing to the repeated data between the first aggregate block and the second aggregate block, and the difference between the first aggregate block and the second aggregate block.

[0012] In the present application, since the N similar data groups do not include the same data features, the reference blocks in the first aggregate block should also not include the same data features. Similarly, the similar blocks in the second aggregate block should also not include the same data features. However, since the two data blocks in each similar data group are respectively allocated to the first aggregate block and the second aggregate block, the first aggregate block and the second aggregate block must be similar. Therefore, during the differential compression of the first data block and the second aggregate block, the duplicate data from multiple data blocks can be calculated, thereby improving the data compression ratio.

[0013] On the other hand, in the process of obtaining similar data groups, similarity detection is performed on two data blocks. When two data blocks are similar and do not include the same data features as other similar data groups, the two data blocks can be used as a similar data group, and then aggregated with other similar data groups, and differential compression is completed. Therefore, the data compression method in the present application does not need to traverse all data blocks in the storage space. It can be applied to storage spaces where new data is stored in real time (such as online storage scenarios or cloud storage scenarios), which improves the flexibility of the solution.

[0014] Based on the first aspect, in an optional implementation, a first data block is obtained, and the first data block is a new data block stored in the storage space. The first data block is repeated with a data block (first similar block) in the second aggregate block. That is, the data content between the first data block and the first similar block in the second aggregate block is exactly the same, and the first similar block is one of the similar blocks in the second aggregate block. At this time, in the present application, the first data block and the first similar block may not be deduplicated, but the first reference block and the first data block may be differentially compressed to obtain a first differential block, the first differential block includes metadata for pointing to the duplicate data between the first reference block and the first data block, and the difference between the first reference block and the first data block, wherein the first reference block is one of the reference blocks in the first aggregate block, and the first reference block and the first similar block are from the same similar data group. In this way, when decompressing and recovering the first data block, it is only necessary to read the first reference block into the memory, without having to read the entire second aggregate block into the memory for decompression, thereby improving the efficiency of data decompression and recovery.

[0015] Based on the first aspect, in an optional implementation, a second data block is obtained, and the second data block is a new data block stored in the storage space. Among them, the second data block and the second similar block in the second aggregate block include at least one identical data feature, and the data content between the second data block and the second similar block is not completely the same, and the second similar block is one of the similar blocks in the second aggregate block. In other words, the second data block is similar to a data block (the second similar block) in the second aggregate block. At this time, when the second reference block is also similar to the second data block, the second reference block and the second data block are differentially compressed to obtain a second differential block, which includes metadata for pointing to the second reference block and the difference between the second reference block and the second data block, wherein the second reference block is one of the reference blocks in the first aggregate block, and the second reference block and the second similar block are from the same similar data group.

[0016] Based on the first aspect, in an optional implementation, the third reference block includes metadata of the third data block, the data content between the third data block and the third reference block is exactly the same, the third data block is a data block outside the N similar data groups, and the third reference block is one of the reference blocks of the first aggregate block. In other words, when a data block in the aggregate block is repeated with other data blocks outside the N similar data groups, the data block in the aggregate block can be the metadata of the repeated data block, that is, the data block in the aggregate block does not have to be an original data, thereby improving the flexibility of the aggregate block.

[0017] Based on the first aspect, in an optional implementation, there is a third aggregate block (for example, after the new data blocks are accumulated, they are aggregated into the third aggregate block), and the data content between the third aggregate block and the second aggregate block is exactly the same. In this case, deduplication processing is performed on the third aggregate block and the second aggregate block, which does not cause read amplification. Therefore, the third aggregate block can be updated, and the updated third aggregate block includes metadata for pointing to the second aggregate block.

[0018] Based on the first aspect, in an optional implementation, there is a fourth aggregate block, the fourth aggregate block includes N data blocks, and the fourth aggregate block includes at least one identical data feature with the second aggregate block, and the data content between the fourth aggregate block and the second aggregate block is not completely the same. In other words, the fourth aggregate block is similar to the second aggregate block. At this time, the first aggregate block and the fourth aggregate block can be differentially compressed to obtain an updated fourth aggregate block, and the updated fourth aggregate block includes the difference between the first aggregate block and the fourth aggregate block, and metadata for pointing to duplicate data between the first aggregate block and the fourth aggregate block.

[0019] Based on the first aspect, in an optional implementation, M data blocks to be compressed are obtained, data contents of the M data blocks to be compressed are not completely the same, and M is an integer greater than or equal to 2N. Similarity detection is performed on the M data blocks to be compressed to determine N similar data groups in the M data blocks to be compressed.

[0020] Based on the first aspect, in an optional implementation, X initial data blocks are obtained, where X is an integer greater than or equal to M. Initial data blocks with completely identical data contents among the X initial data blocks are deduplicated to obtain M data blocks to be compressed.

[0021] In a second aspect, the present application provides a data compression device, comprising:

[0022] an acquisition unit, configured to acquire N similar data groups, each similar data group including a reference block and a similar block, the reference block and the similar block in each similar data group including at least one common data feature, and the data content between the reference block and the similar block in each similar data group is not completely the same, the N similar data groups do not include the same data feature, and N is an integer greater than or equal to 2;

[0023] A processing unit, configured to aggregate the N similar data groups to obtain a first aggregate block and a second aggregate block, wherein the first aggregate block includes the N reference blocks of the N similar data groups, and the second aggregate block includes the N similar blocks in the N similar data groups;

[0024] The processing unit is also used to perform differential compression on the first aggregate block and the second aggregate block to obtain an aggregate differential block, which includes metadata for pointing to duplicate data between the first aggregate block and the second aggregate block, and the difference between the first aggregate block and the second aggregate block.

[0025] Based on the second aspect, in an optional implementation, the acquisition unit is further used to acquire a first data block, wherein the data content between the first data block and a first similar block in the second aggregate block is completely the same, and the first similar block is one of the similar blocks in the second aggregate block;

[0026] The processing unit is also used to perform differential compression on the first benchmark block and the first data block to obtain a first differential block, wherein the first differential block includes metadata for pointing to duplicate data between the first benchmark block and the first data block, and the difference between the first benchmark block and the first data block, wherein the first benchmark block is one of the benchmark blocks of the first aggregate block, and the first benchmark block and the first similar block are from the same similar data group.

[0027] Based on the second aspect, in an optional implementation, the acquisition unit is further used to acquire a second data block, wherein the second data block and a second similar block in the second aggregated block include at least one common data feature, and the data content between the second data block and the second similar block is not completely the same, and the second similar block is one of the similar blocks in the second aggregated block;

[0028] The processing unit is also used to perform differential compression on the second benchmark block and the second data block to obtain a second differential block, wherein the second differential block includes metadata for pointing to the second benchmark block and the difference between the second benchmark block and the second data block, wherein the second benchmark block is one of the benchmark blocks of the first aggregate block, and the second benchmark block and the second similar block are from the same similar data group.

[0029] Based on the second aspect, in an optional implementation, the third benchmark block includes metadata of the third data block, the data content between the third data block and the third benchmark block is exactly the same, the third data block is a data block outside the N similar data groups, and the third benchmark block is one of the benchmark blocks of the first aggregate block.

[0030] Based on the second aspect, in an optional implementation manner, the acquisition unit is further used to acquire a third aggregate block, where the third aggregate block includes N data blocks, and data content between the third aggregate block and the second aggregate block is completely the same;

[0031] The processing unit is further used to update the third aggregate block, where the updated third aggregate block includes metadata for pointing to the second aggregate block.

[0032] Based on the second aspect, in an optional implementation, the acquisition unit is further used to acquire a fourth aggregate block, where the fourth aggregate block includes N data blocks, the fourth aggregate block includes at least one common data feature with the second aggregate block, and the data content of the fourth aggregate block is not completely the same as that of the second aggregate block;

[0033] The processing unit is also used to perform differential compression on the fourth aggregate block according to the first aggregate block to obtain an updated fourth aggregate block, wherein the updated fourth aggregate block includes the difference between the first aggregate block and the fourth aggregate block, and metadata for pointing to duplicate data between the first aggregate block and the fourth aggregate block.

[0034] Based on the second aspect, in an optional implementation manner, the processing unit is specifically configured to:

[0035] Aggregating N benchmark blocks in N similar data groups to obtain a first aggregated block;

[0036] N similar blocks in the N similar data groups are aggregated to obtain a second aggregated block.

[0037] Based on the second aspect, in an optional implementation manner, the acquiring unit is specifically configured to:

[0038] Obtain M data blocks to be compressed, where data contents of the M data blocks to be compressed are not completely the same, and M is an integer greater than or equal to 2N;

[0039] Similarity detection is performed on the M data blocks to be compressed, and N similar data groups among the M data blocks to be compressed are determined.

[0040] Based on the second aspect, in an optional implementation manner, the acquiring unit is specifically configured to:

[0041] Obtain X initial data blocks, where X is an integer greater than or equal to M;

[0042] De-duplication processing is performed on initial data blocks with completely identical data contents among the X initial data blocks to obtain M data blocks to be compressed.

[0043] The information interaction and execution process and other contents of the embodiment shown in this aspect are based on the same concept as the embodiment shown in the first aspect. Therefore, for the description of the beneficial effects shown in this aspect, please refer to the above-mentioned first aspect, and the details will not be repeated here.

[0044] In a third aspect, the present application provides a computing device, comprising: a processor, the processor is coupled to a memory, the memory is used to store instructions, when the instructions are executed by the processor, the computing device implements the method in the above-mentioned first aspect or any possible implementation of the first aspect.

[0045] In a fourth aspect, the present application provides a computer-readable storage medium having instructions stored thereon, which, when executed, causes a computer to execute the method in the above-mentioned first aspect or any possible implementation of the first aspect.

[0046] In a fifth aspect, the present application provides a computer program product, which includes a computer program code. When the computer program code runs on a computer, it enables the computer to execute the method in the above-mentioned first aspect or any possible implementation manner of the first aspect.

[0047] In a sixth aspect, the present application provides a chip, comprising: a processor, the processor is coupled to a memory, the memory is used to store instructions, and when the instructions are executed by the processor, the chip implements the method in the above-mentioned first aspect or any possible implementation of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0049] Figure 1 A schematic diagram of a scenario for similarity detection based on the super eigenvalue method;

[0050] Figure 2 A schematic diagram of a scenario of an existing data compression solution;

[0051] Figure 3 A schematic diagram of another existing data compression scheme;

[0052] Figure 4 Schematic diagram of the data compression method in the embodiment of the present application;

[0053] Figure 5 A schematic diagram of a possible data compression scenario in an embodiment of the present application;

[0054] Figure 6 A possible structural diagram of a redundant data reduction system is shown;

[0055] Figure 7 A schematic diagram of a compression process for existing data in an embodiment of the present application;

[0056] Figure 8 Schematic diagram of an experimental method of a data compression method in an embodiment of the present application;

[0057] Fig. 9is a schematic diagram of a compression method for new data;

[0058] Fig.10 This is a schematic diagram of an optimized compression method for new data in an embodiment of the present application;

[0059] Fig.11 Schematic diagram of a compression process for new data in an embodiment of the present application;

[0060] Fig.12 A schematic diagram of an application scenario of the data compression method in an embodiment of the present application;

[0061] Fig.13 A schematic diagram of another application scenario of the data compression method in an embodiment of the present application;

[0062] Fig.14 A schematic diagram of the structure of a data compression device provided in an embodiment of the present application;

[0063] Fig.15 A schematic diagram of the logical structure of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0064] The embodiments of the present application provide a data compression method, a data compression device and related equipment for improving the data compression ratio.

[0065] The embodiments of the present application are described below in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation mode of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the embodiments of the present application. It is known to those skilled in the art that with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0066] In the embodiments of the present application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A and B can be singular or plural. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0067] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can, for example, be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0068] Some nouns or terms used in the embodiments of the present application are explained below, and these nouns or terms are also considered as part of the content of the invention.

[0069] Data Compression: With the rapid development of big data, massive data has exploded, bringing huge challenges to data storage management. In order to improve the utilization of storage space, data compression technology is an indispensable key technology in the storage system. Data compression is the process of encoding the original data with less space. Data compression refers to a technical method that reduces the amount of data to reduce storage space, improves its transmission, storage and processing efficiency, or reorganizes data according to a certain algorithm to reduce data redundancy and storage space without losing useful information.

[0070] Data deduplication technology is a technology that effectively eliminates redundant data on a large scale. It has become a hot topic in storage system research in recent years. Data deduplication technology can not only greatly save storage space and improve the utilization of storage resources, but also improve the transmission efficiency of network bandwidth by avoiding the transmission of redundant data. However, with the development of data deduplication technology, data deduplication technology also faces many challenges. Since data deduplication technology uses secure hash summaries to uniquely identify data blocks, it greatly simplifies the search and management of duplicate data. However, this will result in data deduplication technology can only identify completely duplicate data blocks, but not those very similar data blocks. For example, when two data blocks A1 and A2 are only a few bytes different, although the data contents of A1 data block and A2 data block are very similar, after the calculation of the secure hash summary, A1 data block and A2 data block will obtain two different data block fingerprints, thereby ignoring the redundant processing of similar data such as A1 data block and A2 data block. Because Delta Compression technology can well identify and eliminate this type of redundant data, it has gained more and more attention.

[0071] Differential compression technology is a technology that eliminates redundant data between two different data, so that only one copy of the differential data needs to be stored. Data deduplication can only eliminate duplicate data with the same content, while differential compression can eliminate redundancy between two data with different contents. The compression objects of differential compression are similar files or data blocks. In order to maximize the differential compression effect and reduce the computational overhead of differential encoding, similarity detection technology is often used to select the objects of differential compression during the differential compression process.

[0072] Most of the existing similar feature calculation methods are based on the classic Broder theory. Broder described the similarity problem as the problem of intersection and union in sets, and proposed MinHash as the theoretical basis for estimating set similarity. In order to quickly evaluate the similarity of two sets, Broder proposed the "Broder Theorem": Assume that A and B are two sets, F(A) and F(B) are the corresponding sets obtained by calculating the elements in A and B using the MinHash function F. If min(S) is used to represent the minimum element in set S, it can be expressed as:

[0073]

[0074] Broder theory describes that the probability that two sets have the same minimum hash element is equal to the Jaccard coefficient of the two sets. If the elements in sets A and B are composed of all shingles (dividing files into overlapping fixed-length continuous strings, called shingles) in two data blocks C1 and C2, according to Broder theory, the probability that F(A) and F(B) have the same minimum element is equal to the similarity of the two data blocks.

[0075] Odess uses a feature calculation method based on Broder's theory, called "NTransform", and combines the super-feature method proposed by Broder. NTransform calculates multiple hash functions and extracts multiple eigenvalues. In practical applications, the cost of calculating multiple hash functions is too high. In order to extract multiple independent eigenvalues, NTransform first calculates a Rabin fingerprint fp for the content of the sliding window, and then performs multiple linear transformations on the fingerprint to generate multiple independent hash values, as shown in the following formula:

[0076] h i (fp)=(m i *fp+a i )mod2^32

[0077] Among them, m i and a iIt is a set of random numbers. In a large-scale storage system, it is computationally expensive to calculate the similarity between a data block and all other data blocks and select the data block with the highest similarity. In most cases, as long as relatively similar data blocks are found, the requirement can be met. Super-feature combines multiple feature values ​​into a super feature value for similarity matching. As long as two data blocks have a same super feature value match, they are similar. The strategy of combining multiple feature values ​​into a super feature value avoids the calculation of specific similarity and greatly reduces the computational overhead of the matching process. Please refer to Figure 1 , Figure 1 Figure 1 is a schematic diagram of a scenario for similarity detection based on the super eigenvalue method. Figure 1 As shown in , Super-feature can pack multiple feature values ​​into a super feature value for matching, which reduces the probability that dissimilar data blocks have the same feature value, and increases the similarity threshold of the similar blocks found. Figure 1 In the scenario shown, 4 eigenvalues ​​are combined into one super eigenvalue, and 12 eigenvalues ​​are combined into 3 super eigenvalues: SF1 super eigenvalue, SF2 super eigenvalue, and SF3 super eigenvalue. As long as any super eigenvalue between two data blocks is equal, it means that the similarity between the two data blocks is very high.

[0078] Delta Compression is a compression technology derived from dictionary encoding. Unlike dictionary encoding, delta compression does not encode within the compression window of a single object, but encodes between two objects. The compression process of the LZ77 compression algorithm can be simply understood as: if the subsequent string is the same as the previously processed string, find the longest match between them and replace it with the COPY command. The most commonly used differential compression tools, such as vcdiff, Xdelta, and Zdelta, can be simply considered as LZ77 compression between two similar files. Suppose there are two similar data blocks A and B, data block B is the "similar block" that needs to be compressed, and data block A is called the "reference block" of data block B. Delta compression will find the content that exists in data block B but does not exist in data block A and write it into a file. The process of generating this file is called differential encoding. Specifically, differential encoding uses sliding window technology to simultaneously detect the content in data block A and data block B. For the content in data block B, if the same content exists in data block A, the COPY command is used for encoding, otherwise the INSERT command is used for encoding. The COPY command contains the offset and length of the repeated content between data block A and data block B in data block A, while the INSERT command contains the length and content of the non-duplicate content between data block A and data block B in data block B. The resulting file containing the COPY and INSERT commands is called a differential block. The size of the differential block is generally smaller than that of data block B. When storing, the differential block will be saved instead of data block B, thereby reducing storage overhead. The rate of differential encoding and the size of the differential block are both related to the similarity between data block A and data block B. The higher the similarity, the faster the differential encoding speed and the smaller the differential block. Differential decoding is to regenerate data block B according to the differential block and data block A according to the command.

[0079] Next, the existing data compression schemes are introduced.

[0080] See also Figure 2 , Figure 2 FIG. 1 is a schematic diagram of a scenario of an existing data compression solution. Figure 2 As shown, it is necessary to traverse all data blocks in the storage space and find a data block (for example Figure 2 A plurality of different similar blocks (e.g., Figure 2 The data block B, data block C, data block D and data block E shown in the figure are then aggregated into a whole with similar blocks of the data block, and then the whole is self-compressed.

[0081] against Figure 2The data compression scheme shown in the figure needs to traverse all data blocks in the storage space. When new data is stored in the storage space, all similar blocks of the new data need to be found, which brings huge computing resource overhead. Therefore, the data compression scheme is difficult to apply to storage spaces where new data is stored in real time (such as online storage scenarios or cloud storage scenarios).

[0082] See also Figure 3 , Figure 3 FIG. 1 is a schematic diagram of another existing data compression scheme. Figure 3 As shown, it is necessary to traverse all data blocks in the storage space, find multiple data blocks with at least one identical feature fingerprint, and merge these data blocks into a joint compression group. A reference block (e.g. Figure 3 The data block A shown in FIG. 1 is the reference block, and the remaining data blocks (eg Figure 3 The data blocks B, C, D and E shown in FIG. 1 are all differentially compressed with the reference block (data block A) as the base block. Finally, the reference block (data block A) is self-compressed.

[0083] against Figure 3 As shown, in a storage space, a data block often has multiple data blocks similar to it. After the data block is differentially compressed with one of the similar data blocks, the obtained reference block or differential block will also contain the same data content as other similar data blocks, so the data compression ratio of this differential compression is relatively low.

[0084] on the other hand, Figure 3 The data compression scheme shown is similar to the above Figure 2 The data compression schemes shown are the same, both of which require traversing all data blocks in the storage space to find similar blocks of the new data. Therefore, Figure 3 The data compression scheme shown is also difficult to apply to storage spaces where new data is stored in real time (such as online storage scenarios or cloud storage scenarios).

[0085] In view of this, the embodiments of the present application disclose a data compression method, a data compression device and related equipment for improving the data compression ratio. Among them, the data compression method in the embodiments of the present application can be applied to a device that stores multiple data blocks and has a data storage function (including but not limited to a mobile phone, a computer, a network device, a cloud storage device or a server, etc.), and is used to compress the data stored in the device. The embodiments of the present application do not limit the execution subject of the data compression method. For example, the execution subject of the data compression method in the embodiments of the present application can be the device itself that stores multiple data blocks, or it can also be a chip, a chip system, or a processor that supports the device to implement the data compression method, or it can also be a logical node, a logical module or software that can implement all or part of the data compression function. Please refer to Figure 4 , Figure 4 FIG. 1 is a flow chart of a data compression method in an embodiment of the present application. Figure 4 As shown, the data compression method in the embodiment of the present application includes but is not limited to step 101, step 102 and step 103. Next, step 101, step 102 and step 103 are described in detail.

[0086] 101. Obtain N similar data groups, each similar data group includes a reference block and a similar block, the N similar data groups do not include the same data features, and N is an integer greater than or equal to 2.

[0087] The compression object of differential compression is two similar data blocks in the storage space. Therefore, it is necessary to first perform similarity detection on the data blocks in the storage space to screen out two similar data blocks. Among the two similar data blocks, any one of them can be selected as the reference block, and the other as the similar block. In the similarity detection technology, the two similar data blocks detected include at least one identical data feature, and the data content between the two similar data blocks is not completely the same. In other words, between the two similar data blocks, the reference block and the similar block include at least one identical data feature, and the data content between the reference block and the similar block is not completely the same.

[0088] Generally speaking, such two similar data blocks (considered as a group) will be detected in multiple groups in the storage space. In the embodiment of the present application, it is necessary to obtain N similar data groups from multiple groups of similar two data blocks, where N is an integer greater than or equal to 2. Among them, the N similar data groups do not include the same data features, but a similar data group in the embodiment of the present application is a group of two similar data blocks, so the two data blocks in a similar data group must include the same data features.

[0089] It should be understood that the data features mentioned in the embodiments of the present application are equivalent to the fingerprints of the data blocks. Exemplarily, the data features may be feature values ​​obtained by performing a hash operation on part of the data content of the data block, or may be a super-feature value obtained by combining multiple feature values, or may be other features used to represent part of the data content of the data block, which is not limited in the embodiments of the present application.

[0090] 102. Aggregate N similar data groups to obtain a first aggregate block and a second aggregate block, wherein the first aggregate block includes N reference blocks of the N similar data groups, and the second aggregate block includes N similar blocks in the N similar data groups.

[0091] After aggregating N data blocks, the aggregated blocks in the embodiment of the present application (such as the first aggregated block and the second aggregated block) will be obtained, and the same data features will not be included in a single aggregated block in the embodiment of the present application. Specifically, the N reference blocks in the N similar data groups are aggregated to obtain the first aggregated block; the N similar blocks in the N similar data groups are aggregated to obtain the second aggregated block. It can be seen from the above that the N similar data groups do not include the same data features. Therefore, the reference blocks in the first aggregated block should not include the same data features. Similarly, the similar blocks in the second aggregated block should not include the same data features.

[0092] In the embodiment of the present application, the process of aggregating multiple data blocks into an aggregate block itself is equivalent to a logical division, and does not change the physical properties of the data blocks, for example, it does not change the data content, data structure or storage location of each data block. Aggregating multiple data blocks into an aggregate block refers to treating multiple data blocks that meet the aggregation conditions as a whole data block for subsequent processing (such as differential compression). Multiple data blocks within a single aggregate block come from different storage addresses, and the storage addresses of these data blocks may be adjacent or non-adjacent, which is not limited in the embodiment of the present application.

[0093] In a possible implementation, the third reference block includes metadata of the third data block, the data content of the third data block and the third reference block are completely the same, the third data block is a data block outside the N similar data groups, and the third reference block is one of the reference blocks of the first aggregate block. In other words, when a data block in the aggregate block is repeated with other data blocks outside the N similar data groups, the data block in the aggregate block can be the metadata of the repeated data block (e.g., a reference pointer of the repeated data block), that is, the data block in the aggregate block does not have to be an original data, thereby improving the flexibility of the aggregate block.

[0094] 103. Perform differential compression on the first aggregate block and the second aggregate block to obtain an aggregate differential block.

[0095] Since the two data blocks (reference block and similar block) in each similar data group will be respectively allocated to the first aggregate block and the second aggregate block, the first aggregate block and the second aggregate block must be similar. Therefore, the first aggregate block and the second aggregate block can also be regarded as two similar data blocks. In the embodiment of the present application, the first aggregate block is used as the reference block, and the second aggregate block is used as the similar block, and differential compression is performed to obtain an aggregate differential block. Among them, the aggregate differential block includes metadata for pointing to the repeated data between the first aggregate block and the second aggregate block, and the difference between the first aggregate block and the second aggregate block.

[0096] In the embodiment of the present application, since the N similar data groups do not include the same data features, the reference blocks in the first aggregate block should also not include the same data features. Similarly, the similar blocks in the second aggregate block should also not include the same data features. However, since the two data blocks in each similar data group will be allocated to the first aggregate block and the second aggregate block respectively, the first aggregate block and the second aggregate block must be similar. Therefore, during the differential compression of the first data block and the second aggregate block, the repeated data from multiple data blocks can be calculated, thereby improving the data compression ratio.

[0097] To better understand the embodiments of this application, please refer to Figure 5 , Figure 5 FIG. 1 is a schematic diagram of a possible data compression scenario in an embodiment of the present application. Figure 5 As shown in Figure 2, three similar data groups were obtained, namely Figure 5As shown in the first group (C1 data block and C1' data block), the second group (C2 data block and C2' data block) and the third group (C3 data block and C3' data block), the data blocks in the similar data groups are similar to each other. Assuming that the C1 data block, the C2' data block and the C3' data block are also similar and compressible, the traditional differential compression only performs differential compression between the C1 data block and the C1' data block, between the C2 data block and the C2' data block, and between the C3 data block and the C3' data block, ignoring the redundant data between the C1 data block and the C2' data block and the C3' data block, resulting in insufficient compression. The data compression method in the embodiment of the present application aggregates the C1 data block, the C2 data block and the C3 data block into one aggregate block and uses it as a reference block (equivalent to the first aggregate block in the embodiment of the present application), aggregates the C1' data block, the C2' data block and the C3' data block into another aggregate block and uses it as a similar block (equivalent to the second aggregate block in the embodiment of the present application), that is, two aggregate blocks are obtained. Then, differential compression is performed on the two aggregate blocks, thereby calculating the redundant data between the C1 data block and the C2' data block and the C3' data block, thereby improving the compression ratio.

[0098] On the other hand, since the similarity detection is performed on two data blocks in the process of obtaining similar data groups. When two data blocks are similar, and these two data blocks do not include the same data features as other similar data groups, then these two data blocks can be used as a similar data group, and then aggregated with other similar data groups, and differential compression is completed. Therefore, the data compression method in the embodiment of the present application does not need to traverse all data blocks in the storage space. It can be applied to storage spaces where new data is stored in real time (such as online storage scenarios or cloud storage scenarios), which improves the flexibility of the solution.

[0099] The data compression method process shown in the above steps 101 to 103 can be deployed as a functional module (such as Figure 6 The differential compression module shown in Figure 1 is integrated into a redundant data reduction system. Figure 6 , Figure 6 A possible structural diagram of a redundant data reduction system is shown in FIG. Figure 6 As shown, the redundant data reduction system includes a hash calculation module, a data deduplication module, a differential compression module, and a data persistence module. For the read data to be compressed, the redundant data reduction system can use a variety of block algorithms such as variable-length block and fixed-length block to block the data, so that the data stream to be compressed becomes a more fine-grained data block to be compressed, and then these data blocks to be compressed will be input into the above modules for processing. Figure 5In the example, each module can be executed in parallel by multiple threads, thereby speeding up the operation of the redundant data reduction system. Figure 5 The various modules shown in are introduced.

[0100] Hash calculation module: After the data to be compressed is divided into fine-grained data blocks by the data block algorithm, these data blocks will be processed by the hash calculation module. Specifically, the hash calculation module calculates a fixed-length secure hash summary for each data block to uniquely identify the data block, which is equivalent to configuring a unique fingerprint for the data block, preparing for the subsequent data deduplication process and differential compression process.

[0101] Data deduplication module: used to compare fingerprints between different data blocks, so as to quickly detect duplicate data blocks, avoid byte-by-byte comparison between data block contents, improve the efficiency of detecting duplicate data blocks, and further improve data compression performance. For multiple duplicate data blocks detected, the data deduplication module deduplicates these duplicate data blocks and only retains one copy to reduce data redundancy and waste of storage space.

[0102] Differential compression module: Since the data deduplication module can only identify completely duplicate data blocks, but cannot identify similar data blocks. The differential compression module finds data blocks that are similar to the current data block by calculating and comparing the data features between the data blocks. For the detected similar data blocks, the differential compression module executes the data compression method provided in the embodiment of the present application on these similar data blocks, updates the similar blocks in the similar data blocks to differential blocks, further reduces the redundant data between the similar data blocks, and further reduces the storage overhead.

[0103] Data persistence module: After the differential compression module has processed similar data blocks, it can persist the generated differential blocks.

[0104] The redundant data reduction system in the embodiment of the present application can be used to compress existing data in the storage space, or it can also be used to compress new data stored in the storage space in real time. Next, the compression process for existing data and the compression process for new data in the redundant data reduction system will be introduced in conjunction with the accompanying drawings. Figure 7 , Figure 7 FIG. 1 is a schematic diagram of the compression process for existing data in an embodiment of the present application. Figure 7 As shown in FIG. 1 , the compression process for existing data includes:

[0105] 201.Configure the capacity threshold of a single aggregation block.

[0106] According to factors such as the performance of the storage device, the needs of the actual application scenario, or the frequency of data updates, the capacity threshold of a single aggregate block (such as the first aggregate block or the second aggregate block in the embodiment of the present application) can be pre-configured, wherein the capacity threshold of a single aggregate block indicates the maximum number of data blocks or the maximum amount of data included in the aggregate block. Exemplarily, the capacity of a single aggregate block can be configured to 70 data blocks, indicating that a single aggregate block can aggregate up to 70 data blocks. If you want to obtain a higher online compression speed, you can appropriately reduce the capacity threshold of a single aggregate block.

[0107] 202. Perform similarity matching on the data blocks.

[0108] like Figure 5 As shown in FIG. 1 , the existing data blocks in the storage space will first be processed for data deduplication. After the data deduplication process, the remaining data blocks are matched for similarity.

[0109] 203. Determine whether the data block passes similarity matching.

[0110] If it passes, it means that the data block is similar to other data blocks in the storage space, and then step 204 is executed; if it fails, it means that the data block cannot be matched with any data block in the storage space as a similar data block, and then step 208 is executed.

[0111] 204. Determine whether the capacity of the aggregation block is full.

[0112] If yes, then execute step 205 and step 207. If no, then execute step 206.

[0113] 205. Create a new aggregation block.

[0114] Although the data block matches other similar data blocks, the capacity of the existing aggregation block is full, so a new aggregation block is created and the data block is placed in the new aggregation block.

[0115] 206. Aggregate into existing aggregation block.

[0116] The data block matches other similar data blocks. Since the capacity of the existing aggregate block is not full, the data block is placed in the existing aggregate block. As can be seen from the above, the single aggregate block in the embodiment of the present application does not include the same data features. Therefore, the existing aggregate block merged by the data block in step 206 does not include the same data features as the data block. For the specific process of step 206, please refer to the aforementioned Figure 4 The corresponding step 102 will not be described again here.

[0117] 207. Perform differential compression on full aggregate blocks.

[0118] Since the capacity of the current aggregate block is full, differential compression can be performed on the aggregate block. Figure 4 The corresponding step 103 is not described again here. After differential compression, the obtained aggregated differential block is executed in step 208 .

[0119] It should be understood that Figure 7 In the example, the data block in the aggregate block reaches the upper capacity limit, which is used as a condition to trigger differential compression of the aggregate block. In actual applications, other conditions can also be used to trigger the process of differential compression of aggregate blocks, for example, differential compression of aggregate blocks is performed periodically, which is not limited here.

[0120] In the embodiment of the present application, step 205 may be performed first, and then step 207; or, step 207 may be performed first, and then step 205; or, step 205 and step 207 may be performed simultaneously, and the specific details are not limited here.

[0121] 208. Persistent storage.

[0122] against Figure 4 Corresponding embodiment (step 101 to step 103) and Figure 7 Corresponding to the embodiment (step 201 to step 208), the embodiment of the present application provides Figure 8 The experimental method shown is used to verify the data compression method in the embodiment of the present application.

[0123] After the compression process for existing data in the storage space is completed, if new data is stored in the storage space, the compression process for the new data can be performed. Generally speaking, after the new data is stored in the storage space, the new data will be deduplicated first. Fig. 9 , Fig. 9 A schematic diagram of a compression method for new data. Fig. 9 As shown, version 1 and version 2 are data already in the storage space, and version 3 is new data in the storage space.

[0124] In the traditional differential compression scheme, since the C1 data block and C2 data block in version 2 are exactly the same as the C1 data block and C2 data block in version 1, when saving the C1 data block and C2 data block in version 2, it is only necessary to save the metadata pointing to the C1 data block and C2 data block (such as the reference pointer of the C1 data block and the reference pointer of the C2 data block). However, the C3' data block and C4' data block in version 2 are similar to the C3 data block and C4 data block in version 1, so the C3' data block in version 2 and the C3 data block in version 1 are differentially compressed to save the differential blocks, and the C4' data block in version 2 and the C4 data block in version 1 are differentially compressed to save the differential blocks. Similarly, the C1 data block and C2 data block in version 3 are exactly the same as those in version 1, so we only need to save the metadata pointing to C1 and C2 (such as the reference pointer of the C1 data block and the reference pointer of the C2 data block). The C3' data block and C4' data block in version 3 are duplicates of the C3' data block and C4' data block in version 2, so we only need to save the metadata pointing to C3' and C4' in version 2 (such as the reference pointer of the C3' data block and the reference pointer of the C4' data block). When restoring the C3' metadata and C4 metadata in version 3, we only need to read the C3 metadata and C4 metadata of version 1 into the memory for decompression.

[0125] In the data compression method of the embodiment of the present application, when storing the C3' data block and the C4' data block in version 2, the aggregated differential block generated by the aggregated block where the C3' data block and the C4' data block are located will be saved. Since the C3' data block and the C4' data block in version 3 and the C3' data block and the C4' data block in version 2 are repeated, we only need to save the pointers to the C3' data block and the C4' data block in version 2. In this way, when we restore the C3' data block and the C4' data block in version 3, we need to read the aggregated block where the C3 data block is located, the aggregated block where the C4 data block is located, and the corresponding aggregated differential block into the memory for decompression, resulting in read amplification and decompression amplification, that is, the reference block read into the memory is changed from a data block to an aggregated block, the random read of the disk increases, and decompressing one data block becomes decompressing one aggregated block, the amount of calculation increases, resulting in a decrease in the speed of recovering the version 3 data block.

[0126] In view of the above problems, an optimization solution is provided in the embodiment of the present application. Specifically, the newly stored data is the first data block as an example, wherein the newly stored first data block is repeated with a data block (first similar block) in the second aggregate block. That is, the data content between the first data block and the first similar block in the second aggregate block is exactly the same, and the first similar block is one of the similar blocks of the second aggregate block. At this time, in the embodiment of the present application, the first data block and the first similar block may not be deduplicated, but the first reference block and the first data block may be differentially compressed to obtain a first differential block, the first differential block includes metadata for pointing to the duplicate data between the first reference block and the first data block, and the difference between the first reference block and the first data block, wherein the first reference block is one of the reference blocks of the first aggregate block, and the first reference block and the first similar block are from the same similar data group. In this way, when decompressing and recovering the first data block, it is only necessary to read the first reference block into the memory, without having to read the entire second aggregate block into the memory for decompression, thereby improving the efficiency of data decompression and recovery.

[0127] See also Fig.10 , Fig.10 Schematic diagram of an optimized compression method for new data in an embodiment of the present application. Fig.10 The left side is a compression scheme for new data that has not been optimized in the embodiment of the present application. In this scheme, when saving the C3' data block and C4' data block of version 3, what is actually saved is the difference between the aggregate block where the C3' data block and the C4' data block are located. So when restoring the C3' data block and C4' data block in version 3, it is necessary to read the aggregate block where the C3 data block and the C4 data block are located into the memory for decompression. Fig.10 As shown on the right, in the optimized compression method for new data in the embodiment of the present application, when storing the C3' data blocks and C4' data blocks in version 3, they will not be deduplicated, but the reference blocks of the C3' data blocks and C4' data blocks in version 1 will be directly found, and one-to-one differential compression will be performed (for example, differential compression will be performed between the C3' data blocks in version 3 and the C3 data blocks in version 1), and the differential part will be saved. Then, when restoring the C3' data blocks and C4' data blocks in version 3, only the C3 data blocks and C4 data blocks of version 1 need to be read into the memory, reducing random reads of the disk. This method can significantly reduce read amplification during data recovery and increase the speed of data recovery.

[0128] See also Fig.11 , Fig.11 Schematic diagram of the compression process for new data in an embodiment of the present application. Fig.11 The compression process for new data shown in the figure adopts the above optimized compression method for new data. The compression process for new data specifically includes:

[0129] 301. Calculate the fingerprint of the new data block.

[0130] After the new data stream is stored in the storage space, it is divided into multiple new data blocks. A fixed-length secure hash digest is calculated for the new data block to uniquely identify the data block, which is equivalent to configuring a unique fingerprint for each new data block.

[0131] 302. Determine whether there are duplicate data blocks.

[0132] If yes, go to step 303; if no, go to step 304

[0133] 303. Determine whether the duplicate data block of the new data block exists in the aggregate block.

[0134] If yes, go to step 305; if no, go to step 306.

[0135] 304. Direct storage.

[0136] Specifically, if there is no data block similar to the new data block in the storage space, then the original data of the new data block is directly stored in the traditional way; if there is a data block similar to the new data block in the storage space, and the data block similar to the new data block does not come from an aggregate block, then the new data block and the similar block of the new data block are differentially compressed.

[0137] If there is a data block similar to the new data block in the storage space, and the data block similar to the new data block comes from an aggregate block, then the new data block and the reference block of the aggregate block are differentially compressed. Specifically, in a possible implementation, the new data block is a second data block, wherein the second data block and the second similar block in the second aggregate block include at least one identical data feature, and the data content between the second data block and the second similar block is not completely the same, and the second similar block is one of the similar blocks in the second aggregate block. In other words, the second data block is similar to a data block (the second similar block) in the second aggregate block. At this time, when the second reference block is also similar to the second data block, the second reference block and the second data block are differentially compressed to obtain a second differential block, which includes metadata for pointing to the second reference block and the difference between the second reference block and the second data block, wherein the second reference block is one of the reference blocks in the first aggregate block, and the second reference block and the second similar block are from the same similar data group.

[0138] 305. Perform differential compression on the new data block and the reference block corresponding to the repeated data block.

[0139] Regarding the description of step 305, Fig.10 The optimized compression scheme for new data shown on the right is similar and will not be described in detail here.

[0140] 306. Direct deduplication processing: for a new data block, metadata pointing to its duplicate data block (such as a reference pointer to the duplicate data block) is retained.

[0141] In a possible implementation, there is a third aggregate block (for example, after the new data blocks are accumulated, they are aggregated into the third aggregate block), and the data content of the third aggregate block is exactly the same as that of the second aggregate block. In this case, deduplication of the third aggregate block and the second aggregate block will not cause read amplification. Therefore, the third aggregate block can be updated, and the updated third aggregate block includes metadata for pointing to the second aggregate block.

[0142] In a possible implementation, there is a fourth aggregate block, the fourth aggregate block includes N data blocks, and the fourth aggregate block includes at least one common data feature with the second aggregate block, and the data content between the fourth aggregate block and the second aggregate block is not completely the same. In other words, the fourth aggregate block is similar to the second aggregate block. In this case, the first aggregate block and the fourth aggregate block can be differentially compressed to obtain an updated fourth aggregate block, and the updated fourth aggregate block includes the difference between the first aggregate block and the fourth aggregate block, and metadata for pointing to duplicate data between the first aggregate block and the fourth aggregate block.

[0143] For ease of understanding, the data compression method in the embodiment of the present application is described below in conjunction with an exemplary scenario. Fig.12 , Fig.12 Schematic diagram of an application scenario of the data compression method in the embodiment of the present application. Fig.12 As shown in FIG. 1 , it is assumed that the capacity of the aggregate block is 2 data blocks, that is, 2 data blocks are aggregated into an aggregate block for differential compression. Fig.12 In the scenario, there are 4 versions of data to be saved.

[0144] For version 1: C1 data block, C2 data block, C3 data block, C4 data block and C5 data block are directly saved as independent data.

[0145] For version 2: C1 data block and C2 data block are duplicates of C1 data block and C2 data block in version 1, respectively, so deduplication is performed. It is only necessary to save the metadata pointing to C1 data block and C2 data block in version 1. For C3' data block, C3' data block is similar to C3 data block in version 1. A new first aggregate block is created and C3 data block is stored in it. A new second aggregate block is created and C3' data block is stored in it. For C4' data block, since it is similar to C4 data block in version 1, C4 data block is stored in the first aggregate block created before, and C4' data block is stored in the second aggregate block created before. At this time, both aggregate blocks (first aggregate block and second aggregate block) have reached capacity, so differential compression is performed on the first aggregate block composed of C3 data block and C4 data block and the second aggregate block composed of C3' data block and C4' data block. C5 data block is duplicated with C5 data block in version 1, so data deduplication is performed.

[0146] For version 3: C1 data block and C2 data block are duplicated with C1 data block and C2 data block in version 1 respectively, so deduplication processing is performed. For C3' data block and C4' data block, although they are duplicated with C3' data block and C4' data block in version 2, deduplication processing can be performed, but if deduplication processing is used, then when restoring C3' data block and C4' data block in version 3, the above-mentioned first aggregate block (C3 data block and C4 data block) needs to be read into the memory for decompression, resulting in a slower decompression speed. Therefore, in the embodiment of the present application, C3' data block and C4' data block will be differentially compressed one-to-one with C3 data block and C4 data block in version 1 respectively, and the differential blocks will be saved. For C5' data block, it will form an aggregate block with the next data block with similar blocks in version 2 for aggregate differential compression.

[0147] For version 4: C3' data block and C4' data block are duplicates of C3' data block and C4' data block in version 3, so data deduplication is performed. C6 data block, C7 data block and C8 data block are independent data blocks and can be directly stored.

[0148] In summary, the aggregate blocks composed of C3 data blocks and C4 data blocks in version 2 and the aggregate blocks composed of C3' data blocks and C4' data blocks are differentially compressed. C3' and C4' in version 3 are duplicates of C3' and C4' in version 2, and can be deduplicated, but this will cause the aggregate blocks to be read into the memory for decompression when restoring the C3' data blocks and C4' data blocks in version 3, resulting in a slower decompression speed. Since the C3' data blocks and C4' data blocks in version 3 are duplicate blocks in the aggregate blocks, we will find the benchmark block of the duplicate block, that is, the C3 data blocks and C4 data blocks in version 1, and perform differential compression and save them. In this way, to restore the C3' data blocks and C4' data blocks in version 3, you only need to read their corresponding benchmark blocks into the memory for decompression, without decompressing the above-mentioned entire first aggregate block (C3 data blocks and C4 data blocks).

[0149] As can be seen from the above, in the embodiment of the present application, multiple data blocks in a single aggregate block come from different storage addresses, and the storage addresses of these data blocks may be adjacent or non-adjacent, which is not limited in the embodiment of the present application. Fig.12 The application scenario shown shows the process of aggregating two adjacent data blocks into an aggregate block and then performing differential compression. Fig.13 , Fig.13 FIG. 1 is a schematic diagram of another application scenario of the data compression method in the embodiment of the present application. Fig.13 In the application scenario shown, a process of aggregating non-adjacent data blocks into aggregate blocks and then performing differential compression is demonstrated. Fig.13 In the application scenario shown, the capacity of the aggregate block is still 2 data blocks, that is, 2 data blocks are aggregated into an aggregate block for differential compression. Fig.13 In the scenario, there are 4 versions of data to be saved.

[0150] For version 1: C1 data block, C2 data block, C3 data block, C4 data block and C5 data block are directly saved as independent data.

[0151] For version 2: C1 data block and C2 data block are duplicated with C1 data block and C2 data block in version 1 respectively, so deduplication is performed. It is only necessary to save metadata pointing to C1 data block and C2 data block in version 1 respectively. C3' data block is similar to C3 data block in version 1. A new first aggregate block is created and C3 data block in version 1 is stored in it. A new second aggregate block is created and C3' data block in version 2 is stored in it. Then the subsequent data blocks are processed. C4 data block is duplicated with C4 data block in version 1, so deduplication is performed. For C5' data block, since it is similar to C5 data block in version 1, C5 data block is stored in the first aggregate block created before, and C5' data block is stored in the second aggregate block created before. At this time, both aggregate blocks (first aggregate block and second aggregate block) have reached capacity, so differential compression is performed on the first aggregate block composed of C3 data block and C5 data block and the second aggregate block composed of C3' data block and C5' data block.

[0152] For version 3: C1 data block and C2 data block are duplicated with C1 data block and C2 data block in version 1 respectively, so deduplication is performed. For C3' data block and C5' data block, although they are duplicated with C3' data block and C5' data block in version 2, deduplication can be performed. However, if deduplication is used, then when restoring C3' data block and C5' data block in version 3, the above-mentioned first aggregate block (C3 data block and C5 data block) needs to be read into memory for decompression, resulting in a slower decompression speed. Therefore, the optimization of converting deduplication blocks to differential blocks is used here. We will perform one-to-one differential compression on C3' data block and C5' data block with C3 data block and C5 data block in version 1 respectively and save the differential blocks. C4 data block can be directly deduplicated with C4 data block in version 1.

[0153] For version 4: C3' data block is duplicated with C3' data block in version 3, so deduplication is performed. C6 data block, C7 data block, C8 data block and C9 data block are independent data blocks and can be directly stored.

[0154] Fig.13 The application scenario shown is similar to Fig.12 Compared with the application scenario shown, the C5 data block and the C3 data block are not adjacent, but the C5 data block and the C3 data block are still aggregated into an aggregate block for differential compression.

[0155] In order to better describe the solution of the embodiment of the present application, accordingly, the embodiment of the present application also provides a related device for implementing the above solution. Fig.14 , Fig.14 A structural diagram of a data compression device provided in an embodiment of the present application. Fig.14As shown, the communication device includes an acquisition unit 401 and a processing unit 402 .

[0156] An acquisition unit 401 is used to acquire N similar data groups, each similar data group includes a reference block and a similar block, the reference block and the similar block in each similar data group include at least one common data feature, and the data content between the reference block and the similar block in each similar data group is not completely the same, and the N similar data groups do not include the same data feature, and N is an integer greater than or equal to 2;

[0157] A processing unit 402 is configured to aggregate the N similar data groups to obtain a first aggregate block and a second aggregate block, wherein the first aggregate block includes the N reference blocks of the N similar data groups, and the second aggregate block includes the N similar blocks in the N similar data groups;

[0158] The processing unit 402 is also used to perform differential compression on the first aggregate block and the second aggregate block to obtain an aggregate differential block, which includes metadata for pointing to duplicate data between the first aggregate block and the second aggregate block, and the difference between the first aggregate block and the second aggregate block.

[0159] In a possible design, the acquisition unit 401 is further configured to acquire a first data block, wherein the data content of the first data block is completely identical to that of a first similar block in the second aggregated block, and the first similar block is one of the similar blocks in the second aggregated block;

[0160] The processing unit 402 is also used to perform differential compression on the first benchmark block and the first data block to obtain a first differential block, wherein the first differential block includes metadata for pointing to duplicate data between the first benchmark block and the first data block, and the difference between the first benchmark block and the first data block, wherein the first benchmark block is one of the benchmark blocks of the first aggregate block, and the first benchmark block and the first similar block are from the same similar data group.

[0161] In a possible design, the acquisition unit 401 is further configured to acquire a second data block, wherein the second data block and a second similar block in the second aggregated block include at least one common data feature, and the data contents of the second data block and the second similar block are not completely the same, and the second similar block is one of the similar blocks in the second aggregated block;

[0162] The processing unit 402 is also used to perform differential compression on the second benchmark block and the second data block to obtain a second differential block, wherein the second differential block includes metadata for pointing to the second benchmark block and the difference between the second benchmark block and the second data block, wherein the second benchmark block is one of the benchmark blocks of the first aggregate block, and the second benchmark block and the second similar block are from the same similar data group.

[0163] In one possible design, the third benchmark block includes metadata of the third data block, the data content between the third data block and the third benchmark block is exactly the same, the third data block is a data block outside the N similar data groups, and the third benchmark block is one of the benchmark blocks of the first aggregate block.

[0164] In a possible design, the acquisition unit 401 is further used to acquire a third aggregate block, where the third aggregate block includes N data blocks, and the data content of the third aggregate block is completely the same as that of the second aggregate block;

[0165] The processing unit 402 is further configured to update the third aggregate block, where the updated third aggregate block includes metadata for pointing to the second aggregate block.

[0166] In a possible design, the acquisition unit 401 is further used to acquire a fourth aggregate block, where the fourth aggregate block includes N data blocks, the fourth aggregate block includes at least one common data feature with the second aggregate block, and the data content of the fourth aggregate block is not completely the same as that of the second aggregate block;

[0167] The processing unit 402 is also used to perform differential compression on the fourth aggregate block according to the first aggregate block to obtain an updated fourth aggregate block, wherein the updated fourth aggregate block includes the difference between the first aggregate block and the fourth aggregate block, and metadata for pointing to duplicate data between the first aggregate block and the fourth aggregate block.

[0168] In one possible design, the processing unit 402 is specifically configured to:

[0169] Aggregating N benchmark blocks in N similar data groups to obtain a first aggregated block;

[0170] N similar blocks in the N similar data groups are aggregated to obtain a second aggregated block.

[0171] In a possible design, the acquisition unit 401 is specifically configured to:

[0172] Obtain M data blocks to be compressed, where data contents of the M data blocks to be compressed are not completely the same, and M is an integer greater than or equal to 2N;

[0173] Similarity detection is performed on the M data blocks to be compressed, and N similar data groups among the M data blocks to be compressed are determined.

[0174] In a possible design, the acquisition unit 401 is specifically configured to:

[0175] Obtain X initial data blocks, where X is an integer greater than or equal to M;

[0176] De-duplication processing is performed on initial data blocks with completely identical data contents among the X initial data blocks to obtain M data blocks to be compressed.

[0177] It should be noted that the information interaction, execution process, etc. between the modules / units in the data compression device are the same as those in the present application. Figure 4 , Figure 7 and Fig.11 The corresponding method embodiments are based on the same concept. For specific contents, please refer to the description in the method embodiments shown above in this application, which will not be repeated here.

[0178] See also Fig.15 , Fig.15 A schematic diagram of a logical structure of a computing device 50 provided in an embodiment of the present application. The computing device 50 may be deployed with Fig.14 The data compression device described in the corresponding embodiment is used to implement Figure 4 , Figure 7 and Fig.11 Corresponding embodiments: The computer device 50 comprises: a memory 501 , a processor 502 , a communication interface 503 and a bus 504 . The memory 501 , the processor 502 , and the communication interface 503 are connected to each other through the bus 504 .

[0179] The memory 501 may be a read only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM). The memory 501 may store a program. When the program stored in the memory 501 is executed by the processor 502, the processor 502 and the communication interface 503 are used to execute steps 101-103 of the above data processing method embodiment.

[0180] The processor 502 may be a central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), a graphics processing unit (GPU), a digital signal processor (DSP), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components or any combination thereof, for executing related programs to implement one or more steps of steps 101-103 of the data processing method embodiment in the present application. The steps of the data processing method disclosed in the embodiment of the present application may be executed by a compiler and an executor, wherein the compiler and the executor may be executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 501, and the processor 502 reads the information in the memory 501, and executes one or more steps of steps 101-103 of the data processing method embodiment in the present application in combination with its hardware.

[0181] The communication interface 503 uses a transceiver device such as but not limited to a transceiver to implement communication between the computer device 50 and other devices or a communication network.

[0182] The bus 504 can realize a path for transmitting information between various components of the computer device 50 (for example, the memory 501, the processor 502, and the communication interface 503). The bus 504 can be a peripheral component interconnect standard (PCI) bus or an extended industry standard architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For convenience of representation, Fig. 9 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0183] It should be further noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the device embodiment drawings provided in the embodiments of the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0184] Through the description of the above implementation mode, the technicians in the field can clearly understand that the embodiments of the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including application-specific integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented with corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the embodiments of the present application, software program implementation is a better implementation method in more cases. Based on such an understanding, the technical solution of the embodiments of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0185] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0186] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. A data compression method, characterized in that: include: Obtaining N similar data groups, each of the similar data groups includes a reference block and a similar block, the reference block and the similar block in each of the similar data groups include at least one common data feature, and the data content between the reference block and the similar block in each of the similar data groups is not completely the same, the N similar data groups do not include the same data feature, and N is an integer greater than or equal to 2; Aggregating the N similar data groups to obtain a first aggregate block and a second aggregate block, wherein the first aggregate block includes N reference blocks of the N similar data groups, and the second aggregate block includes N similar blocks in the N similar data groups; The first aggregate block and the second aggregate block are differentially compressed to obtain an aggregate differential block, wherein the aggregate differential block includes metadata for pointing to duplicate data between the first aggregate block and the second aggregate block, and a difference between the first aggregate block and the second aggregate block.

2. The method according to claim 1, characterized in that The method further comprises: Acquire a first data block, wherein data content of the first data block is completely identical to a first similar block in the second aggregated block, and the first similar block is one of the similar blocks in the second aggregated block; A first benchmark block and the first data block are differentially compressed to obtain a first differential block, wherein the first differential block includes metadata for pointing to duplicate data between the first benchmark block and the first data block, and a difference between the first benchmark block and the first data block, wherein the first benchmark block is one of the benchmark blocks of the first aggregate block, and the first benchmark block and the first similar block are from the same similar data group.

3. The method according to claim 1 or 2, characterized in that: The method further comprises: Acquire a second data block, wherein the second data block and a second similar block in the second aggregated block include at least one common data feature, and the data contents of the second data block and the second similar block are not completely the same, and the second similar block is one of the similar blocks in the second aggregated block; The second benchmark block and the second data block are differentially compressed to obtain a second differential block, wherein the second differential block includes metadata for pointing to the second benchmark block and the difference between the second benchmark block and the second data block, wherein the second benchmark block is one of the benchmark blocks of the first aggregate block, and the second benchmark block and the second similar block are from the same similar data group.

4. The method according to any one of claims 1 to 3, characterized in that: The third benchmark block includes metadata of the third data block, the data content between the third data block and the third benchmark block is exactly the same, the third data block is a data block outside the N similar data groups, and the third benchmark block is one of the benchmark blocks of the first aggregate block.

5. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: Acquire a third aggregate block, where the third aggregate block includes N data blocks, and data content of the third aggregate block is completely the same as that of the second aggregate block; The third aggregate block is updated, where the updated third aggregate block includes metadata for pointing to the second aggregate block.

6. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: Acquire a fourth aggregate block, the fourth aggregate block including N data blocks, the fourth aggregate block including at least one common data feature with the second aggregate block, and data contents of the fourth aggregate block and the second aggregate block are not completely the same; The fourth aggregate block is differentially compressed according to the first aggregate block to obtain an updated fourth aggregate block, wherein the updated fourth aggregate block includes the difference between the first aggregate block and the fourth aggregate block, and metadata for pointing to duplicate data between the first aggregate block and the fourth aggregate block.

7. The method according to any one of claims 1 to 6, characterized in that Aggregating the N similar data groups to obtain a first aggregated block and a second aggregated block includes: Aggregating N reference blocks in the N similar data groups to obtain a first aggregated block; The N similar blocks in the N similar data groups are aggregated to obtain a second aggregated block.

8. The method according to any one of claims 1 to 7, characterized in that The obtaining of N similar data groups comprises: Obtain M data blocks to be compressed, where data contents of the M data blocks to be compressed are not completely the same, and M is an integer greater than or equal to 2N; Similarity detection is performed on the M data blocks to be compressed to determine N similar data groups among the M data blocks to be compressed.

9. The method according to claim 8, characterized in that The step of obtaining M data blocks to be compressed includes: Obtain X initial data blocks, where X is an integer greater than or equal to M; Deduplication processing is performed on initial data blocks with completely identical data contents among the X initial data blocks to obtain M data blocks to be compressed.

10. A data compression device, characterized in that: include: an acquisition unit, configured to acquire N similar data groups, each of the similar data groups comprising a reference block and a similar block, the reference block and the similar block in each of the similar data groups comprising at least one common data feature, and the data content between the reference block and the similar block in each of the similar data groups is not completely the same, the N similar data groups do not comprise common data features, and N is an integer greater than or equal to 2; A processing unit, configured to aggregate the N similar data groups to obtain a first aggregate block and a second aggregate block, wherein the first aggregate block includes the N reference blocks of the N similar data groups, and the second aggregate block includes the N similar blocks in the N similar data groups; The processing unit is further used to perform differential compression on the first aggregate block and the second aggregate block to obtain an aggregate differential block, wherein the aggregate differential block includes metadata for pointing to duplicate data between the first aggregate block and the second aggregate block, and the difference between the first aggregate block and the second aggregate block.

11. The data compression device according to claim 10, characterized in that: The acquisition unit is further configured to acquire a first data block, wherein the data content of the first data block is completely identical to a first similar block in the second aggregated block, and the first similar block is one of the similar blocks in the second aggregated block; The processing unit is also used to perform differential compression on the first benchmark block and the first data block to obtain a first differential block, wherein the first differential block includes metadata for pointing to duplicate data between the first benchmark block and the first data block, and the difference between the first benchmark block and the first data block, wherein the first benchmark block is one of the benchmark blocks of the first aggregate block, and the first benchmark block and the first similar block are from the same similar data group.

12. The data compression device according to claim 10 or 11, characterized in that: The acquisition unit is further configured to acquire a second data block, wherein the second data block and a second similar block in the second aggregated block include at least one common data feature, and the data contents of the second data block and the second similar block are not completely the same, and the second similar block is one of the similar blocks in the second aggregated block; The processing unit is also used to perform differential compression on the second benchmark block and the second data block to obtain a second differential block, wherein the second differential block includes metadata for pointing to the second benchmark block and the difference between the second benchmark block and the second data block, wherein the second benchmark block is one of the benchmark blocks of the first aggregate block, and the second benchmark block and the second similar block are from the same similar data group.

13. The data compression device according to any one of claims 10 to 12, characterized in that: The third benchmark block includes metadata of the third data block, the data content between the third data block and the third benchmark block is exactly the same, the third data block is a data block outside the N similar data groups, and the third benchmark block is one of the benchmark blocks of the first aggregate block.

14. The data compression device according to any one of claims 10 to 13, characterized in that: The acquisition unit is further used to acquire a third aggregate block, the third aggregate block includes N data blocks, and the data content of the third aggregate block is completely the same as that of the second aggregate block; The processing unit is further used to update the third aggregate block, where the updated third aggregate block includes metadata for pointing to the second aggregate block.

15. The data compression device according to any one of claims 10 to 14, characterized in that: The acquisition unit is further used to acquire a fourth aggregate block, the fourth aggregate block includes N data blocks, the fourth aggregate block includes at least one common data feature with the second aggregate block, and the data content of the fourth aggregate block is not completely the same as that of the second aggregate block; The processing unit is further used to perform differential compression on the fourth aggregate block according to the first aggregate block to obtain an updated fourth aggregate block, wherein the updated fourth aggregate block includes the difference between the first aggregate block and the fourth aggregate block, and metadata for pointing to duplicate data between the first aggregate block and the fourth aggregate block.

16. The data compression device according to any one of claims 10 to 15, characterized in that: The processing unit is specifically used for: Aggregating N reference blocks in the N similar data groups to obtain a first aggregated block; The N similar blocks in the N similar data groups are aggregated to obtain a second aggregated block.

17. The data compression device according to any one of claims 10 to 16, characterized in that: The acquisition unit is specifically used for: Obtain M data blocks to be compressed, where data contents of the M data blocks to be compressed are not completely the same, and M is an integer greater than or equal to 2N; Similarity detection is performed on the M data blocks to be compressed to determine N similar data groups among the M data blocks to be compressed.

18. The data compression device according to claim 17, characterized in that: The acquisition unit is specifically used for: Obtain X initial data blocks, where X is an integer greater than or equal to M; Deduplication processing is performed on initial data blocks with completely identical data contents among the X initial data blocks to obtain M data blocks to be compressed.

19. A computing device, characterized in that comprising a processor coupled to a memory, The memory is used to store instructions; The processor is configured to execute instructions in the memory so that the computing device performs the method according to any one of claims 1 to 9.

20. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

21. A computer program product, characterized in that The computer program product stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the method according to any one of claims 1 to 9 is implemented.

22. A chip system, characterized in that: The chip system includes at least one processor, and when program instructions are executed in the at least one processor, the method according to any one of claims 1 to 9 is executed.

Images