Data processing method and related device

Abstract

Embodiments of the present application disclose a data processing method and related equipment, which are used to reduce the calculation cost while ensuring the accuracy and compression rate of feature calculation. The method comprises: determining a plurality of first feature values corresponding to each data block, each first feature value representing the feature of part of the data content contained in the data block; dividing the plurality of first feature values corresponding to each data block into a plurality of content clusters according to the part of the data content corresponding to each first feature value, the number of content clusters corresponding to each data block being equal to the preset number of features; performing a linear transformation operation on each first feature value to obtain a second feature value corresponding to each first feature value; determining the second feature value with the smallest value corresponding to each content cluster as the target feature value of the data block corresponding to each content cluster; and determining the similar data block corresponding to each data block according to the plurality of target feature values corresponding to each data block.

Description

Technical Field

[0001] This application relates to the field of differential compression, and more particularly to data processing methods and related equipment. Background Technology

[0002] Currently, mainstream similarity detection techniques for differential compression primarily search for similar data blocks by generating data block summaries in the form of features or super features. This involves generating one or more data block summaries for each data block to represent it, and storing these summaries in an index. Similar data blocks are then searched for in the index by matching the data block summaries. Using data block summaries to represent data blocks reduces the memory overhead of the index and also reduces the computational overhead of finding similar blocks in the system.

[0003] Generally, the process begins by calculating multiple rolling hash values ​​for each data block. Then, each rolling hash value undergoes N (i.e., multiple) linear transformations to obtain N hash values ​​for each data block. Next, these N hash values ​​are assembled to obtain a predetermined number of super features, which serve as the feature values ​​for each data block. Finally, based on the super features for each data block, similar data blocks are determined.

[0004] However, the above calculation method requires multiple hash transformations on a large number of rolling hash values, which consumes a lot of computing resources and reduces the efficiency of differential compression. Summary of the Invention

[0005] This application provides a data processing method and related equipment to ensure the accuracy and compression rate of feature calculation while reducing computing costs.

[0006] The first aspect of this application provides a data processing method, including:

[0007] Determine multiple first feature values ​​corresponding to each data block, wherein each first feature value corresponding to each data block represents a feature of a portion of the data content contained in each data block;

[0008] Based on the partial data content corresponding to each first feature value, the multiple first feature values ​​corresponding to each data block are divided into multiple content clusters, and the number of content clusters corresponding to each data block is equal to the preset number of features.

[0009] Perform a linear transformation operation on each of the first feature values ​​to obtain a second feature value corresponding to each of the first feature values;

[0010] In each content cluster, the second feature value with the smallest value among the second feature values ​​corresponding to each first feature value is determined as the target feature value of the data block corresponding to each content cluster, and the number of target feature values ​​corresponding to each data block is equal to the preset number of features;

[0011] Based on multiple target feature values ​​corresponding to each data block, similar data blocks are determined for each data block.

[0012] In one specific implementation, determining the multiple first feature values ​​corresponding to each data block includes:

[0013] Based on the data content contained in each data block and the preset rolling hash algorithm, calculate multiple third feature values ​​corresponding to each data block;

[0014] Obtain the sampling mask based on the preset sampling rate;

[0015] Perform a preset logical operation on each third feature value corresponding to each data block and the sampling mask;

[0016] Based on the execution result corresponding to the preset logical operation, the plurality of first feature values ​​corresponding to each data block are determined from each third feature value corresponding to each data block.

[0017] In one specific implementation, determining the plurality of first feature values ​​corresponding to each data block from each third feature value corresponding to each data block based on the execution result corresponding to the preset logical operation includes:

[0018] If any execution result satisfies the preset execution result, then the third feature value corresponding to any execution result is determined to be the first feature value of the corresponding data block;

[0019] If any execution result does not meet the preset execution result, the operation of determining the third feature value corresponding to any execution result as the first feature value of the corresponding data block will not be performed.

[0020] In one specific implementation, the step of dividing the multiple first feature values ​​corresponding to each data block into multiple content clusters based on the partial data content corresponding to each first feature value includes:

[0021] The suffix of the data content corresponding to each first feature value is determined as the data content suffix corresponding to each first feature value, and the length of the data content suffix is ​​equal to the preset suffix length;

[0022] According to the preset suffix length and the preset number of features corresponding to each data block, multiple suffix value ranges are determined, and the number of suffix value ranges is equal to the preset number of features.

[0023] Based on the range of values ​​for the multiple suffixes and the data content suffixes corresponding to each first feature value, the multiple first feature values ​​corresponding to each data block are divided into multiple content clusters corresponding to each data block, and each content cluster corresponds to a different range of values ​​for the suffixes.

[0024] In one specific implementation, the step of dividing the multiple first feature values ​​corresponding to each data block into multiple content clusters corresponding to each data block according to the range of multiple suffix values ​​and the data content suffix corresponding to each first feature value includes:

[0025] Each data block is sequentially identified as the fourth data block, and each first feature value corresponding to the fourth data block is sequentially identified as the fourth feature value.

[0026] Determine the fourth data content suffix corresponding to the fourth feature value, and determine the suffix value range corresponding to the fourth data content suffix;

[0027] If the suffix value range corresponding to any content cluster of the fourth data block is the same as the suffix value range corresponding to the fourth data content suffix, then the fourth feature value is assigned to any content cluster.

[0028] In one specific implementation, determining the similar data blocks corresponding to each data block based on multiple target feature values ​​corresponding to each data block includes:

[0029] Each data block is determined as the current data block in turn, and each target feature value corresponding to the current data block is determined as the current feature value in turn.

[0030] If any target feature value corresponding to any other data block is consistent with the current feature value, then the other data block is determined as a candidate similar block of the current data block, and the other data block is a data block other than the current data block among multiple data blocks;

[0031] Based on the target feature value corresponding to the current data block and the target feature value corresponding to each candidate similar block, determine the number of similar features between each candidate similar block and the current data block;

[0032] The candidate similar block with the most corresponding similar features is determined as the similar data block of the current data block.

[0033] In one specific implementation, after determining the similar data blocks corresponding to each data block based on multiple target feature values ​​corresponding to each data block, the method further includes:

[0034] Differential compression is performed on each data block based on the similar data blocks corresponding to each data block.

[0035] A second aspect of this application provides a data processing apparatus, comprising:

[0036] A determining unit is used to determine multiple first feature values ​​corresponding to each data block, wherein each first feature value corresponding to each data block represents a feature of a portion of the data content contained in each data block;

[0037] The clustering unit is used to divide multiple first feature values ​​corresponding to each data block into multiple content clusters according to the partial data content corresponding to each first feature value. The number of content clusters corresponding to each data block is equal to the preset number of features.

[0038] An execution unit is configured to perform a linear transformation operation on each of the first feature values ​​to obtain a second feature value corresponding to each of the first feature values.

[0039] The determining unit is further configured to determine the second feature value with the smallest value among the second feature values ​​corresponding to each first feature value in each content cluster as the target feature value of the data block corresponding to each content cluster, wherein the number of target feature values ​​corresponding to each data block is equal to the preset number of features;

[0040] The determining unit is further configured to determine similar data blocks corresponding to each data block based on multiple target feature values ​​corresponding to each data block.

[0041] In one specific implementation, the determining unit is specifically used to calculate multiple third feature values ​​corresponding to each data block based on the data content contained in each data block and a preset rolling hash algorithm;

[0042] Obtain the sampling mask based on the preset sampling rate;

[0043] Perform a preset logical operation on each third feature value corresponding to each data block and the sampling mask;

[0044] Based on the execution result corresponding to the preset logical operation, the plurality of first feature values ​​corresponding to each data block are determined from each third feature value corresponding to each data block.

[0045] In one specific implementation, the determining unit is specifically used to determine the third feature value corresponding to any execution result as the first feature value of the corresponding data block if any execution result satisfies the preset execution result.

[0046] If any execution result does not meet the preset execution result, the operation of determining the third feature value corresponding to any execution result as the first feature value of the corresponding data block will not be performed.

[0047] In one specific implementation, the clustering unit is specifically used to determine the suffix of the data content corresponding to each first feature value as the data content suffix corresponding to each first feature value, wherein the length of the data content suffix is ​​equal to a preset suffix length.

[0048] According to the preset suffix length and the preset number of features corresponding to each data block, multiple suffix value ranges are determined, and the number of suffix value ranges is equal to the preset number of features.

[0049] Based on the range of values ​​for the multiple suffixes and the data content suffixes corresponding to each first feature value, the multiple first feature values ​​corresponding to each data block are divided into multiple content clusters corresponding to each data block, and each content cluster corresponds to a different range of values ​​for the suffixes.

[0050] In one specific implementation, the clustering unit is specifically used to sequentially determine each data block as the fourth data block, and sequentially determine each first feature value corresponding to the fourth data block as the fourth feature value;

[0051] Determine the fourth data content suffix corresponding to the fourth feature value, and determine the suffix value range corresponding to the fourth data content suffix;

[0052] If the suffix value range corresponding to any content cluster of the fourth data block is the same as the suffix value range corresponding to the fourth data content suffix, then the fourth feature value is assigned to any content cluster.

[0053] In one specific implementation, the determining unit is specifically used to sequentially determine each data block as the current data block, and sequentially determine each target feature value corresponding to the current data block as the current feature value;

[0054] If any target feature value corresponding to any other data block is consistent with the current feature value, then the other data block is determined as a candidate similar block of the current data block, and the other data block is a data block other than the current data block among multiple data blocks;

[0055] Based on the target feature value corresponding to the current data block and the target feature value corresponding to each candidate similar block, determine the number of similar features between each candidate similar block and the current data block;

[0056] The candidate similar block with the most corresponding similar features is determined as the similar data block of the current data block.

[0057] In one specific implementation, after determining the similar data blocks corresponding to each data block based on multiple target feature values ​​corresponding to each data block, the device further includes: a compression unit;

[0058] The compression unit is used to perform differential compression operation on each data block according to the similar data blocks corresponding to each data block.

[0059] A third aspect of this application provides a data processing apparatus, comprising:

[0060] Central processing unit, memory, and input / output interfaces;

[0061] The memory is either a short-term storage memory or a persistent storage memory;

[0062] The central processing unit is configured to communicate with the memory and execute instructions in the memory to perform the method described in the first aspect.

[0063] A fourth aspect of this application provides a computer program product containing instructions that, when run on a computer, cause the computer to perform the method described in the first aspect.

[0064] A fifth aspect of this application provides a computer storage medium storing instructions that, when executed on a computer, cause the computer to perform the method described in the first aspect.

[0065] As can be seen from the above technical solutions, the embodiments of this application have the following advantages: Multiple first feature values ​​are determined for each data block, each first feature value representing a feature of a portion of the data content contained in each data block; based on the portion of data content corresponding to each first feature value, the multiple first feature values ​​corresponding to each data block are divided into multiple content clusters, the number of content clusters corresponding to each data block being equal to the preset number of features; a linear transformation operation is performed on each first feature value to obtain a second feature value corresponding to each first feature value; the second feature value with the smallest value among the second feature values ​​corresponding to each first feature value in each content cluster is determined as the target feature value of the data block corresponding to each content cluster, the number of target feature values ​​corresponding to each data block being equal to the preset number of features; based on the multiple target feature values ​​corresponding to each data block, similar data blocks corresponding to each data block are determined. Through content clustering and the design of performing only one linear transformation on each first feature value, the efficiency of feature calculation is improved while ensuring the accuracy of feature calculation, and experiments have verified that it does not affect the similarity detection accuracy or the compression ratio of differential compression. Attached Figure Description

[0066] Figure 1 This is an overall framework diagram of the deduplication and compression system disclosed in the embodiments of this application;

[0067] Figure 2 This is a schematic flowchart of a data processing method disclosed in an embodiment of this application;

[0068] Figure 3 This is a schematic diagram of meaningless features disclosed in the embodiments of this application;

[0069] Figure 4 This is another schematic flowchart of the data processing method disclosed in the embodiments of this application;

[0070] Figure 5 This is a schematic diagram of the data processing device disclosed in an embodiment of this application;

[0071] Figure 6 This is another structural schematic diagram of the data processing device disclosed in the embodiments of this application. Detailed Implementation

[0072] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0073] Please see Figure 1 The overall framework of the deduplication and compression system is as follows: Figure 1As shown. Traditional similarity detection techniques mainly have two problems: (1) Existing similarity detection methods cannot achieve both high computational speed and high compression ratio at the same time. Traditional similarity detection methods use the Locality Sensitive Hashing (LSH) algorithm to calculate N features for a data block and merge several features into a super feature, which is then used to detect similar data blocks in the system. This method can achieve high detection accuracy and high differential compression ratio, but the computational overhead is large, and the computational overhead becomes the performance bottleneck of the system. On some datasets, the system throughput is only 60MB / s. Other similarity detection methods improve the speed of LSH-based similarity detection methods and accelerate the feature calculation process, but they all inevitably reduce the overall compression ratio. The similarity detection method called Finesse divides the data block into sub-blocks and extracts features from each sub-block to reduce the computational complexity in the feature calculation process. However, this method is affected by the "boundary offset" problem, which in turn affects the accuracy and compression ratio of similarity detection. Another method called Odess accelerates the feature calculation process by using a content sampling method. While this method is not affected by the "boundary offset" problem, it generates multiple features at the same location within the data block. Since these features are essentially the same, the effectiveness of the features is reduced, which in turn affects the compression ratio.

[0074] (2) Existing similarity detection methods can only detect data blocks with high similarity. Compared with the method of brute-force search for the most similar data blocks in the storage system, traditional similarity detection methods have failed to detect more than half of the similar data blocks in some datasets, and can only detect data blocks with similarity greater than 80%. The number of data blocks with similarity less than 80% is small, which affects the compression rate of the system.

[0075] To address the aforementioned deficiencies in the prior art, this application provides a data processing method that reduces computational costs while maintaining the accuracy and compression rate of feature calculations.

[0076] Please see Figure 2 This application provides a data processing method, including the following steps:

[0077] 201. Determine multiple first feature values ​​corresponding to each data block. Each first feature value corresponding to each data block represents the feature of the partial data content contained in each data block.

[0078] according to Figure 1As shown in the overall framework of the deduplication compression system, the first step in performing differential compression of data blocks is to calculate the feature value corresponding to each data block. Generally, to ensure the accuracy of feature calculation and subsequent similarity detection, multiple feature values ​​are calculated for each data block to fully represent the data characteristics of different parts of the data in each block. Any two first feature values ​​corresponding to each data block represent a portion of the data content of that data block, but the portions they reflect are not entirely the same.

[0079] 202. Based on the partial data content corresponding to each first feature value, divide the multiple first feature values ​​corresponding to each data block into multiple content clusters. The number of content clusters corresponding to each data block is equal to the preset number of features.

[0080] In existing technical solutions, meaningless features may appear during feature calculation, affecting feature effectiveness. Meaningless features refer to features where multiple features (more than one) represent the same sub-block within the N primary features corresponding to a data block. For example... Figure 3 As shown in the figure, there are 12 features, of which 5 features are meaningless: f_4, f_8, f_6, and f_10. The sub-blocks represented by f_4 and f_8 are the same as f_1, and the sub-blocks represented by f_6 and f_10 are the same as f_3; therefore, these 5 features are meaningless.

[0081] Such meaningless features reduce the effectiveness of features. In an extreme case, if all 12 features represent the same sub-block, then these 12 features are essentially identical, failing to accurately represent the characteristics of the entire data block and thus unable to measure the similarity between data blocks. If two data blocks contain 11 meaningless features and their only shared feature (i.e., all 12 features are the same), traditional similarity detection methods would classify them as highly similar. However, in reality, they only share one sub-block and are not necessarily very similar, leading to false similarity detection and the incorrect detection of dissimilar blocks. Therefore, this application's embodiment, through this content clustering-based scheme, modifies the feature selection method to fundamentally solve the problem of meaningless features in the prior art.

[0082] Specifically, for each data block corresponding to multiple first feature values, based on the partial data content identified by each first feature value, the first feature values ​​whose corresponding partial data content is close to each other are assigned to a set of content clusters corresponding to the aforementioned data block. Meanwhile, considering subsequent feature selection, the number of content clusters corresponding to each data block needs to be equal to the preset number of features.

[0083] 203. Perform a linear transformation operation on each first eigenvalue to obtain the second eigenvalue corresponding to each first eigenvalue.

[0084] Unlike existing techniques that require N feature values ​​and thus N feature transformations, this step performs only one linear transformation operation on each first feature value. It's important to note that, to ensure the reliability of step 204, the multiple first feature values ​​contained in each content cluster will be sampled using the same linear transformation method (or linear transformation formula), while different content clusters will use different linear transformation methods (or linear transformation formulas). However, each first feature value undergoes only a sequential linear transformation to obtain a corresponding second feature value.

[0085] In this step, a feature value needs to be calculated from each cluster. First, a linear transformation is performed on the hash value in each cluster, where the linear transformation performed on the i-th cluster is shown in Equation 3.

[0086] T i =(m i *x+a i (Formula 3)

[0087] Where m i and a i These are predefined parameters, where x is the Gear hash value. After performing a linear transformation on all hash values ​​in a cluster, the minimum value after the linear transformation is selected as a feature. With N clusters, this results in N features, where N is the preset number of features. N can be determined based on factors including, but not limited to, the device's memory size and computing power. Furthermore, because the number of features affects the index size and computation speed, factors influencing the choice of N also include the user's required speed for similarity detection.

[0088] 204. Among the second feature values ​​corresponding to each first feature value in each content cluster, the second feature value with the smallest value is determined as the target feature value of the data block corresponding to each content cluster. The number of target feature values ​​corresponding to each data block is equal to the preset number of features.

[0089] Specifically, to avoid extensive computation, feature selection needs to be performed based on each first feature value corresponding to each data block obtained from the aforementioned steps. Since each first feature value corresponds to a second feature value, a target feature value for each data block can be determined using each second feature value corresponding to each content cluster (i.e., the second feature value corresponding to each first feature value contained within that content cluster). Generally, the second feature value with the smallest value among the multiple second feature values ​​corresponding to each content cluster can be determined as a target feature value for the data block corresponding to that content cluster. This process is repeated until a target feature value is determined for each content cluster corresponding to each data block. This method of selecting a target feature value based on the content cluster corresponding to a data block ensures that the selected target feature value for each data block identifies different parts of the data within that data block, effectively avoiding the generation of meaningless features.

[0090] This step has the following advantages: (1) It avoids generating meaningless features. Because the clustering operation divides the first feature value into clusters according to its corresponding partial data content, and then generates features from each cluster, meaningless features are not generated because different clusters are mutually exclusive. (2) It shuffles the first feature value once, which reduces the standard deviation and variance of the similarity estimation. (3) It reduces the number of linear transformations for each first feature value to once, reducing the complexity of the algorithm and speeding up the feature generation.

[0091] 205. Based on the multiple target feature values ​​corresponding to each data block, determine the similar data blocks corresponding to each data block.

[0092] One of the final parts of performing deduplication compression is determining the target feature values ​​for each data block. Therefore, after determining multiple target feature values ​​for each data block, similar data blocks can be identified based on these values. Specifically, there are various methods for detecting similar data blocks, including but not limited to first-match algorithms and local best-match algorithms; these are not specifically limited here.

[0093] Furthermore, after determining the similar data blocks corresponding to each data block, this embodiment of the application further includes the following step: performing differential compression on each data block based on the similar data blocks corresponding to each data block. Specifically, please refer to... Figure 1 The process involves first eliminating identical blocks; then, based on the corresponding similar data blocks, eliminating the identical parts of each data block and its similar data blocks, or the difference data of each data block relative to its similar data blocks; finally, the difference data corresponding to the data blocks are packaged and compressed in sequence.

[0094] In this embodiment, by using content clustering and performing a linear transformation only once for each first feature value, the efficiency of feature calculation is improved while ensuring the accuracy of feature calculation. Furthermore, experiments have verified that this does not affect the similarity detection accuracy or the compression ratio of differential compression.

[0095] Based on the foregoing embodiments, in some specific implementations, step 201 can be implemented through the following steps: calculating multiple third feature values ​​corresponding to each data block based on the data content contained in each data block and a preset rolling hash algorithm; obtaining a sampling mask determined based on a preset sampling rate; performing a preset logical operation on each third feature value and the sampling mask corresponding to each data block; and determining multiple first feature values ​​corresponding to each data block from each third feature value corresponding to each data block according to the execution result corresponding to the preset logical operation.

[0096] Specifically, step 201 involves feature calculation and sampling. We can use a preset rolling hash algorithm to generate a set of hash values ​​for each data block, including each of its third feature values. Then, we sample the hash value set based on the content of the hash values ​​to obtain multiple first feature values ​​corresponding to each data block. Furthermore, according to the content-based sampling algorithm design, the sampling rate can be controlled by designing the sampling mask. The sampling rate is equal to the number of first feature values ​​for each data block divided by the number of third feature values ​​for each data block.

[0097] In practical applications, to further reduce the cost of feature calculation, Gear rolling hash can be used as the preset hash value algorithm, as detailed below.

[0098] Generally, traditional similar data detection techniques use Rabin rolling hashing for feature calculation. The Rabin rolling hashing calculation method is shown in Equation 1.

[0099]

[0100] Among them FP i It refers to the calculated i-th Rabin hash value, also known as the Rabin fingerprint, FP i-1 This refers to the (i-1)th Rabin fingerprint, where U() and T() are two pre-defined arrays, a is the i-th character of the input data block, and b refers to the content of the input data block. This method involves two XOR operations, two shift operations, one OR operation, and two memory access operations. To obtain all features of the entire input data block, it is necessary to perform byte-by-byte sliding fingerprint calculation on the data block. However, the overhead of each Rabin fingerprint calculation is significant, making the fingerprint calculation very expensive. This scheme uses the faster Gear rolling hash, whose calculation method is shown in Equation 2.

[0101] FP i =(FP) i-1 <<1)+G(b) (Equation 2)

[0102] Gear rolling hashing requires only one shift, one summation, and one memory access operation, with a computational overhead of only half that of Rabin rolling hashing, which can greatly save the time cost of feature computation.

[0103] Furthermore, after determining the third eigenvalue, a sampling operation is required.

[0104] Gear rolling hashing calculates a set of hash values ​​for each data block (i.e., each second feature value corresponding to the data block). Traditional similarity detection methods perform a linear transformation on each hash value in the hash value set, which incurs significant feature selection overhead and affects the speed of similarity detection. To reduce the computational overhead of feature selection (i.e., the operation performed in step 204 above), this scheme utilizes a content sampling-based algorithm to reduce the number of hash values ​​contained in the hash value set of each data block, while maintaining the ability of the sampled hash values ​​to represent the characteristics of the data block. The content sampling algorithm samples hash values ​​that satisfy certain specific conditions.

[0105] The following example illustrates that if a hash value is ANDed with a specific value and the result is 0, then the sampling condition is met. This specific value is called the mask (i.e., the sampling mask). The sampling rate is determined by the value of the mask. The number of 1s in the binary mask determines the sampling rate. Assuming both the mask and the hash value are unsigned 64-bit integers, and the mask contains 5 1s, then the sampling rate is... If the mask contains 10 ones, then the sampling rate is... However, content sampling algorithms introduce problems for subsequent feature selection. The sampling operation significantly reduces the number of hash values ​​used for linear transformation, leading to the emergence of meaningless features. For details on how to address the emergence of meaningless features, please refer to the relevant implementation examples for feature selection.

[0106] In some specific implementations, if any execution result satisfies a preset execution result, then the third feature value corresponding to any execution result is determined as the first feature value of the corresponding data block; if any execution result does not satisfy the preset execution result, then the operation of determining the third feature value corresponding to any execution result as the first feature value of the corresponding data block is not performed. Continuing with the previous example, if the hash value (third feature value) and the sampling mask are ANDed to 0, it indicates that the sampling condition is met, that is, the third feature value needs to be determined as the first feature value of the corresponding data block; otherwise, the third feature value is discarded, that is, the operation of determining the third feature value as the first feature value of the corresponding data block is not performed.

[0107] Based on the aforementioned embodiments, the content clustering operation in step 202 can be implemented by referring to the following steps: The suffix of the data content corresponding to each first feature value is determined as the data content suffix corresponding to each first feature value, and the length of the data content suffix is ​​equal to a preset suffix length; Multiple suffix value ranges are determined according to the preset suffix length and the preset number of features corresponding to each data block, and the number of suffix value ranges is equal to the preset number of features; Multiple first feature values ​​corresponding to each data block are divided into multiple content clusters corresponding to each data block according to the multiple suffix value ranges and the data content suffix corresponding to each first feature value, and each content cluster corresponds to a different suffix value range.

[0108] Considering that the suffix of the data content corresponding to each feature value can represent the corresponding part of the data content, we first determine the data content suffix corresponding to each first feature value, and then perform content clustering based on the corresponding data content suffix. Specifically, the partitioning rules are predefined. For example: the preset number of features is three (this can be set according to requirements and actual application scenarios), and the preset suffix length is 4 bits. Then, the data content suffix (corresponding decimal) values ​​0-4 are classified as the first suffix value range, the data content suffix (corresponding decimal) values ​​5-9 are classified as the second suffix value range, and the data content suffix (corresponding decimal) values ​​10-15 are classified as the last suffix value range. As can be seen from the above, the third suffix value range corresponds to more (decimal) values, i.e., a larger range. Experiments show that this slight imbalance will not significantly affect the final differential compression result. Therefore, to further ensure the balance of clustering, the preset number of features can be adjusted to 4 to ensure the balance of each suffix value range. This section does not specify the exact method for determining the range of values ​​for multiple suffixes.

[0109] It should be noted that the data content-based clustering technology provided in this application is not limited to using the suffix of the data content corresponding to each feature value, but can also be the prefix of the corresponding data content. After changing the determination condition for content-based clustering from the suffix to the prefix, other implementation methods are similar to the suffix scheme, and will not be described again here. Furthermore, to ensure that the suffix of the data content corresponding to each first feature value is different, the preset suffix length should not be less than the length of each sliding window.

[0110] Content-based clustering offers three advantages: (1) It avoids generating meaningless features. Because the clustering operation categorizes hash values ​​according to their content and generates features from each category, meaningless features are avoided since different categories are mutually exclusive. (2) It shuffles the set of hash values, which reduces the standard deviation and variance of similarity estimation. Understandably, content clustering involves placing the first feature values ​​of corresponding data with similar content into a cluster, resulting in a smaller conditional variance within a cluster, thus reducing the estimated variance. This is similar to stratified sampling. (3) It reduces the number of linear transformations performed on each hash value to once, lowering the algorithm's complexity and accelerating feature generation.

[0111] Furthermore, the aforementioned step of dividing the multiple first feature values ​​corresponding to each data block into multiple content clusters corresponding to each data block according to the multiple suffix value ranges and the data content suffix corresponding to each first feature value can be implemented in the following way: each data block is determined as the fourth data block in turn, and each first feature value corresponding to the fourth data block is determined as the fourth feature value in turn; the fourth data content suffix corresponding to the fourth feature value is determined, and the suffix value range corresponding to the fourth data content suffix is ​​determined; if the suffix value range corresponding to any content cluster corresponding to the fourth data block is the same as the suffix value range corresponding to the fourth data content suffix, then the fourth feature value is assigned to any content cluster.

[0112] Specifically, in addition to pre-constructing multiple content clusters corresponding to each data block and determining the suffix value range corresponding to each content cluster, each first feature value is then placed into the corresponding content cluster. Alternatively, each first feature value can be assigned to a corresponding content cluster separately, as described in the above embodiment. It should be noted that the description of the first feature value as the fourth feature value and the way the data content suffix corresponding to the first feature value is described as the fourth data content suffix corresponding to the fourth feature value are merely descriptive modifications for differentiation, and the above steps are actually executed. In other words, the above clustering method is actually based on each first feature value and its corresponding data content suffix.

[0113] Generally, traditional similarity detection methods divide N features into M groups of T features each, where M*T = N. Then, each group's features are hashed using XXHash to obtain a super feature, as shown in Equation 4.

[0114] SuperFeature x =XXHash(Feature) x*t ,…,Feature x*t+t-1 (Formula 3)

[0115] For example, if 12 features are generated and 3 super features need to be generated, then features 1-4 are bundled together to calculate the first super feature, features 5-8 are bundled together to calculate the second super feature, and features 9-12 are calculated to calculate the third super feature. If two data blocks have the same super feature, then the probability that the two data blocks are similar is high. According to Broder's theorem, if two data blocks have more identical feature values, it indicates that the content of the two data blocks is more similar.

[0116] It is worth noting that traditional similarity detection can generate multiple super feature values. A super feature is composed of multiple feature values. In traditional similarity detection methods, if any two data blocks have the same super feature value, the two data blocks are considered similar.

[0117] However, experimental results show that traditional similarity detection methods can only detect highly similar data blocks, while missing many other low-similarity data blocks that can be compressed. The false positive rate of 64-bit Gear fingerprints is often very low; if two data blocks have the same Gear hash value, they are likely to have the same sliding window content (32 or 64 bytes). Therefore, continuously increasing the number of features included in each super-feature may actually reduce the efficiency of similarity detection and miss similar blocks. While a larger number of super-features results in more detected similar blocks, the overhead of indexing and feature computation also increases. Therefore, to minimize memory overhead, this invention uses the features selected in feature selection (i.e., the target feature value corresponding to each data block) as the features used for similarity detection, called naive features, which eliminates the need to perform super-feature computation.

[0118] Traditional similarity detection methods use a first-match algorithm to match similar data blocks. That is, when the algorithm detects that the current data block shares any super-feature with another data block, it considers the two data blocks similar, then stops detecting similar data blocks for the current block and continues calculating similar data blocks for the next data block. Considering that traditional similarity detection methods use super-features to match similar blocks, super-features often match highly similar data blocks, so the first-match algorithm can efficiently match highly similar data blocks. However, traditional similarity detection techniques can only detect a portion of highly similar data blocks. To further improve the compression ratio, this embodiment designs a set of fine-grained naive features and employs an optimal matching strategy to detect the most similar data blocks possible.

[0119] Specifically, the optimal matching strategy selects the data block with the most matching naive features as the similar data block of the current data block. First, multiple candidate similar blocks are detected through multiple naive features (i.e., multiple target feature values ​​corresponding to the current data block). Then, this scheme designs a set of fine-grained naive feature blocks to find the block with the most matching features as the final similar block, and performs differential compression with the current data block.

[0120] In some specific implementations, based on the aforementioned embodiments, the similar data block detection in step 205 can be implemented by referring to the following steps: Each data block is sequentially identified as the current data block, and each target feature value corresponding to the current data block is sequentially identified as the current feature value; if any target feature value corresponding to any other data block is consistent with the current feature value, then any other data block is identified as a candidate similar block for the current data block, where other data blocks are data blocks other than the current data block among multiple data blocks; based on the target feature value corresponding to the current data block and the target feature value corresponding to each candidate similar block, the number of similar features between each candidate similar block and the current data block is determined; the candidate similar block with the largest number of similar features is identified as a similar data block of the current data block.

[0121] by Figure 4 For example, in the fourth step of the matching process, firstly, candidate similar blocks are found as candidates by using multiple target feature values ​​corresponding to the current data block. Then, each target feature value of the current data block is compared with the target feature value of each candidate similar block one by one. The second data block is found to be the data block with the most similar features. Therefore, the second data block is taken as the final similar block of the target data block and differential compression is performed.

[0122] The advantages of the method in this application compared to traditional differential compression-based similar data detection techniques are analyzed from two aspects: 1. Computational overhead: N-Transform requires N linear transformations to generate features, selecting the minimum value after the linear transformation as the feature. Its computational complexity for feature generation is O(N*L), where L is the length of the data block. This invention uses content sampling to greatly reduce the number of hash values ​​for linear transformations and employs a content-based clustering algorithm, performing one linear transformation per cluster. Assuming a sampling rate of 1 / r, the computational complexity for feature generation in this invention is O(1 / r*L). This invention achieves the lowest computational complexity. 2. Memory overhead: The number of naive features in this invention is consistent with the number of super features in traditional similarity detection methods. Therefore, the memory overhead of this invention is consistent with that of traditional similarity detection methods.

[0123] In summary, addressing the challenge of achieving both high computational speed and high compression ratio in traditional differential compression-based similarity data detection techniques, this solution employs content-based sampling and content-based clustering strategies. This effectively reduces the overhead of feature computation and significantly improves the accuracy of similarity detection. Furthermore, to address the limitation of traditional similarity detection techniques that can only detect a subset of highly similar data blocks, this solution designs a set of fine-grained naive features, greatly increasing the number of similar blocks detected and achieving a higher compression ratio. In addition, a local optimal matching strategy matches each data block with the most similar data block possible, further enhancing the compression ratio.

[0124] Verification experiments were conducted using real-world datasets containing data blocks with varying degrees of similarity. These datasets represent various typical workloads in storage systems, including source code projects, database key-value pair backups, virtual machine snapshots, system backups, and database images. These datasets have been used in multiple data compression projects. Experimental results are as follows: Compared to traditional similarity detection schemes, this approach achieves a 25%-124% improvement in differential compression ratio. Throughput tests for feature generation show that this approach is 19.33-20.1 times faster than traditional similarity detection schemes. Throughput tests for the deduplication compression system show that this approach is 1.72 times faster than traditional similarity detection schemes.

[0125] Please see Figure 5 This application provides a data processing device, including:

[0126] The determining unit 501 is used to determine multiple first feature values ​​corresponding to each data block, where each first feature value corresponding to each data block represents the feature of a portion of the data content contained in each data block.

[0127] Clustering unit 502 is used to divide multiple first feature values ​​corresponding to each data block into multiple content clusters according to the partial data content corresponding to each first feature value. The number of content clusters corresponding to each data block is equal to the preset number of features.

[0128] The execution unit 503 is used to perform a linear transformation operation on each first feature value to obtain a second feature value corresponding to each first feature value.

[0129] The determining unit 501 is further configured to determine the second feature value with the smallest value among the second feature values ​​corresponding to each first feature value in each content cluster as the target feature value of the data block corresponding to each content cluster, and the number of target feature values ​​corresponding to each data block is equal to the preset number of features;

[0130] The determining unit 501 is also used to determine the similar data blocks corresponding to each data block based on the multiple target feature values ​​corresponding to each data block.

[0131] In one specific implementation, the determining unit 501 is specifically used to calculate multiple third feature values ​​corresponding to each data block based on the data content contained in each data block and a preset rolling hash algorithm;

[0132] Obtain the sampling mask based on the preset sampling rate;

[0133] Perform a preset logical operation on each third feature value and sampling mask corresponding to each data block;

[0134] Based on the execution result corresponding to the preset logical operation, determine multiple first feature values ​​corresponding to each data block from each third feature value corresponding to each data block.

[0135] In one specific implementation, the determining unit 501 is specifically used to determine the third feature value corresponding to any execution result as the first feature value of the corresponding data block if any execution result satisfies the preset execution result.

[0136] If any execution result does not meet the preset execution result, the operation of determining the third characteristic value corresponding to any execution result as the first characteristic value of the corresponding data block will not be executed.

[0137] In one specific implementation, the clustering unit 502 is specifically used to determine the suffix of the data content corresponding to each first feature value as the data content suffix corresponding to each first feature value, and the length of the data content suffix is ​​equal to the preset suffix length.

[0138] Based on the preset suffix length and the preset number of features corresponding to each data block, multiple suffix value ranges are determined, and the number of suffix value ranges is equal to the preset number of features.

[0139] Based on the range of multiple suffix values ​​and the data content suffix corresponding to each first feature value, the multiple first feature values ​​corresponding to each data block are divided into multiple content clusters corresponding to each data block, and each content cluster corresponds to a different range of suffix values.

[0140] In one specific implementation, the clustering unit 502 is specifically used to sequentially determine each data block as the fourth data block, and sequentially determine each first feature value corresponding to the fourth data block as the fourth feature value.

[0141] Determine the fourth data content suffix corresponding to the fourth feature value, and determine the range of values ​​for the suffix corresponding to the fourth data content suffix;

[0142] If the range of suffix values ​​corresponding to any content cluster of the fourth data block is the same as the range of suffix values ​​corresponding to the fourth data content suffix, then the fourth feature value will be assigned to any content cluster.

[0143] In one specific implementation, the determining unit 501 is specifically used to determine each data block as the current data block in turn, and to determine each target feature value corresponding to the current data block as the current feature value in turn;

[0144] If any target feature value corresponding to any other data block is consistent with the current feature value, then any other data block is determined as a candidate similar block of the current data block. Other data blocks are data blocks other than the current data block among multiple data blocks.

[0145] Based on the target feature value corresponding to the current data block and the target feature value corresponding to each candidate similar block, determine the number of similar features between each candidate similar block and the current data block;

[0146] The candidate similar block with the most corresponding similar features is determined as the similar data block of the current data block.

[0147] In one specific implementation, after determining the similar data blocks corresponding to each data block based on multiple target feature values ​​corresponding to each data block, the device further includes: a compression unit;

[0148] The compression unit is used to perform differential compression on each data block based on the similar data blocks corresponding to each data block.

[0149] Figure 6 This is a schematic diagram of a data processing device structure provided in an embodiment of this application. The data processing device 600 may include one or more central processing units (CPUs) 601 and a memory 605, in which one or more applications or data are stored.

[0150] The memory 605 can be volatile or persistent storage. The program stored in the memory 605 can include one or more modules, each module including a series of instruction operations on the data processing device. Furthermore, the central processing unit 601 can be configured to communicate with the memory 605 and execute the series of instruction operations in the memory 605 on the data processing device 600.

[0151] The data processing device 600 may also include one or more power supplies 602, one or more wired or wireless network interfaces 603, one or more input / output interfaces 604, and / or one or more operating systems, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0152] The central processing unit 601 can perform the aforementioned... Figures 1 to 5The specific operations performed by the data processing device in the illustrated embodiment will not be described in detail here.

[0153] It should be noted that although the steps in the flowcharts of the various embodiments are drawn sequentially according to the arrows, unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the various embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps.

[0154] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0155] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0156] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0157] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0158] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0159] This application also provides a computer program product containing instructions that, when run on a computer, cause the computer to perform the data processing method described above.

Claims

1. A data processing method, characterized in that, include: Determine multiple first feature values ​​corresponding to each data block, wherein each first feature value corresponding to each data block represents a feature of a portion of the data content contained in each data block; The determination of multiple first feature values ​​corresponding to each data block includes: calculating multiple third feature values ​​corresponding to each data block based on the data content contained in each data block and a preset rolling hash algorithm; obtaining a sampling mask based on a preset sampling rate; performing a preset logical operation on each third feature value corresponding to each data block and the sampling mask; and determining the multiple first feature values ​​corresponding to each data block from each third feature value corresponding to each data block according to the execution result corresponding to the preset logical operation. Based on the partial data content corresponding to each first feature value, the multiple first feature values ​​corresponding to each data block are divided into multiple content clusters, and the number of content clusters corresponding to each data block is equal to the preset number of features. Perform a linear transformation operation on each of the first feature values ​​to obtain a second feature value corresponding to each of the first feature values; In each content cluster, the second feature value with the smallest value among the second feature values ​​corresponding to each first feature value is determined as the target feature value of the data block corresponding to each content cluster, and the number of target feature values ​​corresponding to each data block is equal to the preset number of features; Based on multiple target feature values ​​corresponding to each data block, the method for determining similar data blocks corresponding to each data block includes: sequentially determining each data block as the current data block, and sequentially determining each target feature value corresponding to the current data block as the current feature value; if any target feature value corresponding to any other data block is consistent with the current feature value, then determining the other data block as a candidate similar block of the current data block, wherein the other data block refers to data blocks other than the current data block among multiple data blocks; determining the number of similar features between each candidate similar block and the current data block based on the target feature value corresponding to the current data block and the target feature value corresponding to each candidate similar block; and determining the candidate similar block with the largest number of similar features as the similar data block of the current data block.

2. The method according to claim 1, characterized in that, The step of determining the plurality of first feature values ​​corresponding to each data block from each third feature value corresponding to each data block based on the execution result corresponding to the preset logical operation includes: If any execution result satisfies the preset execution result, then the third feature value corresponding to any execution result is determined to be the first feature value of the corresponding data block; If any execution result does not meet the preset execution result, the operation of determining the third feature value corresponding to any execution result as the first feature value of the corresponding data block will not be performed.

3. The method according to claim 1, characterized in that, The step of dividing multiple first feature values ​​corresponding to each data block into multiple content clusters based on the partial data content corresponding to each first feature value includes: The suffix of the data content corresponding to each first feature value is determined as the data content suffix corresponding to each first feature value, and the length of the data content suffix is ​​equal to the preset suffix length; According to the preset suffix length and the preset number of features corresponding to each data block, multiple suffix value ranges are determined, and the number of suffix value ranges is equal to the preset number of features. Based on the range of values ​​for the multiple suffixes and the data content suffixes corresponding to each first feature value, the multiple first feature values ​​corresponding to each data block are divided into multiple content clusters corresponding to each data block, and each content cluster corresponds to a different range of values ​​for the suffixes.

4. The method according to claim 3, characterized in that, The step of dividing the multiple first feature values ​​corresponding to each data block into multiple content clusters corresponding to each data block according to the multiple suffix value ranges and the data content suffix corresponding to each first feature value includes: Each data block is sequentially identified as the fourth data block, and each first feature value corresponding to the fourth data block is sequentially identified as the fourth feature value. Determine the fourth data content suffix corresponding to the fourth feature value, and determine the suffix value range corresponding to the fourth data content suffix; If the suffix value range corresponding to any content cluster of the fourth data block is the same as the suffix value range corresponding to the fourth data content suffix, then the fourth feature value is assigned to any content cluster.

5. The method according to any one of claims 1 to 4, characterized in that, After determining the similar data blocks corresponding to each data block based on multiple target feature values ​​corresponding to each data block, the method further includes: Differential compression is performed on each data block based on the similar data blocks corresponding to each data block.

6. A data processing device, characterized in that, include: A determining unit is used to determine multiple first feature values ​​corresponding to each data block, wherein each first feature value corresponding to each data block represents a feature of a portion of the data content contained in each data block; The determination of multiple first feature values ​​corresponding to each data block includes: calculating multiple third feature values ​​corresponding to each data block based on the data content contained in each data block and a preset rolling hash algorithm; obtaining a sampling mask based on a preset sampling rate; performing a preset logical operation on each third feature value corresponding to each data block and the sampling mask; and determining the multiple first feature values ​​corresponding to each data block from each third feature value corresponding to each data block according to the execution result corresponding to the preset logical operation. The clustering unit is used to divide multiple first feature values ​​corresponding to each data block into multiple content clusters according to the partial data content corresponding to each first feature value. The number of content clusters corresponding to each data block is equal to the preset number of features. An execution unit is configured to perform a linear transformation operation on each of the first feature values ​​to obtain a second feature value corresponding to each of the first feature values. The determining unit is further configured to determine the second feature value with the smallest value among the second feature values ​​corresponding to each first feature value in each content cluster as the target feature value of the data block corresponding to each content cluster, wherein the number of target feature values ​​corresponding to each data block is equal to the preset number of features; The determining unit is further configured to determine similar data blocks corresponding to each data block based on multiple target feature values ​​corresponding to each data block, including: sequentially determining each data block as the current data block, and sequentially determining each target feature value corresponding to the current data block as the current feature value; if any target feature value corresponding to any other data block is consistent with the current feature value, then determining the other other data block as a candidate similar block of the current data block, wherein the other data block is a data block other than the current data block among multiple data blocks; determining the number of similar features between each candidate similar block and the current data block based on the target feature value corresponding to the current data block and the target feature value corresponding to each candidate similar block; and determining the candidate similar block with the largest number of corresponding similar features as the similar data block of the current data block.

7. A data processing device, characterized in that, include: Central processing unit, memory, and input / output interfaces; The memory is either a short-term storage memory or a persistent storage memory; The central processing unit is configured to communicate with the memory and execute instructions in the memory to perform the method of any one of claims 1 to 5.

8. A computer storage medium, characterized in that, The computer storage medium stores instructions that, when executed on the computer, cause the computer to perform the method as described in any one of claims 1 to 5.

Images