Method for parallel extraction of data block features based on simd instruction set and delta compression method

Abstract

The application discloses a data block feature parallel extraction method and a differential compression method based on a SIMD instruction set, and belongs to the technical field of computer storage, and comprises the following steps: dividing a data block into N equal-length sub-blocks, and overlapping adjacent sub-blocks; setting N sliding windows with the same size to synchronously slide from left to right on the N sub-blocks, and the sliding step length is K bytes; when sliding to a position, K bytes on the left in each window are loaded into a vector register L in parallel, and K bytes on the right outside each window are loaded into a vector register R in parallel; then, corresponding bytes of each sub-block are taken from the vector registers L and R in parallel byte by byte to calculate a fingerprint, and features of each sub-block stored in a vector register F are updated; when the sliding window reaches the tail of the sub-block, the features in the F are combined into data block features and written into a memory. The application can effectively improve the throughput of similarity detection under the premise of ensuring the detection effect of similarity detection.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer storage, and more particularly, relates to a data block feature parallel extraction method and a difference compression method based on a SIMD instruction set. BACKGROUND

[0002] With the vigorous development of information technology and the continuous popularity of the Internet, enterprises and individuals will generate a large amount of data every day. According to the statistics and forecasts of IDC, the global data generation reached 33ZB in 2018, and this number will reach a staggering 175ZB by 2025, and the total amount of global data will experience a larger explosion. According to a research report by Microsoft, there is a large amount of redundant data in the storage devices of enterprises and individuals, causing waste of storage resources. Therefore, redundant data elimination technologies such as difference compression have become a hot topic of research in the current academic and industrial circles. The compression object of difference compression is similar files or data blocks. In order to improve the effect of difference compression as much as possible and reduce the computational overhead of difference encoding, similarity detection technology is often used to select the objects of difference compression in the difference compression process.

[0003] Directly comparing data content is the simplest method to judge the similarity between two files or data blocks, but this method is too time-consuming and cannot meet the performance requirements of large-scale storage systems. The current storage systems mainly use feature values of files or data blocks to replace data content for similarity judgment. This type of similarity detection method is generally divided into two stages: feature extraction and feature matching. Feature extraction refers to the process of extracting representative feature values from files or data blocks, while feature matching refers to the process of finding similar files or data blocks by matching feature values.

[0004] Super-feature value method (Super-feature) is the most commonly used similarity detection method at present, and is widely used in storage systems. In the feature extraction stage, the super-feature value method first uses a sliding hash algorithm based on Rabin fingerprints to calculate the Rabin fingerprints of all data segments in the data block, then performs N linear transformation operations on all Rabin fingerprints, and finally selects the maximum hash value corresponding to each linear transformation operation as the feature value of the data block. After the end of the feature extraction stage, the super-feature value method will package a fixed number of feature values into multiple super-feature values. If the super-feature values of two data blocks are the same, it means that they have a high probability of being similar. This packaging strategy simplifies the feature matching process to querying the super-feature value index, greatly reducing the detection overhead of similarity detection, and achieving good detection effect.

[0005] However, the super eigenvalue method has a large computational overhead, which limits the throughput of data block similarity detection and affects the execution efficiency of subsequent differential compression. At the same time, with the continuous development of SIMD (Single Instruction Multiple Data) technology, more and more researchers have begun to use hardware to optimize the computationally intensive part of the redundant data elimination process, such as SS-CDC which uses AVX-512 instruction set and multi-threading technology to realize two-stage parallelism of data blocking process, improving the blocking throughput. Similar to data blocking, the feature extraction process of data block similarity detection also belongs to the computationally intensive part of the redundant data elimination process, and some work has been done to reduce the computational overhead of the feature extraction stage of the similarity detection method. However, these works all execute the feature extraction of data blocks in a serial manner, without optimizing the existing similarity detection method from the perspective of parallelization.

[0006] Therefore, how to improve the throughput of the feature extraction method while ensuring the detection effect of the similarity detection method is a problem to be solved. SUMMARY

[0007] In view of the defects and improvement needs of the prior art, the present application provides a data block feature parallel extraction method based on SIMD instruction set and a differential compression method, which aims to effectively solve the time delay problem of feature extraction in the data block similarity detection process, thereby effectively improving the throughput of similarity detection while ensuring the detection effect of similarity detection in differential compression.

[0008] To achieve the above-mentioned purpose, according to one aspect of the present application, a data block feature parallel extraction method based on SIMD instruction set is provided, comprising:

[0009] After equally dividing the data block into N non-overlapping data segments, the right boundary of the first data segment is shifted right by M bytes, and the left boundaries of the second to Nth data segments are shifted left by M bytes respectively, obtaining N sub-blocks;

[0010] N sliding windows with a size of W bytes are set to slide synchronously from left to right on the N sub-blocks, with a sliding step of K bytes; when the sliding window slides to a position, K bytes on the left side in each sliding window are loaded into a vector register L in parallel, and K bytes on the right side outside each sliding window are loaded into a vector register R in parallel, then the corresponding bytes of each sub-block are taken from the vector registers L and R in parallel byte by byte to calculate the fingerprints at the corresponding bytes of each sub-block, and the features of each sub-block are updated based on the fingerprint calculation results; the features of each sub-block are stored in a vector register F;

[0011] When the sliding window reaches the rightmost side of each sub-block, the features of each sub-block in the vector register F are combined into data block features and written into memory;

[0012] Where N, W, M, and K are all positive integers, and M is less than the data segment length, and N*K does not exceed the number of bytes that can be loaded at one time by the parallel data loading instruction in the SIMD instruction set.

[0013] Furthermore, M = Wn; n is a positive integer, and n <W。

[0014] Furthermore, n = 1.

[0015] Furthermore, the parallel loading of the K bytes on the left side of each sliding window into the vector register L, and the parallel loading of the K bytes on the right side outside each sliding window into the vector register R, are both accomplished using the gather instruction.

[0016] Furthermore, N = 16, K = 4; or, N = 8, K = 8.

[0017] Furthermore, W = 48, or W = 32.

[0018] Furthermore, the corresponding bytes of each sub-block are retrieved in parallel from the vector registers L and R, using the AND instruction.

[0019] Furthermore, for a single sub-block, after performing Rabin fingerprint calculation on the P-th byte retrieved from vector registers L and R, the resulting fingerprint calculation result is used as the fingerprint at the corresponding byte of that sub-block;

[0020] Where P = 1, 2, ..., K.

[0021] Furthermore, the features of each sub-block are updated based on the fingerprint calculation results, including:

[0022] For each sub-block B i The calculated fingerprint M i With the feature F stored in the vector register F i Compare, if M i >F i Then the feature F stored in the vector register F will be... i Replace with the calculated fingerprint M i Otherwise, the feature F stored in the vector register F i It remains unchanged.

[0023] According to another aspect of the present invention, a differential compression method is provided, comprising:

[0024] The above-mentioned parallel feature extraction method for data blocks based on the SIMD instruction set provided by the present invention is used to extract features from each data block;

[0025] The feature matching is performed on the extracted data block features to obtain basic data blocks and similar data blocks thereof;

[0026] The difference between the similar data blocks and the corresponding basic data blocks is encoded to obtain difference encoding data blocks, and the basic data blocks and the corresponding difference encoding data blocks are stored respectively to complete the difference compression.

[0027] According to still another aspect of the present application, a storage controller is provided, comprising a processor and a computer readable storage medium;

[0028] The computer readable storage medium stores a computer program therein; and the processor is configured to read the computer program stored in the computer readable storage medium, and execute the above-mentioned parallel feature extraction method of data blocks based on the SIMD instruction set and / or the above-mentioned difference compression method.

[0029] Overall, the above technical solutions conceived by the present application can achieve the following beneficial effects:

[0030] (1) The present application divides a data block into sub-blocks, and uses the SIMD instruction set to extract features of multiple sub-blocks in parallel, and finally the features of each sub-block jointly constitute the features of the entire data block. In the sub-blocks obtained by dividing the data block, each sub-block retains the byte information at the end of the previous sub-block, thereby retaining the content dependency in the fingerprint calculation process, eliminating the influence of the block operation on the fingerprint calculation at the boundary, and achieving the same extraction effect as the traditional serial feature extraction method. The traditional method of extracting features of data blocks in a serial manner has the number of fingerprint and feature value calculations equal to the length of the data block in the feature extraction process, while the present application parallelizes the feature extraction process of the data block, and the number of fingerprint and feature value calculations is the length of the sub-block, so the throughput can be much higher than that of the traditional serial feature extraction method. Overall, the present application can effectively improve the throughput of feature extraction while ensuring the feature extraction effect, thereby effectively improving the throughput of similarity detection while ensuring the detection effect of similarity detection in difference compression.

[0031] (2) In the process of dividing the block, the present application controls the offset of the data slice boundary to be less than the size of the sliding window, so that the size of the overlapping area between adjacent sub-blocks in the 2nd to Nth sub-blocks is less than the size of the sliding window, thereby ensuring that there is no repeated calculation in the subsequent fingerprint calculation process. In the preferred scheme, the offset is 1 byte less than the window length, thereby maximizing the dependency between the front and rear contents while avoiding repeated fingerprint calculation, and ensuring the feature extraction effect. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 A flow chart of a data block feature parallel extraction method based on a SIMD instruction set is provided for an embodiment of the present application.

[0033] Figure 2 A sub-block division method schematic diagram is provided for an embodiment of the present application.

[0034] Figure 3 A schematic diagram of loading byte data of each sub-block into a vector register based on a gather instruction in an AVX-512 instruction set is provided for an embodiment of the present application.

[0035] Figure 4 A schematic diagram of selecting byte data from a vector register to calculate a Rabin fingerprint and update a feature value based on an and instruction in an AVX-512 instruction set is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0036] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application. In addition, the technical features involved in each embodiment of the present application described below can be combined with each other as long as they do not conflict with each other.

[0037] In the present application, the terms "first", "second", etc. (if any) in the present application and the accompanying drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.

[0038] In order to solve the technical problem of low similarity detection throughput caused by the serial data block feature extraction method of the existing differential compression technology, the present application provides a data block feature parallel extraction method based on a SIMD instruction set and a differential compression method, and the overall idea is as follows: using the parallelism of the data recording instruction in the SIMD instruction set, the data block is divided into multiple overlapping sub-blocks to retain the content dependency between the front and rear data, and the multiple sub-blocks are extracted in parallel, thereby effectively improving the throughput of feature extraction on the basis of ensuring the effect of feature extraction.

[0039] The present application can be implemented based on any kind of SIMD instruction set supporting parallel data loading instruction, without losing generality, in the following embodiments, AVX-512, a specific SIMD instruction set, is taken as an example for description. It is easy to understand that in the following embodiments, "left" and "right" of the data block respectively represent the low address end and the high address end.

[0040] The following is an embodiment.

[0041] Embodiment 1:

[0042] A parallel feature extraction method for data blocks based on SIMD instruction sets is illustrated in the flowchart below. Figure 1 As shown.

[0043] This embodiment first divides the data into multiple overlapping sub-blocks. The parallel data loading instruction in the AVX-512 instruction set, namely the gather instruction, can load 4 bytes of data from 16 sub-blocks in parallel at once, or load 8 bytes of data from 8 sub-blocks in parallel. In order to make full use of this parallelism, as a preferred implementation, in this embodiment, the number of overlapping sub-blocks is specifically set to N=16, and the number of bytes loaded in each sub-block each time byte data is loaded into the vector register is K=4.

[0044] Based on the above parameter settings, such as Figure 2 As shown, this embodiment divides the data block into 16 overlapping sub-blocks in the following way:

[0045] After dividing the data block into 16 non-overlapping data segments, the right boundary of the first data segment is shifted to the right by M bytes, and the left boundaries of the second to 16th data segments are shifted to the left by M bytes respectively, resulting in 16 sub-blocks, which are denoted as sub-block 0 to sub-block 15; M is a positive integer and M is less than the length of the data segment;

[0046] In the 16 sub-blocks obtained in this embodiment, the size of the overlapping area between the first and second sub-blocks is 2M bytes, and the size of the overlapping area between other adjacent sub-blocks is M bytes. That is to say, each sub-block retains the byte information at the end of the previous sub-block.

[0047] After the sub-blocks are divided, as follows: Figure 3 As shown, 16 sliding windows of size W bytes are set to slide synchronously from left to right on 16 sub-blocks, with a sliding step size of K bytes; corresponding to the number of bytes loaded in each sub-block each time, in this embodiment, the sliding step size of the sliding window is K=4; optionally, in this embodiment, the size of the sliding window is W=48 bytes.

[0048] Each time the sliding window slides to a position, the gather instruction is used to load the K bytes on the left side of each sliding window (bytes of data about to leave the sliding window) into vector register L in parallel. The gather instruction is also used to load the K bytes on the right side of each sliding window (bytes of data about to enter the sliding window) into vector register R in parallel. Then, the corresponding bytes of each sub-block are retrieved byte by byte in parallel from vector registers L and R to calculate the fingerprint of the corresponding byte of each sub-block. The features of each sub-block are updated based on the fingerprint calculation results. The features of each sub-block are stored in vector register F.

[0049] Since the data block is directly divided into non-overlapping data segments for parallel computing, the dependency of the generation of the fingerprint by the bytes around the sliding window is not taken into account, which may cause errors in the fingerprint calculation and inaccurate subsequent detection results. The overlapping sub-blocks provided by the embodiment can retain the dependency between the front and rear data and eliminate the influence of the block boundary operation on the Rabin fingerprint calculation of the boundary, since each sub-block retains the byte information at the end of the previous sub-block. In order to avoid repeated calculation when calculating the fingerprint of the data around the sliding window, the offset of the sub-block boundary is controlled to be less than the byte length of the sliding window when the overlapping sub-blocks are divided. In the embodiment, the offset is M=W-1=47. Except for the first sub-block, the hash values calculated by the first M bytes of each sub-block do not participate in the feature value calculation.

[0050] Referring to Figure 1 and Figure 4 , the process of calculating the fingerprint based on the sliding window and updating the features specifically includes the following steps:

[0051] (S0) loading the K bytes on the left side of each sliding window into the vector register L in parallel, and loading the K bytes on the right side of each sliding window outside the sliding window into the vector register R in parallel;

[0052] (S1) initializing P=1;

[0053] (S2) taking the Pth byte of each sub-block from the vector register L and taking the Pth byte of each sub-block from the vector register R, and for each sub-block, calculating the hash value of the two bytes read from the vector register by using the Rabin hash function to obtain the corresponding fingerprint; in order to extract the bytes at a specific position, the data of the Pth byte selected from the vector registers L and R are subjected to Rabin fingerprint calculation by using the and instruction in the embodiment;

[0054] (S3) determining whether the current byte is the first 47 bytes of the last 15 sub-blocks of the data block, and if so, directly entering step (S5), otherwise, entering step (S4);

[0055] (S4) updating the feature values of the sub-blocks in the vector register F according to the calculated fingerprint; specifically, comparing the fingerprints of the sub-blocks with the feature values of the sub-blocks in the vector register F, and if the fingerprint value is greater than the feature value, updating the feature value in the vector register F as the fingerprint of the corresponding sub-block, otherwise, keeping the feature value in the vector register F unchanged;

[0056] (S5) if P=K, the fingerprint calculation of the byte data in the current sliding window is completed, and step (S6) is entered; otherwise, P=P+1, and then step (S2) is entered;

[0057] (S6) simultaneously sliding each sliding window to right by K bytes, and judging whether the sliding window reaches the tail of the sub-block, if yes, combining the features of each sub-block in the vector register F into the data block feature and writing into the memory, and the data block feature extraction ends; otherwise, turning to step (S0).

[0058] Embodiment 2

[0059] A differential compression method, comprising:

[0060] performing feature extraction on each data block by using the parallel data block feature extraction method based on the SIMD instruction set provided in Embodiment 1;

[0061] performing feature matching on the extracted data block features to obtain basic data blocks and similar data blocks thereof;

[0062] performing differential encoding on the difference between the similar data blocks and the corresponding basic data blocks to obtain differential encoding data blocks, and respectively storing the basic data blocks and the corresponding differential encoding data blocks, thereby completing the differential compression.

[0063] Embodiment 3

[0064] A storage controller, comprising a processor and a computer readable storage medium;

[0065] The computer readable storage medium stores a computer program; the processor is configured to read the computer program stored in the computer readable storage medium, and execute the parallel data block feature extraction method based on the SIMD instruction set provided in Embodiment 1, and / or the differential compression method provided in Embodiment 2.

[0066] Those skilled in the art can easily understand that the above description is only the preferred embodiment of the present application, and is not intended to limit the present application, and any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A parallel feature extraction method for data blocks based on SIMD instruction sets, characterized in that, include: After dividing the data block into N non-overlapping data segments, the right boundary of the first data segment is shifted to the right by M bytes, and the left boundaries of the second to Nth data segments are shifted to the left by M bytes respectively, resulting in N sub-blocks; N sliding windows, each with a size of W bytes, slide synchronously from left to right across N sub-blocks, with a sliding step size of K bytes. Each time a sliding window reaches a position, the leftmost K bytes within each window are loaded in parallel into vector register L, and the rightmost K bytes outside each window are loaded in parallel into vector register R. Then, the corresponding bytes for each sub-block are retrieved byte-by-byte and in parallel from vector registers L and R to calculate the fingerprint at the corresponding byte in each sub-block. The features of each sub-block are updated based on the fingerprint calculation results. The features of each sub-block are stored in vector register F. When the sliding window reaches the rightmost side of each sub-block, the features of each sub-block in the vector register F are combined into data block features and written into memory; Where N, W, M and K are all positive integers, and M is less than the data segment length, and N*K does not exceed the number of bytes that can be loaded at one time by the parallel data loading instruction in the SIMD instruction set.

2. The parallel feature extraction method for data blocks based on the SIMD instruction set as described in claim 1, characterized in that, M = Wn; n is a positive integer, and n <W。 3. The parallel feature extraction method for data blocks based on the SIMD instruction set as described in claim 1, characterized in that, n＝1。 4. The parallel extraction method for data block features based on the SIMD instruction set as described in any one of claims 1 to 3, characterized in that, The K bytes on the left side of each sliding window are loaded in parallel into the vector register L, and the K bytes on the right side of each sliding window are loaded in parallel into the vector register R, both of which are accomplished using the gather instruction.

5. The parallel feature extraction method for data blocks based on the SIMD instruction set as described in claim 4, characterized in that, N = 16, K = 4; Alternatively, N=8, K=8.

6. The parallel feature extraction method for data blocks based on the SIMD instruction set as described in claim 3, characterized in that, The corresponding bytes of each sub-block are retrieved in parallel from vector registers L and R, using the AND instruction.

7. The parallel feature extraction method for data blocks based on the SIMD instruction set as described in claim 6, characterized in that, For a single sub-block, the fingerprint calculation result obtained after performing Rabin fingerprint calculation on the P-th byte taken from vector registers L and R is used as the fingerprint at the corresponding byte of the sub-block. Where P = 1, 2, ..., K.

8. The parallel extraction method for data block features based on the SIMD instruction set as described in claim 7, characterized in that, The features of each sub-block are updated based on the fingerprint calculation results, including: For each sub-block B i The calculated fingerprint M i With the feature F stored in the vector register F i Compare, if M i >F i Then the feature F stored in the vector register F will be... i Replace with the calculated fingerprint M i Otherwise, the feature F stored in the vector register F i It remains unchanged.

9. A differential compression method, characterized in that, include: The feature extraction method for data blocks based on the SIMD instruction set according to any one of claims 1 to 8 is used to extract features from each data block; Feature matching is performed on the extracted data block features to obtain the basic data block and its similar data blocks; Differential encoding is performed on the difference between similar data blocks and their corresponding basic data blocks to obtain differential encoded data blocks. The basic data blocks and their corresponding differential encoded data blocks are stored separately to complete differential compression.

10. A storage controller, characterized in that, Includes processors and computer-readable storage media; The computer-readable storage medium stores a computer program; the processor is used to read the computer program stored in the computer-readable storage medium and execute the parallel extraction method for data block features based on the SIMD instruction set according to any one of claims 1 to 8, and / or the differential compression method according to claim 9.

Images