Method for data compression and related device

Abstract

Embodiments of this application provide a method for data compression and a related electronic device. The method includes obtaining one or more feature words from at least one non-repeating part of an input data block and determining at least one dictionary for delta compressing the input data block from processed data blocks according to the feature words. One or more dictionaries determined according to the method provided in this disclosure will result in better compression efficiency and a smaller CR when they are used to delta compress the input data block. The data compression method and its related electronic device could be widely used in file compression, image compression, network protocols, etc.

Description

METHOD FOR DATA COMPRESSION AND RELATED DEVICETECHNICAL FIELD[oooi] Embodiments of the present invention relate to the field of information technologies, and more specifically, to a method for data compression and a related device.BACKGROUND

[0002] Data similarity (resemblance) detection is widely used in data storage and data transmission over networks, plagiarism detection, web search, etc. Similarity detection, when used in data storage and transmission devices, makes it possible to apply deduplication or delta compression for similar data, thus further improving the efficiency of these devices.

[0003] Similarity candidates used as dictionaries in delta compression could be selected from input data blocks, while how to select them matters when taking the efficiency of delta compression into account.SUMMARY

[0004] Embodiments of this application provide a method for data compression and a related device. The technical solution provides several methods to obtain one or more feature words and sequences of bytes from non-repeating parts in a input data block and use the feature words for similarity detection of the input data block, which has a good effect on improving delta compression efficiency.

[0005] According to a first aspect, an embodiment of this application provides a method for data compression, including: obtaining one or more feature words from at least one nonrepeating part of an input data block, where a feature word is a sequence of bytes; and determining at least one dictionary for delta compressing the input data block from processed data blocks according to the feature words.

[0006] In some cases, a non-repeating part is also known as a literal or a non-repeat.£0007] In some cases, the method further includes: delta compressing the input data block by using the dictionary or jointly compressing the input data block and another input data block that contains the feature word.

[0008] Here, jointly compressing the input data block with another input data block can also be interpreted as delta compressing two input data blocks together that contain the same feature word.

[0009] In some cases, before obtaining one or more feature words from the non-repeating part of the input data block, the method further includes: compressing the input data block by an LZ-class compression method and obtaining the non-repeating part.[ooioj Optionally, the compressing process and the obtaining process can be two different stages or be separated in time to accelerate the compressing process.

[0011] The LZ-class compression method could detect repeating parts and non-repeating parts in the input data block and compress the input data block in the meantime.

[0012] In some cases, the input data block is highly compressible, so there may be no nonrepeating part in the input data block longer than a given length, then no feature word would be obtained. It would accelerate the feature word-obtaining progress by ignoring the highly compressible input data block.

[0013] There may be some similar non-repeating parts between one input data block and one processed data block or different input data blocks. The feature word obtained from the non-repeating part will be beneficial to discover that processed data block and input data block. If more input data blocks that contain the feature word could be compressed further, then a better CR and improved compression efficiency could be achieved.

[0014] In a possible design, obtaining one or more feature words from at least one nonrepeating part of an input data block, includes: obtaining one feature word from each nonrepeating part of the input data block if the input data block includes more than one nonrepeating part; or obtaining more than one feature word from one non-repeating part if the input data block only includes the non-repeating part.

[0015] A more detailed method for feature word obtaining is provided in this technical solution, which aims at ensuring the compression method disclosed in this application could bewidely applied to different types of non-repeating parts.

[0016] In a possible design, obtaining one feature word from each non-repeating part of an input data block if the input data block includes more than one non-repeating part, includes: obtaining multiple pending words by a sliding window from each non-repeating part, where a pending word is a sequence of bytes; and selecting the feature word from the pending words by casting them to integers, where the feature word is the one with a minimum integer value.

[0017] In this way, the feature words can be selected based on the content of the nonrepeating part rather than on the feature word position in the non-repeating part, and this selection method shows quite good results in practical experiments regarding compression efficiency.

[0018] In a possible design, obtaining one feature word from each non-repeating part of an input data block if the input data block includes more than one non-repeating part, includes: calculating rolling hash values for bytes inside a sliding window; and selecting the feature word according to the rolling hash values, where the feature word is a sequence of bytes with a minimum hash value.

[0019] Another position-independent method to select a feature word is provided in this technical solution. Calculating hash values of numerous feature words may be more computationally expensive compared to casting feature words to integers. These two kinds of methods may be suitable for different devices with different computational resources.

[0020] In a possible design, if the input data block includes more than one non-repeating part, the feature word is a sequence of bytes at the beginning or the end of the non-repeating part.

[0021] In a possible design, the feature word is the one with a smaller hash value.

[0022] A position-dependent selection method may be less computationally expensive, so a less power consumption data compression result could be achieved by using this method.

[0023] In a possible design, obtaining more than one feature word from one non-repeating part if the input data block only includes the non-repeating part, includes: selecting the feature words from all words contained in the non-repeating part by casting them to integers, where the feature words are words with N smallest integer values, where N is an integer such that N>1.

[0024] In this way, the feature words can be selected based on the content of the non-repeating part rather than on the feature word position in the non-repeating part, and this selection method shows quite good results in practical experiments regarding compression efficiency.

[0025] In a possible design, obtaining more than one feature word from one non-repeating part if the input data block only includes the non-repeating part, includes: selecting the feature words by a sliding window sliding from one end of the non-repeating part until N feature words with N smallest integer values or N smallest hash values are obtained or the sliding window hits the other end of the non-repeating part, where N is an integer such that N>1.

[0026] By using a sliding window, fewer pending words need to be cast to integers or fewer hash values need to be calculated for pending words. This technical solution provides a less computationally expensive method to select feature words.

[0027] In a possible design, the input data block contains multiple feature words that are corresponding to at least N different hash values, where N is an integer such that N>1.

[0028] The Hash values could help to simplify the process of searching for a suitable processed data block. For example, checking if one string is equal to another string is much more computationally expensive than checking if their hash values are identical.

[0029] Even though different feature words may be corresponding to one hash value, the number of different hash values can still stand for the number of different feature words obtained from one input data block to some extent.

[0030] Ensuring there are plenty of different hash values corresponding to one input data block is beneficial to find a similar processed data block with the input data block. As a result, a more suitable dictionary to delta compress the input data block will be discovered and a much smaller CR and better efficiency will be achieved.

[0031] In a possible design, the input data block contains at least N different feature words, where N is an integer such that N>1.

[0032] Ensuring there are plenty of different feature words contained in one input data block is more directly helpful to find a similar processed data block with the input data block and achieve a smaller CR and better compression efficiency.

[0033] In a possible design, the dictionary contains at least M feature words that are equal respectively to the feature words of the input data block, where M is an integer such that M>1.

[0034] Optionally, if there are fewer M feature words in the processed data block that are equal respectively to the feature words of the input data block, then the input data block is not considered for delta compression.

[0035] The more identical feature words existing in both the processed data block and the input data block, the more similar the two data blocks will be. A threshold for the number of these identical feature words will guarantee the effectiveness of delta compression.

[0036] In a possible design, determining at least one dictionary for delta compressing the input data block from processed data blocks according to the feature words, includes: determining one dictionary for delta compressing the input data block from processed data blocks according to the feature words, where the dictionary contains the most feature words that are equal respectively to the feature words of the input data block.

[0037] The best candidate among processed blocks for delta compression is the block with the maximum number of feature words equal respectively to feature words of the input data block.

[0038] In a possible design, in case that the input data block contains at least N different feature words, the method further includes: storing hash values of the feature words as keys and a storage address of the input data block as a value in <key, value> pairs, where N is an integer such that N>1.

[0039] All hash values calculated for feature words of the input data block could be stored in a key-value store for processing future input data blocks. The <value> in the <key, value> pair contains meta-information about the input data block, and the <key> in the <key, value> pair contains meta-information about the feature words.

[0040] For brevity, technical details and benefits of the solutions in the following aspects will not be repeated and could refer to the description in the first aspect.

[0041] According to a second aspect, provided is an electronic device, including: an obtaining module configured to obtain one or more feature words from at least one nonrepeating part of an input data block, where a feature word is a sequence of bytes; and a determining module configured to determine at least one dictionary for delta compressing the input data block from processed data blocks according to the feature words.

[0042] In a possible design, the obtaining module is further configured to obtain one featureword from each non-repeating part of the input data block if the input data block includes more than one non-repeating part; or obtain more than one feature word from one non-repeating part if the input data block only includes the non-repeating part.

[0043] In a possible design, if the input data block includes more than one non-repeating 5 part, the obtaining module is further configured to obtain multiple pending words by a sliding window from each non-repeating part, where a pending word is a sequence of bytes; and the determining module is further configured to select the feature word from the pending words by casting them to integers, where the feature word is a pending word with a minimum integer value.> 10

[0044] In a possible design, if the input data block includes more than one non-repeating part, the determining module is further configured to calculate rolling hash values for bytes inside a sliding window and select the feature word according to the rolling hash values, where the feature word is a sequence of bytes with a minimum hash value.

[0045] In a possible design, if the input data block includes more than one non-repeating 15 part, the feature word is a sequence of bytes at the beginning or the end of the non-repeating part.

[0046] In a possible design, the feature word is the one with a smaller hash value.

[0047] In a possible design, if the input data block includes only one non-repeating part, the determining module is further configured to select the feature words from all words 20 contained in the non-repeating part by casting them to integers, where the feature words are words with N smallest integer values.

[0048] In a possible design, if the input data block includes only one non-repeating part, the determining module is further configured to select the feature words by a sliding window sliding from one end of the non-repeating part until N feature words with N smallest integer 25 values are obtained or the sliding window hits the other end of the non-repeating part, where N is an integer such that N>1.

[0049] In a possible design, the input data block contains multiple feature words that are corresponding to at least N different hash values, where N is an integer such that N>1.

[0050] In a possible design, the input data block contains at least N different feature words, 30 where N is an integer such that N>1.

[0051] In a possible design, the dictionary contains at least M feature words that are equal respectively to the feature words of the input data block, where M is an integer such that M>1.

[0052] In a possible design, the determining module is further configured to determine one dictionary for delta compressing the input data block from processed data blocks according to the feature words, where the dictionary contains the most feature words that are equal respectively to the feature words of the input data block.

[0053] In a possible design, the determining module is further configured to store hash values of the feature words as keys and a storage address of the input data block as a value in <key, value> pairs if there are at least N different feature words contained in the input data block, where N is an integer such that N>1.

[0054] According to a third aspect, an electronic device is provided, including a processor and a memory. The processor is connected to the memory. The memory is configured to store instructions, and the processor is configured to execute the instructions. When the processor executes the instructions stored in the memory, the processor is enabled to perform the method in the first aspect or any possible design of the first aspect.

[0055] According to a fourth aspect, an embodiment of this application provides a computer readable storage medium, including instructions. When the instructions are run on a computer, the computer is enabled to perform the method in the first aspect or any possible design of the first aspect.

[0056] According to a fifth aspect, a chip system is provided, where the chip system includes a memory and a processor, where the memory is configured to store a computer program, and the processor is configured to invoke the computer program from the memory and run the computer program, so that a server on which the chip is disposed performs the method in the first aspect or any possible design of the first aspect.

[0057] According to a sixth aspect, a computer program product is provided, where when the computer program product is run on an electronic device, the electronic device is enabled to perform the method in the first aspect or any possible design of the first aspect.DESCRIPTION OF DRAWINGS

[0058] FIG. 1 shows a schematic diagram of an input data block that is composed of NRPs and an RP.

[0059] FIG. 2 shows a schematic diagram of another input data block that is composed of an NRP.

[0060] FIG. 3 shows a flowchart of an embodiment of methods to obtain a feature word.

[0061] FIG. 4 shows a schematic diagram for a sliding window method.

[0062] FIG. 5 shows a flowchart of an embodiment of methods to use a feature word for similarity detection.

[0063] FIG. 6 shows a schematic diagram of methods to determine candidates for delta compression dictionaries according to a feature word.

[0064] FIG. 7 shows a flowchart of an embodiment of a data compression method disclosed in this application generally.

[0065] FIG. 8 shows a schematic block diagram of an electronic device according to an embodiment of this application.

[0066] FIG. 9 shows a schematic block diagram of another electronic device according to an embodiment of this application.DESCRIPTION OF EMBODIMENTS

[0067] The following describes the technical solutions in this application with reference to the accompanying drawings.

[0068] Delta encoding is one kind of lossless compression method, which is a way of storing or transmitting data in the form of differences (deltas) between sequential data rather than complete files, and more generally, this is known as data differencing. Delta encoding is sometimes called as delta compression, particularly in a case where archival histories of changes are required (e.g., in revision control software).

[0069] Delta compression is suitable for data that have continuity or periodicity, such as audio, video, image, etc. The data compression method disclosed in this application could be treated as an improved compression method that focuses on how to select candidates of deltacompression dictionaries from input data blocks, and it could be applied to all types of data as delta compression could be.

[0070] Before introducing specific embodiments, some key terms that would be used in this application will be introduced as follows.

[0071] Lempel-Ziv (LZ) compression or LZ-class compression is a family of lossless data compression methods that uses the principle of dictionary coding, where repeated or common sequences of data are stored as references to a dictionary. LZ compression algorithms are widely used in various applications, such as file compression, image compression, and network protocols.

[0072] A delta compression dictionary is a data structure that stores a set of common elements that appear in both the target file and the reference file and could be used to construct the delta file. A dictionary can reduce the size of the delta file by avoiding the repetition of identical elements and can also speed up the encoding and decoding process by simple lookups. A dictionary can be either static or dynamic. A dynamic dictionary is generated on-the-fly from the reference file or files.

[0073] A key-value store, also known as a hash value table, is a type of database that stores data as a set of unique identifiers, each of which has an associated value. Key-value stores are optimized for reading and writing data by using the key to fetch or store the value. The value can be anything from a simple string to a complex object.

[0074] The sliding window method is a computational technique that aims to reduce the use of nested loops and replace them with a single loop, thereby reducing the time complexity. It involves using a window of fixed size that moves over the data sample by sampling and computing the desired static over the data in the window. The window length and overlap length affect the output and performance of the method.

[0075] Huffman coding, also named as a Huffman encoder, uses a specific method for choosing the representation for each symbol, resulting in a prefix code.

[0076] Entropy coding (or entropy encoding) is any lossless data compression method that attempts to approach the lower bound declared by Shannon’s source coding theorem, which states that any lossless data compression method must have an expected code length greater than or equal to the entropy of the source.

[0077] A data compression ratio or compression ratio (CR), also known as compression power, is a measurement of the relative reduction in size of data representation produced by a data compression algorithm. It is typically expressed as the division of uncompressed size by compressed size.

[0078] As shown in FIG. 1, block A is composed of a first non-repeating part (or NRP#1), a second non-repeating part (or NRP#2), and a first repeating part (or RP#1). A non-repeating part is also known as a literal and a repeating part is also known as a repeat. Non-repeating parts and repeating parts could be detected by LZ-class compression algorithms and LZ-class-like compression algorithms, like LZ4, LZAP, and LZ78, etc.

[0079] As block A is compressed by LZ-compression separately, NRP#1 and NRP#2 are stored as uncompressed parts and RP#1 is compressed as a compressed part. The CR of block A is determined by the division of the size of NRP#1 and NRP#2 by the size of RP#1.

[0080] Since NRP#1 and NRP#2 may contain plenty of bytes and there may be at least one feature word (like FW#1) existing in both NRP#1 and NRP#2. It will be helpful to compress block A further.

[0081] FIG. 2 shows another data block, block B, which is composed of one non-repeating part named as NRP#3. In one possible embodiment, NRP#3 may also contain FW#1 that also exists in block A as mentioned above.

[0082] Both block A and block B may be input data blocks, or one of them is an input data block and the other is a processed data block. In other words, there may be at least one feature word that exists in two input data blocks or one input data block and one processed data block, which may have a good effect on a better CR for the whole compression process.

[0083] To find such a helpful feature word and its related data block that could be used to improve the efficiency of the compression process, one data compression method is provided as follows.

[0084] S 110, obtaining NRPs (short for non-repeating parts) from an input data block.

[0085] In some cases, NRPs could be detected by scanning the input data block by an algorithm like LZ-class compression. The NRPs are non-repeating byte sequences (“words” or “phrases”) in the input data block.

[0086] Usually, the compression method of LZ-class considers sequences of at least 3 bytesto search for repeats, and the faster algorithm LZ4 considers repeats of at least 4 bytes in length, which should not be explained as a limitation of this application.

[0087] SI 20, obtaining pending words from NRPs by the sliding window method.

[0088] FIG. 4 shows an example of a sliding window method. Direction 10 indicates the moving direction of the sliding window. The window length of a sliding window is shorter than the length of an NRP, or in other words, the size of the sliding window is less than the size of the NRP. The dotted rectangle stands for a first position and the solid rectangle stands for a second position, and the two different rectangles overlap each other. As the window moves from the first position to the second position by step S, a pending word with the window length could be obtained. Several pending words of one NRP are obtained during the process of moving the sliding window from one end of the NRP to the other.

[0089] The pending word obtained at the beginning of an NRP could be treated as the first pending word and the one obtained at the end of the NRP would be named as the last pending word. In some possible cases, referring to FIG. 4, if the sliding window moves along direction 10, then the first pending word is the leftmost one and the last pending word is the rightmost one.

[0090] The size of a pending word is dependent on the size of the sliding window. As the length of the window alters, different kinds of pending words with different sizes would be obtained. For example, if the size of an NRP is L, the size of the sliding window can be L-l, L- 2, L / 2, etc. and the size of a pending word will be L-l, L-2, L / 2, etc.

[0091] It should be mentioned that the size of a sliding window or the size of pending words should not be too small, as smaller sizes usually result in more pending words and longer obtaining processes, which would cost much more computational resources and storage resources.

[0092] Input data blocks may sometimes be highly compressible, and they may not have non-repeating parts with the size, where a pending word with a suitable size could be obtained by all possible moving steps. In such cases, these input data blocks may not be considered for similarity detection and delta compression.

[0093] SI 30, determining the feature word according to a given rule.

[0094] The feature word here refers to one or more feature words selected from all pendingwords obtained in SI 20, which will have a good effect on compression efficiency.

[0095] In some cases, the feature word may be the first pending word or the last pending word mentioned in SI 20.

[0096] In some cases, the feature word may be the one with the minimum conversion value or the maximum conversion value among all obtained pending words. The value of a pending word could be calculated by casting it to a number according to ASCII Table. For instance, the pending word ‘abed’ cast to a 32-bit integer in binary is 01100001 01100010 01100011 01100100.

[0097] In some cases, rolling hash values are calculated for bytes inside a sliding window, and the feature word may be the one with the minimum hash value or the one with the maximum hash value among all obtained pending words.

[0098] With the help of conversion value or hash value, the selected feature word is more position-independent, which shows quite good results in practical experiments regarding compression efficiency.

[0099] In some cases, an input data block may include more than one NPR, as described above, at least one feature word may be obtained in one NRP, and then more than one feature word will be obtained in the input data block. The feature word obtained in one NRP with a minimum conversion value or a minimum hash value may not be the minimum one among all feature words obtained from one data block, so it would be named as a local minimum feature word in the following description. Similarly, the feature word obtained in one NRP with a maximum hash value or a maximum conversion value would be named as a local maximum feature word in the following description.[oiooj In some cases, an input data block can’t be compressed by the LZ-class compression method, in other words, only one non-repeating part is contained in the input data block. In this case, more than one feature word is selected from the single non-repeating part.

[0101] In some cases, all words contained in the non-repeating part are cast to integers, and feature words are words with N smallest integer values, where N is an integer such that N>1.

[0102] In some cases, a sliding window is moving from one end of the non-repeating part until N feature words with N smallest integer values or N smallest hash values are obtained or the sliding window hits the other end of the non-repeating part, where N is an integer such thatN>1.

[0103] In one possible embodiment, found feature words in input data blocks are stored in a data structure, named as FW array, and they could be used to detect similarity between input data blocks and processed data blocks or between different input data blocks.

[0104] FIG. 5 shows another compression method that focuses on how to use a feature word of an input data block for similarity detection.

[0105] S210, calculating hash values for all feature words of one input data block.

[0106] In some possible embodiments, feature words of one input data block are stored in an FW array. If the feature words in the FW array are 32-bit integers, then cyclic redundancy check 32 (or CRC32) could be used to calculate hash values for each element of this array.

[0107] The feature words in the FW array could be replaced by their hash values, and then the array filled with hash values could be renamed as an HV array.

[0108] S220, purging duplicate hash values.

[0109] Limited to CRC32 itself, duplicate hash values may happen due to hash collisions, and it is helpful to purge duplicate hash values from the HV array. As the duplicate hash values are removed, the HV array is filled with unique hash values, and it is named as a UHV array in the following descriptions.

[0110] In some cases, the number of elements in the UHV array corresponding to an input data block is less than a given threshold, and it is less suitable to consider this input data block for similarity detection and delta compression when considering the compression efficiency.

[0111] S230, searching for processed keys that are identical to elements of the UHV array.

[0112] For each processed data block, considered for similarity detection and delta compression, several <key, value> pairs are created and stored in the key-value store. The key in <key, value> pair is the hash value calculated for a feature word of a processed data block (or stored data block), and the value is the processed data block identification (e.g. block address in memory or on a storage device). For convenience, the processed key-value pairs will be further denoted as <ProcessedKey, ProcessedBlockId>, and the hash value stored in the UHV array will be further denoted as InputKey.

[0113] Since a feature word obtained in an input data block may already exist in a processed data block, in other words, there may be repetition in the UHV array and key-value store. Usingeach element of the UHV array as InputKey to look up the key-value store will help to discover the repetition.

[0114] S240, storing blockid of the matched keys.

[0115] If the look-up resulted in matching <ProcessedKey, ProcessedBlockId> pair, that is,5 InputKey is equal to ProcessedKey, then the ProcessedBlockld is store in a data structure SBI (short for selected block identification) array. So, look-ups with elements of the UHV array will fill the SBI array with blockids of already processed data blocks. Equal blockids correspond to the same data blocks.

[0116] Because the same hash value may not correspond to the same feature word, it will10 result in a more accurate match of an NRP if more information is checked.

[0117] Optionally, feature words that are related to the same hash value are identical or not are checked, and they are contained in the processed data block and the input data block respectively.

[0118] Optionally, NRPs that are related to the same hash value are identical or not are15 checked, and they are contained in the processed data block and the input data block respectively.

[0119] S250, determining candidates according to repeated time of each blockid.

[0120] Since more than one ProcessedKey may correspond to one blockid, one blockid may appear more than once in the SBI array. As elements of the SBI array are sorted, the number20 of equal blockids will be counted. The number of some blockids may be more than a given threshold, and the others may not. Processed data blocks indicated by blockids that appear more than the given threshold will be more appropriate for candidates for the delta compression dictionaries to compress input data blocks. Namely, the higher frequency of blockid appears in the SBI array, the bigger the probability is that the respective block has the same NRP, and the25 more effective it will be to delta compress the input data block by using the blockid-indicated data block as a dictionary.

[0121] In some cases, it is a good choice to treat one or more blockslds with maximum count as candidates for dictionaries to delta compress input data blocks. The best candidate among processed blocks for delta compression is the block with the maximum number of■ 30 feature words equal respectively to feature words of the input data block.

[0122] In some cases, there is no blockid with a satisfied appearing time in the SBI array, and then the input data block is not considered for delta compression.

[0123] The given threshold is introduced to guarantee the effectiveness of delta compression. The more equal blockids are in the SBI array, the higher the effectiveness of delta compression will be.

[0124] A blockid repeated in the SBI array means there may be more than one unique hash value corresponding to the blockid, and this may indicate that there may be more than one feature word that exists in both the input data block and the processed data block.

[0125] Equal Feature Words in two different blocks-the dictionary block and the input data block will guarantee the effectiveness of delta compression. In other words, if the LZ-class delta compression algorithm can detect that one NRP with a Feature Word in the input data block is equal to another NRP with an equal Feature Word in the dictionary block, then this NRP becomes repetition with respect to the dictionary block. Therefore, delta compression of input data blocks will result in a better CR compared with compressing the input data block separately.

[0126] FIG. 6 shows diagrams for some key nodes when determining candidates according to feature words obtained in an input data block.

[0127] The FW array contains all feature words obtained from NRPs of an input data block. Since there may be more than one NRP in the input data block, some repetitions may exist in the FW array. The HV array is filled with hash values of feature words in the FW array. Because of repetitions in the FW array and the hash collision, there may be some duplicates in the HV array. As the duplicates are purged from the HV array, a UHV array comes with different unique hash values filled.

[0128] Each element in the UHV array can be used to search for the matched key in the key-value store in which key-value pairs for processed data blocks are stored. Exemplarily, there are ten elements in an MHV (short for matched hash value) array: matched hash value# 1 to matched hash value #10. While matched hash value# 1 is related to Blockld#a, matched hash value#2 to matched hash value#6 are all related to Blockld#b, and matched hash value#? and matched hash value#8 are both related to Blockld#c. So, if blockids in the SBI array are sorted and equal Blockids are counted, then we could get a result like the SBI array in FIG. 6. Numbers in brackets indicate the appearing time of each blockid.

[0129] If we set the rule that blocks indicated by blockids with the maximum appearing time is a dictionary to delta compress the input data block, then we will get a dictionary related to Blockld#c. If we set the rule that blocks indicated by blockids with at least 3 appearing times, then we will get two candidates that are related to Blockld#b and Blockld#c respectively, which should not be explained as a limitation of this application.

[0130] FIG. 7 shows another data compression method that describes the method introduced in FIG. 3 and FIG. 5 generally.

[0131] S310, obtaining a first feature word from a first NRP in a first input data block.

[0132] NRP is short for non-repeating part, also known as literal, and it is a non-repeating byte sequence in an input data block.

[0133] In some cases, the NRP is longer than a given length, like 4 bytes. The feature word is obtained by the sliding window method with a window length shorter than the NRP.

[0134] In some cases, the feature word is the one with the minimum conversion value among all pending words obtained from the NRP, which is calculated by casting pending words to numbers with a uniform format, where a pending word is a sequence of bytes.

[0135] In some cases, before obtaining a first feature word from a first non-repeating part of a first input data block, the first input data block is compressed by the LZ- compression method, and the first non-repeating part is obtained.

[0136] S320, determining if there is one processed data block that contains the feature word that could be used as a dictionary.

[0137] In some cases, the processed data block that could be used as a dictionary contains several unique feature words more than a given threshold, and the number of the unique feature words is determined by the number of unique hash values of feature words obtained from nonrepeating parts of the input data block.

[0138] In some cases, the processed data block is determined by confirming there is at least one second feature word in the processed data block whose hash value is the same as the hash value of the first feature word.

[0139] In some cases, the processed data block is determined by confirming there is at least a third feature word in the processed data block that is identical to the first feature word.

[0140] In some cases, the processed data block contains at least M feature words that areequal respectively to the feature words of the input data block, where M is an integer such thatM>1.

[0141] Optionally, a second input data block that contains the first feature word could be found.5

[0142] S330, using the dictionary to delta compress the input data block or jointly delta compressing the first input data block and a second input data block that contains the first feature word.

[0143] The processed data block described in S320 that is selected by methods this application provided owns a much bigger possibility that has identical feature words or NRPs 10 to the input data block, so, by using this selected processed data block as a dictionary, a better CR and compression efficiency will be achieved.

[0144] S340, storing the feature word as <key, blockid > for future similarity detection.

[0145] Whether the processed data block related to the feature word is found or not, it is beneficial to keep the feature word for future similarity detection. In some cases, feature words 15 are stored as <key, blockld>, where the key is the hash value of the feature word and the input data blockid (e.g. address of the input data block in memory or on storage device).

[0146] FIG. 8 is a schematic block diagram of an electronic device 800 according to an embodiment of this application. As shown in FIG. 8, the electronic device 800 includes: an obtaining module 801 and a determining module 802.20

[0147] The obtaining module 801 is configured to obtain a feature word from an NRP in an input data block.

[0148] The obtaining module 801 is configured to obtain a first feature word from a first non-repeating part of a first input data block. The determining module 802 is configured to delta compress the first input data block by using a processed data block that contains the first feature 25 word as a dictionary or jointly delta compressing the first input data block and a second input data block that contains the first feature word.

[0149] The non-repeating part is longer than a given length. The obtaining module 801 is further configured to obtain the first feature word by the sliding window method with a window r length no longer than the first non-repeating part.30

[0150] Optionally, the first feature word is the one with the minimum conversion valueamong all pending words obtained from the non-repeating part, and the conversion value is calculated by casting pending words to numbers with a uniform format, where a pending word is a sequence of bytes.

[0151] Optionally, the first feature word is selected according to rolling hash values that are calculated for bytes inside a sliding window, and the first feature word is a sequence of bytes with a minimum hash value.

[0152] Optionally, the first feature word is a sequence of bytes at the beginning or the end of the non-repeating part.

[0153] Optionally, the first feature word is a sequence of bytes with a smaller hash value selected from the sequence of bytes at the beginning and the end of the non-repeating part.

[0154] In some cases, more than one feature word is selected from the single non-repeating part of an input data block.

[0155] Optionally, all words contained in the non-repeating part are cast to integers, and the feature words are words with N smallest integer values, where N is an integer such that N>1.

[0156] Optionally, a sliding window is moving from one end of the non-repeating part until N feature words with N smallest integer values or N smallest hash values are obtained or the sliding window hits the other end of the non-repeating part, where N is an integer such that N>1.

[0157] In some cases, the number of unique hash values is more than a first threshold, and they are calculated for feature words obtained from non-repeating parts of the first input data block.

[0158] In some cases, before delta compressing the first input data block by using a processed data block that contains the first feature word as a dictionary, the determining module 802 is further configured to determine the processed data block by confirming there is at least one second feature word in the processed data block whose hash value is the same as the hash value of the first feature word.

[0159] In some cases, before delta compressing the first input data block by using a processed data block that contains the first feature word as a dictionary, the determining module 802 is further configured to determine the processed data block by confirming there is at least a third feature word in the processed data block that is identical to the first feature word.

[0160] In some cases, the processed data block contains at least M feature words that areequal respectively to the feature words of the input data block, where M is an integer such thatM>1.

[0161] In some cases, before obtaining a first feature word from a first non-repeating part of a first input data block, the determining module 802 is further configured to compress the first input data block by the LZ- compression method and obtain the first non-repeating part.

[0162] In some cases, the determining module 802 is further configured to store the first feature word in the key-value store as <key, value> pair, where the key is the hash value of the first feature word, and the value is the data block address that contains the first feature word in a memory.

[0163] FIG. 9 shows a schematic block diagram of an electronic device 900 according to an embodiment of this application. As shown in FIG. 9, the electronic device 900 may include a transceiver 901, a processor 902, and a memory 903. The memory 903 may be configured to store codes, instructions, and the like executed by the processor 902.

[0164] It should be understood that the processor 902 may be an integrated circuit chip and has a signal processing capability. In an implementation process, steps of the foregoing method embodiments may be completed by using a hardware integrated logic circuit in the processor, or by using instructions in the form of software. The processor may be a general purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or another programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The processor may implement or perform the methods, the steps, and the logical block diagrams that are disclosed in the embodiments of the present invention. The general purpose processor may be a microprocessor, or the processor may be any conventional processor or the like. The steps of the methods disclosed with reference to the embodiments of the present invention may be directly performed and completed by a hardware decoding processor, or may be performed and completed by using a combination of hardware in the decoding processor and a software module. The software module may be located in a mature storage medium in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, or a register. The storage medium is located in thememory, and the processor reads information in the memory and completes the steps of the foregoing methods in combination with hardware in the processor.

[0165] It may be understood that the memory 603 in the embodiments of the present invention may be a volatile memory or a nonvolatile memory, or may include both a volatile memory and a nonvolatile memory. The nonvolatile memory may be a read-only memory (Read-Only Memory, ROM), a programmable read-only memory (Programmable ROM, PROM), an erasable programmable read-only memory (Erasable PROM, EPROM), an electrically erasable programmable read-only memory (Electrically EPROM, EEPROM), or a flash memory. The volatile memory may be a random access memory (Random Access Memory, RAM) and is used as an external cache. By way of example rather than limitation, many forms of RAMs may be used, and are, for example, a static random access memory (Static RAM, SRAM), a dynamic random access memory (Dynamic RAM, DRAM), a synchronous dynamic random access memory (Synchronous DRAM, SDRAM), a double data rate synchronous dynamic random access memory (Double Data Rate SDRAM, DDR SDRAM), an enhanced synchronous dynamic random access memory (Enhanced SDRAM, ESDRAM), a synchronous link dynamic random access memory (Synchronous link DRAM, SLDRAM), and a direct rambus random access memory (Direct Rambus RAM, DR RAM).

[0166] It should be noted that the memory in the systems and the methods described in this specification includes but is not limited to these memories and may be a memory of any other appropriate type.

[0167] An embodiment of this application further provides a system chip, where the system chip includes an input / output interface, at least one processor, at least one memory, and a bus. The at least one memory is configured to store instructions, and the at least one processor is configured to invoke the instructions of the at least one memory to perform operations performed in the methods in the foregoing embodiments.

[0168] An embodiment of this application further provides a computer storage medium, where the computer storage medium may store a program instruction for performing the steps performed in the foregoing methods.

[0169] Optionally, the storage medium may be specifically the memory 903.

[0170] An embodiment of this application further provides a computer program product,where when the computer program product is run on an electronic device, the electronic device is enabled to perform the steps performed by the electronic device 900 in the foregoing methods.

[0171] A person of ordinary skill in the art may be aware that, in combination with the examples described in the embodiments disclosed in this specification, units and algorithm steps can be implemented by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are performed by hardware or software depends on particular applications and design constraints of the technical solutions. A person skilled in the art may use different methods to implement the described functions for each particular application, but it should not be considered that the implementation goes beyond the scope of this application.

[0172] It may be clearly understood by a person skilled in the art that, for the purpose of convenient and brief description, for a detailed working process of the foregoing system, apparatus, and unit, refer to a corresponding process in the foregoing method embodiment. Details are not described herein again.

[0173] In the several embodiments provided in this application, it should be understood that the disclosed system, apparatus, and method may be implemented in other manners. For example, the described apparatus embodiment is merely an example. For example, the unit division is merely logical function division and may be other division in actual implementation. For example, a plurality of units or components may be combined or integrated into another system, or some features may be ignored or not performed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections may be implemented through some interfaces. The indirect couplings or communication connections between the apparatuses or units may be implemented in electronic, mechanical, or other forms.

[0174] The units described as separate parts may be or may not be physically separate, and parts displayed as units may be or may not be physical units, may be located in one position, or may be distributed on a plurality of network units. Some or all of the units may be selected based on actual requirements to achieve the objectives of the solutions of the embodiments.

[0175] In addition, functional units in the embodiments of this application may be integrated into one processing unit, or each of the units may exist alone physically, or two or more units are integrated into one unit.

[0176] When the functions are implemented in a form of a software functional unit and sold or used as an independent product, the functions may be stored in a computer readable storage medium. Based on such an understanding, the technical solutions in this application essentially, or the part contributing to the prior art, or some of the technical solutions may be implemented in a form of a software product. The computer software product is stored in a storage medium, and includes several instructions for instructing a computer device (which may be a personal computer, a server, a network device, or the like) to perform all or some of the steps of the methods described in the embodiments of this application. The foregoing storage medium includes: any medium that can store program code, such as a USB flash drive, a removable hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disc.

[0177] The foregoing descriptions are merely specific implementations of this application but are not intended to limit the protection scope of this application. Any variation or replacement readily figured out by a person skilled in the art within the technical scope disclosed in this application shall fall within the protection scope of this application. Therefore, the protection scope of this application shall be subject to the protection scope of the claims.

Claims

CLAIMSWhat is claimed is:

1. A method for data compression, comprising: obtaining one or more feature words from at least one non-repeating part of an input data block, wherein a feature word is a sequence of bytes; and determining at least one dictionary for delta compressing the input data block from processed data blocks according to the feature words.

2. The method according to claim 1, wherein obtaining one or more feature words from at least one non-repeating part of an input data block, comprises: obtaining one feature word from each non-repeating part of the input data block if the input data block includes more than one non-repeating part; or obtaining more than one feature word from one non-repeating part if the input data block only includes the non-repeating part.

3. The method according to claim 2, wherein obtaining one feature word from each nonrepeating part of an input data block if the input data block includes more than one nonrepeating part, comprises: obtaining multiple pending words by a sliding window from each non-repeating part, wherein a pending word is a sequence of bytes; and selecting the feature word from the pending words by casting them to integers, wherein the feature word is a pending word with a minimum integer value.

4. The method according to claim 2, wherein obtaining one feature word from each nonrepeating part of the input data block if the input data block includes more than one nonrepeating part, comprises: calculating rolling hash values for bytes inside a sliding window; and selecting the feature word according to the rolling hash values, wherein the feature word is a sequence of bytes with a minimum hash value.

5. The method according to claim 2, wherein if the input data block includes more than one non-repeating part, the feature word is a sequence of bytes at the beginning or the end ofthe non-repeating part.

6. The method according to claim 5, wherein the feature word is the one with a smaller hash value.

7. The method according to claim 2, wherein obtaining more than one feature word from one non-repeating part if the input data block only includes the non-repeating part, comprises: selecting the feature words from all words contained in the non-repeating part by casting them to integers, wherein the feature words are words with N smallest integer values.

8. The method according to claim 2, wherein obtaining more than one feature word from one non-repeating part if the input data block only includes the non-repeating part, comprises: selecting the feature words by a sliding window sliding from one end of the non-repeating part until N feature words with N smallest integer values or N smallest hash values are obtained or the sliding window hits the other end of the non-repeating part, wherein N is an integer such that N>l.

9. The method according to any one of claims 1 to 8, wherein the input data block contains multiple feature words that are corresponding to at least N different hash values, wherein N is an integer such that N>1.

10. The method according to any one of claims 1 to 9, wherein the input data block contains at least N different feature words, wherein N is an integer such that N>1.

11. The method according to any one of claims 1 to 10, wherein the dictionary contains at least M feature words that are equal respectively to the feature words of the input data block, wherein M is an integer such that M>1.

12. The method according to claim 11, wherein determining at least one dictionary for delta compressing the input data block from processed data blocks according to the feature words, comprises: determining one dictionary for delta compressing the input data block from processed data blocks according to the feature words, wherein the dictionary contains the most feature words that are equal respectively to the feature words of the input data block.

13. The method according to any one of claims 1 to 12, wherein the method further comprises: in case that the input data block contains at least N different feature words, storing hashvalues of the feature words as keys and a storage address of the input data block as a value in <key, value> pairs, wherein N is an integer such that N>1.

14. An electronic device, comprising: an obtaining module configured to obtain one or more feature words from at least one nonrepeating part of an input data block, wherein a feature word is a sequence of bytes; and a determining module configured to determine at least one dictionary for delta compressing the input data block from processed data blocks according to the feature words.

15. The electronic device according to claim 14, wherein the obtaining module is further configured to: obtain one feature word from each non-repeating part of the input data block if the input data block includes more than one non-repeating part; or obtain more than one feature word from one non-repeating part if the input data block only includes the non-repeating part.

16. The electronic device according to claim 15, wherein if the input data block includes more than one non-repeating part, the obtaining module is further configured to obtain multiple pending words by a sliding window from each non-repeating part, wherein a pending word is a sequence of bytes; and the determining module is further configured to select the feature word from the pending words by casting them to integers, wherein the feature word is a pending word with a minimum integer value.

17. The electronic device according to claim 15, wherein if the input data block includes more than one non-repeating part, the determining module is further configured to calculate rolling hash values for bytes inside a sliding window and select the feature word according to the rolling hash values, wherein the feature word is a sequence of bytes with a minimum hash value.

18. The electronic device according to claim 15, wherein if the input data block includes more than one non-repeating part, the feature word is a sequence of bytes at the beginning or the end of the non-repeating part.

19. The electronic device according to claim 18, wherein the feature word is the one with a smaller hash value.

20. The electronic device according to claim 15, wherein if the input data block includes only one non-repeating part, the determining module is further configured to select the feature words from all words contained in the non-repeating part by casting them to integers, wherein the feature words are words with N smallest integer values.

21. The electronic device according to claim 15, wherein if the input data block includes only one non-repeating part, the determining module is further configured to select the feature words by a sliding window sliding from one end of the non-repeating part until N feature words with N smallest integer values are obtained or the sliding window hits the other end of the nonrepeating part, wherein N is an integer such that N>1.

22. The electronic device according to any one of claims 14 to 21, wherein the input data block contains multiple feature words that are corresponding to at least N different hash values, wherein N is an integer such that N>1.

23. The electronic device according to any one of claims 14 to 22, wherein the input data block contains at least N different feature words, wherein N is an integer such that N>1.

24. The electronic device according to any one of claims 14 to 23, wherein the dictionary contains at least M feature words that are equal respectively to the feature words of the input data block, wherein M is an integer such that M>1.

25. The electronic device according to claim 24, the determining module is further configured to determine one dictionary for delta compressing the input data block from processed data blocks according to the feature words, wherein the dictionary contains the most feature words that are equal respectively to the feature words of the input data block.

26. The electronic device according to any one of claims 13 to 25, wherein the determining module is further configured to store hash values of the feature words as keys and a storage address of the input data block as a value in <key, value> pairs in case that there are at least N different feature words contained in the input data block, wherein N is an integer such that N>1.

27. A computer readable storage medium, wherein the computer readable storage medium stores instructions, and when the instructions are run on a server, the server is enabled to perform the method according to any one of claims 1 to 13.

28. An electronic device, comprising a memory and a processor, wherein the memory is configured to store a computer program, and the processor is configured to invoke the computerprogram from the memory and run the computer program, so that a server on which a chip is disposed performs the method according to any one of claims 1 to 13.

29. A computer program product, wherein when the computer program product is run on a server, the server is enabled to perform the method according to any one of claims 1 to 13.