Data compression method and electronic device

Abstract

Embodiments of the present application provide a method and an electronic device for similarity detection and delta compression. The method comprises: calculating a hash value of one or more byte sets obtained from the sub-blocks of an input block according to a preset rule; finding a candidate dictionary for the delta compression of the input block in the processed blocks according to the calculated hash value; priority sorting the candidate dictionaries according to the number of hash values matched with the input block; selecting a fixed number of optimal candidate dictionaries (first block) for the delta compression, and performing the delta compression; determining an effective dictionary (second block) for the delta compression by the number of matching strings between each candidate dictionary and the input block for the delta compression during the process of the delta compression, the number being greater than a threshold value; and prefixing the input block after the delta compression with the ID of the effective dictionary and further storing or transmitting. According to the technical solution, the method can reduce the calculation cost and improve the compression efficiency.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the field of data similarity in data storage and transmission, and more particularly, to a data compression method and an electronic device, which improves the differential compression of the electronic device in data storage and data transmission. BACKGROUND

[0002] With the development of computer technology, the amount of data that needs to be stored or transmitted has increased dramatically. As a process of reducing the size of data without losing information, lossless compression is widely used to minimize the amount of data stored and transmitted.

[0003] Through similarity detection, similar data can be deduplicated or differentially compressed, and the efficiency of data storage and transmission can be further improved. Known similarity detection methods are very costly and time-consuming. SUMMARY

[0004] Embodiments of the present application provide a data similarity detection method, which improves the differential compression of the electronic device in data storage and data transmission.

[0005] According to a first aspect, embodiments of the present application provide a method for data similarity detection, comprising: dividing an input block into a plurality of input partitions, wherein each input partition in the plurality of input partitions includes X bytes; selecting one or more byte sets from the X bytes of each input partition in the plurality of input partitions according to a preset rule, wherein the one or more byte sets include Y bytes selected from the X bytes that are not continuous in the original order; based on the one or more byte sets, calculating one or more input hash values for each input partition in the plurality of input partitions; and matching the input block with other blocks according to the calculated hash values of the one or more partitions to determine their similarity. Then, one or more best similarity candidate blocks found in this way are used as a dictionary to differentially compress the input block.

[0006] According to the above technical solution, the input block is divided into a plurality of input partitions, and a small number of bytes (byte sets) are selected from each input partition for hash calculation. Compared with using all bytes in the partition for hash calculation, selecting byte sets can reduce the computational cost involved in hash calculation. The calculated hash values can be used to match the input block with other blocks to determine their similarity. Then, one or more best similarity candidate blocks found in this way are used as a dictionary to differentially compress the input block.

[0007] In an alternative implementation, when the number of byte sets is multiple for each input chunk, several byte sets are selected from each chunk, and the number of corresponding hash values is calculated for each chunk. The first byte of each selected byte set is offset from the beginning of the chunk by 0, 1, 2, 4, …, 2 m , where m is a natural number unique to different byte sets.

[0008] According to the above technical solution, if multiple byte sets are selected in an input chunk, or multiple input hash values are calculated for an input chunk, the selected byte sets can be shifted by 0, 1, 2, 4, …, 2 m , which can help detect similar features shifted due to insertion and deletion of 2 m bytes, which is quite frequent due to the ubiquitous 8-bit, 16-bit, 32-bit, and 64-bit data types in modern SW, thus improving similarity detection between chunks and thus improving the delta compression ratio.

[0009] In an alternative implementation, the preset rule includes that the intervals between the discontinuous Y bytes are determined by a modified Fibonacci sequence, a logarithmic function, or an exponential function. The modified Fibonacci sequence is a Fibonacci sequence without the first 1, such as 1, 2, 3, 5, …

[0010] In an alternative implementation, the preset rule includes that for any two bytes in each of the byte sets, the two bytes are selected from non-adjacent positions in the input chunk.

[0011] According to the above technical solution, the bytes are selected with intervals, which can reduce the number of selected bytes and the computational cost involved in hash calculation and similarity detection.

[0012] In an alternative implementation, similarity detection of the input chunk with other chunks according to the input hash values of the multiple input chunks includes determining one or more first chunks according to the input hash values, where the one or more first chunks are processed chunks, and each of the one or more first (processed) chunks includes one or more first chunks with the same hash value as one of the input hash values. Then, the one or more best similarity candidate chunks found in this way are used as a dictionary for delta compression of the input chunk.

[0013] According to the above technical solution, the input hash values calculated for the input chunks of the input chunk are used to find information about processed chunks in a key-value store, where the processed chunks include chunks with the same hash value.

[0014] The similarity detection of the input chunk with one or more processed data chunks is performed by looking for one or more pairs of input and processed chunks, each pair comprising an input chunk and a processed chunk, whose input and processed hash values are equal, each chunk being associated with a respective data chunk identifier (also referred to as input chunk ID and processed chunk ID), by comparing the input hash value with the hash values computed for the processed chunks of the data chunks (further referred to as processed hash values and processed chunks, respectively).

[0015] The processed chunks whose hash values are equal to the input hash value are ordered by the processed chunk IDs, so that the respective processed chunk IDs whose hash values are equal to the input chunk's hash value are ordered by the number of chunks.

[0016] One or more processed chunks are selected as candidate dictionaries for the delta compression of the input chunk, wherein the one or more processed chunks match the largest number of chunks with the ordered chunk IDs and whose hash values are equal to the input chunk's hash value.

[0017] The input chunk is delta compressed using the selected processed chunks as candidate dictionaries.

[0018] During the delta compression, it is determined whether each candidate dictionary comprises a byte string equal to a byte string in the input chunk, so that the delta compression is valid, i.e. the byte string in the input chunk is encoded as a reference to the equal byte string in the dictionary chunk, whose number of bits is smaller than the number of bits in the respective byte string.

[0019] Dictionary candidates that do not comprise any byte string for a valid delta compression of the input chunk are discarded.

[0020] Dictionary candidates that comprise a byte string for a valid delta compression of the input chunk are kept as valid dictionaries for further references.

[0021] The result of the delta compression of the input chunk and the processed chunk IDs of the valid dictionaries are stored.

[0022] A subset of the input hash values to be stored as processed hash values is selected, the subset of input hash values being associated with respective chunk IDs for future similarity detection between the processed chunks and future input chunks, so that the subset comprises only distinct hash values; in addition, the subset comprises one or more minimum hash values, one or more maximum hash values.

[0023] The selected input hash values associated with the input chunk IDs are stored for later use in the appropriate data structure for similarity detection between the processed chunks and future input chunks for fast lookup by hash as a key to obtain the processed chunk IDs and retrieve the respective processed chunks as candidate dictionaries for the delta compression of the input chunk.

[0024] According to a second aspect, an electronic device is provided. The electronic device comprises: a dividing module configured to divide an input block into a plurality of input sub-blocks, wherein each of the plurality of input sub-blocks comprises X bytes; a selecting module configured to select one or more byte sets from the X bytes of each of the plurality of input sub-blocks according to a preset rule, wherein the one or more byte sets comprise Y bytes selected from the X bytes in a non-continuous order; a calculating module configured to calculate one or more input hash values for each of the plurality of input sub-blocks based on the one or more byte sets; and a compressing module configured to compress the input block according to the input hash values calculated by the calculating module for the plurality of input sub-blocks.

[0025] According to the above technical solution, the dividing module is configured to divide the input block into one or more sub-blocks. The selecting module is configured to select one or more byte sets for hash calculation by the calculating module. The hash calculation result is used by the compressing module to compress the input block. Selecting a small number of bytes instead of the entire sub-block for hash calculation reduces the calculation cost involved in the compression process and improves the compression efficiency.

[0026] In an optional implementation, when the number of byte sets is multiple for each input sub-block, the difference between the offset0 position of a first byte set and any of the other byte sets is 2 m wherein the m is a natural number unique to different byte sets, the first byte set is one of the plurality of byte sets, and the offset0 position is the position of the first byte of each of the one or more byte sets.

[0027] In an optional implementation, the preset rule comprises that the difference between the byte position of the non-continuous Y bytes and the offset0 position satisfies a modified Fibonacci sequence or a logarithmic function sequence.

[0028] In an optional implementation, the compressing module is specifically configured to: determine one or more first blocks according to the input hash values, wherein the one or more first blocks are processed blocks, and each of the one or more first blocks comprises one or more first sub-blocks with the same hash value as one of the input hash values; and compress the input block based on the one or more first blocks.

[0029] In an optional implementation, the compression module is configured to: determine the one or more first sub-blocks based on the input hash values, wherein each of the one or more first sub-blocks is a processed sub-block and has a hash value that is the same as one of the input hash values; and determine the one or more first blocks based on the one or more first sub-blocks, wherein the one or more first blocks are blocks divided into the one or more first sub-blocks.

[0030] In an optional implementation, the one or more first sub-blocks are determined based on a key-value store, wherein the key-value store stores associations between hash values of the processed sub-blocks and corresponding sub-block identifiers and block identifiers.

[0031] In an optional implementation, a number of the one or more first sub-blocks is greater than a first preset threshold.

[0032] In an optional implementation, the electronic device further includes a determination module configured to determine one or more second blocks from the one or more first blocks according to lengths of matching strings between the one or more first blocks and the input block, wherein the lengths of the matching strings are greater than a second preset threshold.

[0033] In an optional implementation, a result of the compressed input block includes the block identifiers of the one or more second blocks.

[0034] In an optional implementation, the electronic device further includes an update module configured to update the key-value store using the sub-block identifiers of the input sub-blocks, the block identifiers, and corresponding input hash values.

[0035] According to a third aspect, an embodiment of the present application provides a data compression method, including: dividing an input block into a plurality of input sub-blocks, wherein each of the plurality of input sub-blocks includes X bytes; selecting one or more byte sets from the X bytes of each of the plurality of input sub-blocks according to a preset rule, wherein the one or more byte sets include Y bytes that are discontinuous in an original order and are selected from the X bytes; calculating one or more input hash values for each of the plurality of input sub-blocks based on the one or more byte sets; and compressing the input block according to the input hash values of the plurality of input sub-blocks.

[0036] According to the technical solution, the input block is divided into multiple input sub-blocks, and a small number of bytes (byte set) are selected from the divided input sub-blocks for hash calculation. Instead of selecting the sub-block as a whole for hash calculation to obtain the feature, the selection of the byte set can reduce the calculation cost involved in the hash calculation. The calculated input hash value can be used to compress the input block.

[0037] Optionally, one or more byte sets can be selected for hash calculation as a whole, and then the hash value of the processed block can be used to compress the input block.

[0038] In an optional implementation, when the number of byte sets is multiple for each input sub-block, the difference between the offset0 position of the first byte set and any one of the other byte sets is 2 m , where m is a natural number unique to different byte sets, the first byte set is one of the multiple byte sets, and the offset0 position is the position of the first byte of each byte set in the one or more byte sets.

[0039] According to the technical solution, if multiple byte sets are selected in one input sub-block or multiple input hash values are calculated for one input sub-block, the difference between the offset0 position of the first byte set and any one of the other byte sets is 2 m , in other words, the selected bytes may be shifted by 2 m bytes for different byte sets, which can help detect similar features shifted by inserting and deleting 2 m bytes, which is quite frequent due to the widespread 8-bit, 16-bit, 32-bit, and 64-bit data types in modern SW, thus improving the compression ratio. Optionally, the offset0 position of the first byte set is 0, the offset0 position of the second byte set is 1, the offset0 position of the third byte set is 2, and so on. In other words, the difference between the offset0 positions of the first byte set and the subsequent byte sets is 1, 2, 4, 8, 16, and so on.

[0040] In an optional implementation, the preset rule includes that the difference between the byte position of the discontinuous Y bytes and the offset0 position satisfies a modified Fibonacci sequence or a logarithmic function sequence.

[0041] According to the technical solution, the difference between the byte positions of the byte sets satisfies a modified Fibonacci sequence or a logarithmic function sequence or an exponential function sequence. The bytes are selected at intervals so that only a small number of bytes are selected for hash calculation, thereby reducing the calculation cost and improving the compression speed.

[0042] In an optional implementation, the preset rule comprises: for any two bytes in each of the byte sets, the two bytes are selected from non-adjacent positions in the input chunks.

[0043] According to the above technical solution, the bytes are selected at intervals, which can reduce the number of selected bytes and the calculation cost involved.

[0044] In an optional implementation, the compressing the input block according to the input hash values of the plurality of input chunks comprises: determining one or more first blocks according to the input hash values, wherein each of the one or more first blocks is a processed block, and each of the one or more first blocks comprises one or more first sub-chunks having the same hash value as one of the input hash values; and compressing the input block based on the one or more first blocks.

[0045] According to the above technical solution, the input hash values calculated for the input chunks of the input block are used to find blocks in the key-value store, and the blocks comprise sub-chunks having the same hash value.

[0046] For example, the first block comprises all blocks corresponding to matched records in the key-value store.

[0047] For another example, the first block can be selected from blocks corresponding to records obtained from the key-value store.

[0048] For another example, blocks corresponding to records obtained from the key-value store can be assigned a priority level according to the number of sub-chunks each block has. Then, if two sub-chunks in different blocks are obtained from the key-value store according to the same hash value, the sub-chunk with a lower priority in the block will be cleared because the sub-chunk with a higher priority in the block can compress a specific input sub-chunk in the input block having the same hash value. The blocks are then reordered according to the number of remaining sub-chunks. The reordered blocks can be regarded as the first blocks that are likely to perform differential compression on the input block. This operation will improve the quality of the first blocks as candidate dictionary blocks.

[0049] In an optional implementation, the determining one or more first blocks according to the input hash values comprises: determining the one or more first sub-chunks based on the input hash values, wherein each of the one or more first sub-chunks is a processed sub-chunk and has the same hash value as one of the input hash values; and determining the one or more first blocks based on the one or more first sub-chunks, wherein the one or more first blocks are blocks divided into the one or more first sub-chunks.

[0050] In an optional implementation, the one or more first chunks are determined based on a key-value store, wherein the key-value store stores associations of hash values of the processed chunks and corresponding chunk identifiers and block identifiers.

[0051] In an optional implementation, the number of the one or more first chunks is greater than a first preset threshold.

[0052] According to the above technical solution, the number of chunks included in the first block is greater than the first preset threshold, which improves the possibility of the first block being used for compressing the input block and improves the compression ratio.

[0053] In an optional implementation, after the input block is compressed based on the one or more first blocks, the method comprises: determining one or more second blocks from the one or more first blocks according to lengths of matching strings between the one or more first blocks and the input block, wherein the length of the matching string is greater than a second preset threshold.

[0054] According to the above technical solution, the second block is selected from the first block according to the string matching result, and only the first block with the length of the matching string greater than the second preset threshold is considered as the second block. The second block is a block that can be used for compressing the input block.

[0055] In an optional implementation, the result of the compressed input block includes the block identifier of the one or more second blocks.

[0056] According to the above technical solution, the compressed input block includes the block identifier of the second block, for example, the compressed input block is prefixed with the block identifier of the second block, which can be used for compression of the input block.

[0057] In an optional implementation, the method further comprises: updating the key-value store using the chunk identifier of the input chunk, the block identifier, and the corresponding input hash value.

[0058] According to the above technical solution, after the input block is compressed, the key-value store is updated using the chunk identifier of the input chunk, the block identifier, and the corresponding input hash value, so that when a subsequent block is compressed, the chunk identifier and the block identifier of the input chunk can be used for the similarity detection process.

[0059] According to a fourth aspect, the embodiments of the present application provide a computer readable storage medium including instructions. When the instructions run on a computer, the computer executes the method in the first aspect or any optional implementation of the first aspect.

[0060] According to a fifth aspect, there is provided an electronic device comprising a processor and a memory. The processor is connected to the memory. The memory is configured to store instructions, and the processor is configured to execute the instructions. When the processor executes the instructions stored in the memory, the processor is caused to perform the method in the first aspect or any of the optional implementations of the first aspect.

[0061] According to a sixth aspect, there is provided a chip system, wherein the chip system comprises a memory configured to store a computer program and a processor configured to invoke the computer program from the memory and run the computer program, so as to cause an electronic device in which the chip system is located to perform the method in the first aspect or any of the optional implementations of the first aspect.

[0062] According to a seventh aspect, there is provided a computer program product, which, when running on an electronic device, causes the electronic device to perform the method in the first aspect or any of the optional implementations of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0063] Figure 1 A flowchart showing an embodiment of the method 100 for data compression is shown.

[0064] Figure 2 An embodiment of selecting one byte set for each chunk in an input block is shown.

[0065] Figure 3 An embodiment of selecting more than one byte set for chunk 0 in Figure 1

[0066] Figure 4 is a schematic diagram of a key-value store.

[0067] Figure 5 is a schematic diagram describing the delta compression process provided by the present application.

[0068] Figure 6 is a schematic diagram describing the delta compression process.

[0069] Figure 7 An input block based key-value store update result is shown.

[0070] Figure 8 A flowchart of an embodiment of the method 700 for data compression is shown.

[0071] Figure 9 A flowchart of an embodiment of the method 800 for data compression is shown.

[0072] Figure 10 is a schematic block diagram of an electronic device 900.​

[0073] Figure 11 is a schematic block diagram of the electronic device 1000. DETAILED DESCRIPTION

[0074] The technical solutions in the present application are described below with reference to the accompanying drawings.

[0075] Definitions:

[0076] Unless otherwise stated or implied by context, the following terms and phrases have the meanings provided below.

[0077] The term "chunk" refers to a sequence of bytes having a predetermined length.

[0078] The term "chunklet" refers to a sequence of bytes having a predetermined length that is less than the predetermined length of a chunk. In some cases, a chunklet is a sequence of bytes having the same length as a chunk when the entire chunk is considered as one chunklet.

[0079] The term "byte" refers to a sequence of 8 bits, where a bit is the smallest unit in the field of computer science.

[0080] The phrase "hash value" refers to the output returned after applying a hash function algorithm. The phrases "hash function algorithm" and "hash function" refer to an algorithm or subroutine that maps large datasets (optionally, of variable length) to smaller datasets of fixed size for a particular hash function. The value returned by the algorithm can also be referred to as a hash function value, hash code, hash sum, checksum, or hash value.

[0081] Data compression, source coding, or bitrate reduction is the process of encoding information using fewer bits than the original representation. Any particular compression is either lossless or lossy. Lossless compression reduces bits by identifying and eliminating statistical redundancy. Lossless compression does not lose any information. Lossy compression reduces bits by deleting unnecessary or less important information. Typically, a device that performs data compression is called an encoder, and a device that performs the inverse of the data compression (decompression) is called a decoder.

[0082] Compression is very useful because it can reduce the resources needed to store and transmit data. The compression and decompression processes consume computational resources. There is a space-time complexity tradeoff in data compression. The design of a data compression scheme involves a balance between various factors, including the degree of compression, the amount of distortion introduced (when lossy data compression is used), and the computational resources needed to compress and decompress the data.

[0083] Delta encoding is a lossless compression method that is a way of storing or transmitting data in the form of the difference (delta) between sequential data rather than the full file; more generally, this is called data differencing. Delta encoding is sometimes called delta compression, especially in cases where a history of changes is needed (for example, in revision control software).

[0084] Data compression ratio or compression ratio (CR), also known as compression power, is a measure of the relative reduction in size of the data representation produced by a data compression algorithm. It is usually expressed as the uncompressed size divided by the compressed size, denoted as:

[0085]

[0086] When delta compression is applied, the input data is divided into smaller units, and a hash value is calculated for these units. The hash value is considered as the fingerprint of these units. Therefore, for units with the same hash value, one original unit and one delta value can be stored instead of two full units.

[0087] However, there is a balance between the number of hash values that need to be calculated and the calculation cost of each calculated hash value. That is, when the byte / bit of the divided unit is less, the input data is divided into more units, thereby increasing the overall calculation cost of the hash calculation, and when the byte / bit of the divided unit is more, each unit is longer, so the calculation cost of each hash value is higher. It is important to reduce the calculation cost of compression.

[0088] Embodiments of the present application address the challenge of reducing the calculation cost of the compression algorithm and accelerating the compression algorithm.

[0089] Figure 1 is a flowchart of a data compression method 100 provided by embodiments of the present application. The method 100 includes the following steps.

[0090] S102: dividing an input block into a plurality of input partitions, wherein each input partition in the plurality of input partitions includes X bytes.

[0091] The input block can be received from a host through a wired or wireless network. The data to be compressed can be transmitted through a plurality of blocks, or these blocks can be defined by a device that performs the data compression method. A block refers to a sequence of bytes with a predetermined byte length. The input block is the block being processed. Before the input block, there can be several blocks that have already been processed. After the input block, there can be several blocks to be processed. The block size is not fixed. For example, the size of the block can be 4 kilobytes (KB). The block can include a header to identify the block. The header can include information such as, but not limited to, a block identifier or block ID associated with the block. A block with a block ID of 2 can be denoted as block 2.

[0092] The input block can be divided into multiple input chunks. Chunk refers to a sequence of bytes with a predetermined length smaller than that of the block. For example, an input block of 4KB can be divided into 4*1024B input chunks. In addition, each chunk has a chunk ID for identifying a unique chunk. Therefore, the four input chunks of the input block can be represented as chunk 0, chunk 1, chunk 2, and chunk 3. In order to distinguish the chunks in different blocks, (block ID, chunk ID) is introduced as the chunk unique ID (UID) to help illustrate the embodiments.

[0093] S104: According to a preset rule, one or more byte sets are selected from X bytes of each input chunk in the plurality of input chunks, wherein the one or more byte sets include Y bytes selected from the X bytes and not continuous in the original order.

[0094] In some embodiments, one byte set is selected from each chunk, and the bytes are selected according to a preset rule for all chunks in the data stream. The byte set includes Y bytes that are not continuous from X bytes. In other words, the selected bytes are not continuous. The byte set is selected starting from the offset0 position with an incremental step, wherein the selected bytes are separated by a predefined pattern. For example, the preset rule includes that the difference between the Y bytes that are not continuous and the offset0 position satisfies a modified Fibonacci sequence or a logarithmic function sequence. The logarithmic sequence is a set of real numbers a[i], i = 1, 2,..., such that

[0095] (1) 0 <= a[i] <= 1,

[0096] (2) a[k] + a[i] < a[r] + a[s], if k*l < r*s,

[0097] (3) a[k] + a[i] = a[r] + a[s], if k*l = r*s.

[0098] In other words, the step can be increased in a logarithmic, exponential, or linear manner through a modified Fibonacci or even a self-designed sequence.

[0099] The difference between the Y bytes that are not continuous and the offset0 position is the result of subtracting the offset0 position from the byte position or byte ID of one byte in the Y bytes. Taking the case where the difference satisfies the modified Fibonacci sequence as an example. The input block is a 64-byte string, which is divided into four input chunks, represented as chunk 0, chunk 1, chunk 2, and chunk 3. As shown in Figure 2 each input chunk is a 16-byte string. Figure 2Each number in the Chinese box represents a byte ID or byte position. In this application, "offset0 position", "byte number", "byte ID", "byte position" all refer to the position of a byte in the original input chunk, where the byte is selected from the input chunk. Chunk 0 includes byte 0 to byte 15, chunk 1 includes byte 16 to byte 31, chunk 2 includes byte 32 to byte 47, and chunk 3 includes byte 48 to byte 63.

[0100] The Fibonacci sequence is [1, 1, 2, 3, 5, 8, 13, 21…]. The first 1 is replaced with 0 to avoid repetition. Then the modified Fibonacci sequence will be [0, 1, 2, 3, 5, 8, 13, 21…].

[0101] If the offset0 position is the start position of the input chunk or the byte ID of the offset0 position is 0 in the case of selecting the first byte in Y bytes, since the position difference satisfies the modified Fibonacci sequence, the byte ID of the selected byte for chunk 0 is [0, 1, 2, 3, 5, 8, 13], which is emphasized in Figure 2 According to the modified Fibonacci sequence, the byte ID of the selected byte for chunk 1, chunk 2, and chunk 3 is [16, 17, 18, 19, 21, 24, 29], [32, 33, 34, 35, 37, 40, 45], and [48, 49, 50, 51, 53, 56, 61], respectively.

[0102] If the offset0 position becomes m, since the position difference satisfies the modified Fibonacci sequence, the byte ID of the selected byte for chunk 0 is [m, 1+m, 2+m, 3+m, 5+m, 8+m, 13+m], without considering the case that the byte ID exceeds the maximum byte ID in chunk 0.

[0103] It should be noted that in the above example, the length of the byte set is 7 bytes for each chunk. Alternatively, the selected byte can follow any part of the modified Fibonacci sequence. The length can be limited by a small constant. For example, if the length of the selected byte is 5 bytes, the byte ID of the selected byte for chunk 0 can be [0, 1, 2, 3, 5], [1, 2, 3, 5, 8], or [2, 3, 5, 8, 13]. In addition, the first 1 can be removed to obtain the modified Fibonacci sequence, which will be [1, 2, 3, 5, 8, 13, 21…].

[0104] It should be noted that the preset rule can be that, for any two bytes in each byte set, the two bytes are selected from non-adjacent positions in the block, or the number of continuous bytes selected from the block is no more than n bytes, where n is a preset value. These preset rules will reduce the number of selected bytes and expand the selection range in the block, thereby detecting more features and reducing the cost involved.

[0105] According to a third aspect, the embodiments of the present application provide a computer readable storage medium including instructions. When the instructions are run on a computer, the computer is caused to perform the method in the first aspect or any optional implementation of the first aspect.

[0106] According to a fourth aspect, an electronic device is provided, including a processor and a memory. The processor is connected to the memory. The memory is configured to store instructions, and the processor is configured to execute the instructions. When the processor executes the instructions stored in the memory, the processor is caused to perform the method in the first aspect or any optional implementation of the first aspect.

[0107] According to a fifth aspect, a chip system is provided, including a memory and a processor, wherein the memory is configured to store a computer program, and the processor is configured to call the computer program from the memory and run the computer program, so that an electronic device in which the chip system is located performs the method in the first aspect or any optional implementation of the first aspect.

[0108] According to a sixth aspect, a computer program product is provided, wherein when the computer program product is run on an electronic device, the electronic device is caused to perform the method in the first aspect or any optional implementation of the first aspect.

[0109] In an optional implementation, if only one byte set is selected for a block, the offset0 position, i.e., the start position of the byte set, can be randomly selected. In this implementation, the offset0 position does not necessarily represent the first byte position of the byte set. For example, if the modified Fibonacci sequence is [0, 1, 2, 3, 5, 8, 13, 21…], the offset0 position is the position of the first byte in a byte set. However, if the modified Fibonacci sequence is [1, 2, 3, 5, 8, 13, 21…], the first byte of the byte set is the byte after the offset0 position.

[0110] In another alternative implementation, when the number of byte sets is multiple for each input chunk, the difference between the offsetO position of a first byte set and any of the other byte sets is 2m, where m is a natural number unique to the different byte sets, the first byte set is one of the multiple byte sets, and the offsetO position is the position of the first byte of each of the one or more byte sets. In other words, the difference follows an exponential function with base 2, i.e., the difference of the offsetO positions of different byte sets of one chunk can be 1 byte, 2 bytes, 4 bytes, 8 bytes, 16 bytes, 32 bytes, and so on. The value of m for different byte sets in a chunk can be consecutive integers or non-consecutive integers. The offsetO positions of different byte sets in a chunk are intended to detect similar content shifted by 2m bytes due to insertion and deletion, which is very frequent due to the ubiquitous 8-bit, 16-bit, 32-bit, 64-bit data types in modern SW. For example, if both chunk 0 in block 3 and chunk 1 in block 5 have 16 bytes, and chunk 1 can be achieved by inserting one byte before chunk 0 and deleting the last byte in chunk 0, then the first fifteen bytes in chunk 0 are the same as the last fifteen bytes in chunk 1. If only one byte set is selected for each chunk in the same way, the matching fifteen bytes will not be detected by the selected byte set. However, if two byte sets are selected and the two byte sets are shifted by one byte, then the matching fifteen bytes will be detected by the selection method proposed in this application.

[0111] If one offsetO position (denoted as the reference offsetO position) is the first byte of a chunk, the offsetO position of a byte set is spaced 0 bytes, 1 byte, 2 bytes, 4 bytes, 8 bytes, 16 bytes, and so on from the first byte of the chunk, or the offsetO position is the first byte, the second byte, the third byte, the fifth byte, the ninth byte, the seventeenth byte, and so on of the chunk. For example, if the reference offsetO position is the first byte of a chunk, the offsetO position of a byte set is spaced 0 bytes, 1 byte, 2 bytes, 4 bytes, 8 bytes, 16 bytes, and so on from the first byte of the chunk, or the offsetO position is the first byte, the second byte, the third byte, the fifth byte, the ninth byte, the seventeenth byte, and so on of the chunk. Figure 3As shown, for example, three byte sets are selected for each input chunk. For chunk 0, the byte IDs at offset0 position are 0, 1, 2, which are the 1st byte, 2nd byte, 3rd byte of the chunk respectively. The offset0 position is equal to 0 byte, 1 byte, 2 byte, 4 byte, 8 byte, 16 byte, 32 byte… from the reference offset0 position. The byte IDs of the three byte sets are [0, 1, 2, 3, 5, 8, 13], [1, 2, 3, 4, 6, 9, 14] and [2, 3, 4, 5, 7, 10, 15] respectively. Multiple byte sets are selected for hash calculation to detect the features of the chunk (to be discussed later). In this way, several byte sets are selected for subsequent hash calculation.

[0112] S106: Calculate one or more input hash values for each input chunk in the plurality of input chunks based on the one or more byte sets.

[0113] The hash calculation can be interpreted as mapping the byte set to a scalar value (a number). The calculated scalar value is used as an index to determine a bucket in the key-value store (to be discussed later). The hash calculation is based on the selected byte set. Therefore, if only one byte set is selected for a chunk, one hash value is calculated for the chunk, and if multiple byte sets are selected for a chunk, multiple hash values are calculated for the chunk. However, it should be noted that the multiple hash values for a chunk are not necessarily different. In addition, if different chunks have similar features, their hash values can also be the same.

[0114] The embodiments of the present application do not limit the algorithm of the hash calculation. For example, the hash calculation can use Rabin-Karp, MD, SHA algorithm (Secure Hash Algorithm), crc 32, etc.

[0115] S108: Compress the input block according to the input hash values of the plurality of chunks.

[0116] In some embodiments, compressing the input block according to the input hash values of the plurality of input chunks comprises: determining one or more first blocks according to the one or more input hash values, wherein the one or more first blocks are processed blocks, and each first block in the one or more first blocks comprises one or more first chunks whose hash values are the same as one hash value in the one or more input hash values; compressing the input block based on the one or more first blocks.

[0117] For the convenience of clearly describing the technical solutions in the embodiments of the present application, in the embodiments of the present application, the terms of “first”, “second”, and the like are used to distinguish the same items or similar items with basically the same functions and effects. Those skilled in the art can understand that the terms of “first” and “second” do not limit the quantity or the execution order, and the terms of “first” and “second” do not indicate the difference.

[0118] To obtain the one or more first blocks, it is necessary to look up a key-value store (or hash table) using the input hash value as a key, thereby obtaining the one or more first blocks.

[0119] In this implementation, determining the one or more first blocks according to the input hash value comprises: determining one or more first sub-blocks based on the input hash value, wherein each of the one or more first sub-blocks is a processed sub-block, and the hash value is the same as one of the input hash values; and determining the one or more first blocks based on the one or more first sub-blocks, wherein the one or more first blocks are blocks divided into the one or more first sub-blocks.

[0120] The input hash value is used as a key, and the (block ID, sub-block ID) record is used as a value. The key-value store saves the association of the hash values of the processed sub-blocks and the corresponding (block ID, sub-block ID) records. The data stream to be compressed is composed of multiple blocks, and the processed sub-blocks come from the processed blocks before the input block. The processed blocks have been divided into one or more X-byte sub-blocks, and a Y-byte byte set is selected from the sub-blocks for hash calculation, in the same way as the selection of the byte set for the input sub-block. Assuming that the first block of the data stream to be compressed is represented as block 0, and the input block is block 5, the key-value store saves the association of the records of the sub-blocks from block 0, block 1, block 2, block 3, and block 4. It is obvious that the key-value store will be dynamically updated as the blocks are processed.

[0121] For example, the key-value store can have multiple buckets that save the records of sub-blocks with different hash values. Each bucket saves the records of sub-blocks with the same hash value. Therefore, if a hash value is calculated for a sub-block, the records of 4 sub-blocks in a processed block can be stored in at most 4 buckets of the key-value store.

[0122] Looking up the key-value store using the hash value of the input sub-block as a key can result in multiple buckets. Then all the records in these buckets are used for the input block. The sub-block corresponding to the record determined based on the key-value store is the first sub-block mentioned above. The block mentioned above is the first block that can be used in the compression process of the input block.

[0123] In some embodiments, the records of the chunks are used as the first chunks to be used as the dictionary chunks for compressing the input chunk. In other embodiments, the dictionary chunks are also selected from the first chunks. For example, the number of the one or more first chunks is greater than a first preset threshold, which indicates that the chunks with more similar chunks to the input chunk are qualified as the first chunks.

[0124] To obtain the one or more first chunks of the input chunk, the method can include the following steps:

[0125] (a) Sort the records by chunk ID.

[0126] After the first sorting, the records with the same chunk ID are grouped together, and any group includes the records with the same chunk ID. Among the records in each group, there can be records with the same chunk ID. The duplicate records are removed, and then each group includes the records with the same chunk ID and unique chunk ID.

[0127] (b) Sort the groups by the number of records (chunk ID).

[0128] The chunks corresponding to the groups with more records or more chunks include more features (hash values) of the input chunk. Therefore, after the second sorting, N groups can be selected from the M groups as the first chunks of the input chunk, where N≤M, M is the number of chunks including at least one record, and N is the number of chunks determined from the M chunks based on certain conditions (for example, the first chunk includes at least 2 records).

[0129] After the two sorts, the first chunks are selected as the dictionary chunks of the input chunk, and then the input chunk is compressed based on the first chunks.

[0130] If it needs to be noted that the two sorting processes provided here are only one implementation to determine the first chunks, there can also be other means to determine the first chunks, for example, the number of records corresponding to each chunk can be recorded through a statistical table. In any case, the first chunks will be realized.

[0131] If the number of the first chunks is zero, the input chunk is compressed alone, and the input chunk is stored or transmitted through the network. The following process of this technical solution is: updating the key-value storage using the chunk identifier of the input chunk, the block identifier, and the corresponding input hash value. In other words, the calculated hash value of (chunk ID, chunk ID) and the corresponding record are inserted into the key-value storage for further searching for subsequent input chunks in the first chunk.

[0132] It needs to be noted that if at least two hash values are calculated for a single chunk, the chunks with one of the at least two hash values, rather than the chunks with all of the at least two hash values, are qualified as the candidate dictionary chunks, which can reduce the computational cost of the determination process of the first chunks.

[0133] It is also noted that if at least two sets of bytes are selected for each input chunk, in one possible implementation, one set of hash values can be selected based on one set of bytes for insertion into the key-value store.

[0134] In an optional implementation, after compressing the input chunk based on the one or more first chunks, the method comprises determining one or more second chunks from the one or more first chunks according to the length of matching strings between the one or more first chunks and the input chunk, wherein the length of the matching strings is greater than a second preset threshold.

[0135] In this implementation, the first chunks are the chunks that are likely to be dictionary chunks, but not necessarily useful dictionary chunks for compressing the input chunk. Therefore, the efficiency of the first chunks for compressing the input chunk is evaluated, and the unused first chunks are discarded to reduce storage cost. The number of first chunks is variable. For example, there can be one or more first chunks after hash matching. The one or more first chunks are then copied into a buffer, which is transmitted as input to a delta compressor, followed by the input chunk. The output of the delta compressor includes the delta compressed input chunk and statistical information on how to find multiple substrings (‘words’) from the input chunk that are repeated in each dictionary chunk.

[0136] The statistical information can be collected by further pseudo code:

[0137]

[0138] input_word_in_dict is a pseudo code function that searches for words (or substrings) in the input chunk from the dictionary chunks (dictZ, dictY, dictA), with two parameters, w (word or substring) and dictionary chunk (dictZ, dictY, dictA). According to the pseudo code, the statistical information on how many substrings (‘words’) in the input chunk are repeated in each dictionary chunk is stats[dictZ], stats[dictY] and stats[dictA].

[0139] The second chunk selected from the first chunk based on the specified condition is a chunk that is effective for performing delta compression on the input chunk. The first dictionary chunk that qualifies as the second chunk can be determined by certain rules. For example, the total length of matching strings in the first chunk and the input chunk is greater than a certain threshold. By using the pseudo code above, the string matching results (stats[dictZ], stats[dictY] and stats[dictA]) between the input chunk and the dictionary chunks (dictZ, dictY and dictA) can be obtained, and the second chunk can be determined according to the matching results.

[0140] In some embodiments, the number of second chunks of the input chunk is 0, and the input chunk is compressed independently.

[0141] For example, the number of first chunks is one or more, but none of the first chunks meets the requirement to be a second chunk, i.e., all the first chunks have a matching string length of zero or less than a certain threshold with the input chunk. Therefore, all the first chunks will be discarded.

[0142] In other embodiments, one or more second chunks are available for compressing the input chunk. One or more second chunks are selected from the first chunks, and the one or more second chunks have a matching string length greater than a threshold with the input chunk. In this case, the input chunk can be compressed using the second chunk, and the compression result is prefixed with the ID of the second chunk (second chunk ID).

[0143] It should be noted that the second chunk is a chunk used to compress the input chunk. The input chunk has been compressed in the previous process. The second chunk is determined to evaluate the efficiency of the first chunk, so as to discard the unused first chunk and save the storage cost.

[0144] After determining the second chunk that is evaluated to be available for compressing the input chunk, the result of the compressed input chunk is prefixed with the block identifier of the one or more second chunks.

[0145] Assuming that the ID of the input chunk is chunk 10, and one chunk calculates only one hash value, the following describes the determination process of the first chunk and the second chunk through a case.

[0146] The input chunk is divided into 4 input chunks, and the chunk IDs are chunk 0, chunk 1, chunk 2, and chunk 3, respectively.

[0147] As described previously, the key-value store saves the association of the hash values of the chunks divided from chunk 0 to chunk 9 and the corresponding (chunk ID, chunk ID) records. As shown in Figure 4 , the current key-value store includes multiple buckets corresponding to the records of (chunk ID, chunk ID) with unique hash values, where the chunk ID is between 0 and 9, and the chunk ID is between 0 and 3. For example, the bucket 401 saves the records of (4, 2), (1, 3), and (5, 0), and the corresponding chunks share a common hash value X. The hash values of the bucket 402, the bucket 403, the bucket 404, the bucket 405, and the bucket 406 are U, W, Z, V, and M, respectively. It should be understood that since the 10 chunks from chunk 0 to chunk 9 are divided into 40 chunks in total, and only one hash value is calculated for each chunk, there are 40 records of (chunk ID, chunk ID) in the current key-value store. In other words, there are multiple buckets and corresponding records Figure 4 not shown in Figure 4 . Since these records do not affect the determination of the first chunk and the second chunk, these records are omitted in .

[0148] The first block can then be selected from the processed blocks according to the hash value computed for the input block. Specifically, for chunk 0, the hash value X matches bucket 401, and all records in bucket 401 are taken from bucket 401, including (4, 2), (1, 3), (5, 0). For chunk 1, no bucket matches the hash value Y (assuming the hash value of the bucket not shown in FIG. 4 is not Y). For chunk 2 and chunk 3, the hash values are Z and W, and records (3, 3), (3, 1), (5, 3) are taken from bucket 404 and records (3, 2), (1, 1), (3, 0), (5, 2), (6, 3) are taken from bucket 403. For these records, block 6, block 4, block 5, block 3 and block 1 can be used as dictionary blocks to compress the input block. Figure 4

[0149] To compress more efficiently, several of the above 5 blocks can be selected as the first block. The result after sorting the records twice is as follows:

[0150] Group 1: (3, 3), (3, 1), (3, 2), (3, 0);

[0151] Group 2: (5, 0), (5, 2), (5, 3);

[0152] Group 3: (1, 3), (1, 1);

[0153] Group 4: (4, 2);

[0154] Group 5: (6, 3).

[0155] Group 1, Group 2, Group 3, Group 4 and Group 5 represent block 3, block 5, block 1, block 4 and block 6 respectively, where Group 4 and Group 5 both have one record. Based on the hash value computed for chunk 2 and chunk 3 of the input block, block 3 has the potential to be a dictionary block. Block 5 can help compress chunk 0, chunk 2, chunk 3. Block 1 is expected to compress chunk 0 and chunk 3 as a dictionary block. Block 4 and block 6 are used to compress chunk 0 and chunk 3 respectively.

[0156] As explained before, the 5 blocks can be used as dictionary blocks to compress the input block. More desirably, several blocks that qualify as dictionary blocks or first blocks are selected according to certain rules. For example, the corresponding group of the first block needs to have at least 2 records, and the first block will be block 3, block 5 and block 1. Or the first block is determined as the three largest groups, and then block 3, block 5 and block 1 are determined.

[0157] ​Since chunk 3 (the largest group) matches two subchunks in total (twice to subchunk 2 and twice to subchunk 3), in some embodiments these matched subchunks can be removed from other groups, so the record (5, 2) corresponding to subchunk 3 and the record (5, 3) corresponding to subchunk 2 are removed from group 2 to avoid duplication. In other words, the matched groups are assigned with corresponding priority according to the number of matched subchunks, where the group with more number of matched subchunks has higher priority. The subchunks with the same hash value are removed, and then the groups with the remaining number of matched subchunks are reordered. After similar steps are performed on group 3, group 4, and group 5, the new groups are as follows:

[0158] Group 1: (3, 3), (3, 1), (3, 2), (3, 0);

[0159] Group 2: (5, 0).

[0160] Then after this step, group 3, group 4, and group 5 are empty, and the first chunks are chunk 3 and chunk 5. In addition, since chunk 5 only matches a single subchunk in the input chunk, it is possible that only chunk 3 is considered as the first chunk for compressing the input chunk.

[0161] In an alternative implementation, a plurality of second chunks are also selected from the first chunk when evaluating the efficiency of the first chunk as a used dictionary chunk. As previously described, string matching is performed between the first chunk and the input chunk while compressing the input chunk, where the chunk with longer matched string is determined as the second chunk.

[0162] For example, chunk 1, chunk 3, and chunk 5 all satisfy the requirement of being a second chunk. Then these three chunks are useful dictionary chunks for compressing the input chunk, but only chunk 3 and chunk 5 can be effectively used to compress the input chunk, where chunk 3 and chunk 5 include all strings required to compress the input chunk.

[0163] The unused first chunk will be discarded, and the compression result of the input chunk can be prefixed with the block identifiers of chunk 3 and chunk 5.

[0164] Figure 5 is a schematic diagram describing the delta compression process provided by the present application.

[0165] The input chunk 504 is divided into one input subchunk, or not, and the string matching is performed between the input subchunk and the dictionary chunks as follows: Figure 5The byte shown calculates two input hashes. After using the hash value as a key to look up in the key-value store, three dictionary blocks (501, 502, 503) are determined. Then the string matching result is: block 501 and block 502 have 1 word matching to the input block 504, and block 503 has 3 words matching to the input block 504. Then, the three dictionary blocks can be used to compress the input block 504. Optionally, block 501 and block 502 can be discarded because they only have 1 word matching to the input block 504.

[0166] In the technical scheme provided by the embodiments of the present application, the input block can be compressed according to the plurality of dictionary blocks (the first block or the second block), which improves the overall compression ratio of the input block.

[0167] After all the above steps are completed, the input block can be compressed according to the dictionary block (the first block or the second block). If the number of dictionary blocks is zero, the input block is independently compressed. If at least one dictionary block is effectively used, the obtained compressed data prefixed with the block ID of the effectively used dictionary block is stored or transmitted by the network. The matching part of the input block and the dictionary block can be represented as (a, b), where a is the offset of the first byte of the matching string, and b is the total length of the matching string. In different compression algorithms, the encoding result is different, such as Huffman encoding or arithmetic encoding.

[0168] Figure 6 is a schematic diagram describing the differential compression process. As Figure 6 shown, for the input block 604, block 601, block 602 and block 603 are candidate dictionary blocks (which can be the first block), and block 603 is a dictionary block (which can be the second block). Then, the input block prefixed with the block ID of block 603 is transmitted to the compression area for compression.

[0169] For example, if more than one dictionary is selected as a dictionary block to compress the input block 604, block 601 and block 603 are selected as dictionary blocks to compress the input block 604. However, when performing string matching, block 601 as a dictionary block can be discarded because the matching string "ZETTA" already exists in block 603.

[0170] Since the input block has been compressed, the key-value store is updated using the input block hash calculation result. Figure 7 The key-value store update result is shown. Bucket 407 is a new bucket that saves the association of hash value Y with record (10, 1). Buckets 401, 403 and 404 are updated with records (10, 0), (10, 3), (10, 2), respectively.

[0171] Then repeat similar steps on the new input block (block 11). After all the blocks are processed, the data compression process is completed.

[0172] It is noted that the key-value store can not save all records of processed chunks in the following cases:

[0173] (a) When a chunk is deleted, information about the chunk will be deleted from the key-value store.

[0174] (b) If there is not enough memory (RAM) for the key-value store, some information can be purged, for example but not limited to, information about chunks that have not been selected as dictionary chunks for a long time.

[0175] (c) If recursive dictionary chunks are not needed, information about delta compressed chunks will not be stored in the key-value store. Recursive dictionary chunks will affect the speed of delta decompression.

[0176] Figure 8 is a flowchart of an embodiment of a method 700 for data compression. The method comprises the following steps.

[0177] S701: Calculate hash values of one or more byte sets obtained from the sub-chunks from the input chunk according to a preset rule.

[0178] S702: Find one or more candidate dictionaries for the input chunk according to the calculated hash values.

[0179] S703: Prioritize the candidate dictionaries according to the number of hash values matched with the input chunk.

[0180] S704: Select one or more optimal candidate dictionaries for delta compression, and then perform delta compression on the input chunk.

[0181] S705: Determine one or more valid dictionaries from the candidate dictionaries according to the length of the matching string.

[0182] In S705, the length of the matching string is greater than a threshold value.

[0183] S706: Store or transmit the delta compressed input chunk prefixed with the ID of the valid dictionary.

[0184] Figure 9 is a flowchart of an embodiment of a method 800 for data compression. The method 800 comprises the following steps.

[0185] S801: Divide the input chunk into sub-chunks of fixed size.

[0186] In this step, the input chunk can be divided into a plurality of sub-chunks of fixed size. S802: Select a byte set for hash calculation.

[0187] In this step, for a chunk, one or more byte sets can be selected for hash computation. Selecting a small number of bytes instead of using the whole chunk for hash computation can reduce the computation cost involved in the compression process.

[0188] S803: Compute one or more hash values for each byte set.

[0189] S804: Use the hash value as a key to look up the key-value store and collect (chunk ID, chunk ID) records with the same key.

[0190] In this step, the hash value is used as a key to look up the first block in the key-value store, which has been mentioned before.

[0191] S805: Sort the (chunk ID, chunk ID) records to find the maximum number of records with the same chunk ID 1, a smaller number of records with other chunk ID 2, etc.

[0192] In this step, the blocks found in step S804 can be sorted by the number of chunks.

[0193] In an optional embodiment, blocks with a number of chunks less than a threshold value can be discarded to improve the quality of the dictionary blocks.

[0194] In another optional embodiment, blocks with more number of chunks can be given higher priority. Similar chunks (chunks with the same hash value) can be removed in lower priority blocks, which will improve the quality of the candidate dictionary blocks.

[0195] S806: Retrieve the block as a dictionary block to compress the input block.

[0196] S807: Prepare the buffer for delta compression: copy the found dictionary block and copy the input block.

[0197] In this step, both the dictionary block and the input block are put into the buffer for compression.

[0198] S808: Compress the input block relative to the dictionary block.

[0199] S809: Is the compression successful?

[0200] S8091: Update the key-value store using the hash value of the input block.

[0201] S8092: Store the input block that cannot be compressed.

[0202] In steps S8091 and S8092, the input block is incompressible. For example, it may include image data that has already been compressed, so the input block is not successfully compressed. However, it can still be a candidate dictionary block for other blocks, so the key-value store needs to be updated using the (block ID, sub-block ID) record based on the hash value of the input block.

[0203] S810: Should dictionary blocks be used?

[0204] In this step, the input block is successfully compressed, but the dictionary block may or may not be used for the compression of the input block.

[0205] S8101: Update the key-value store based on the hash value of the input block.

[0206] S8102: Store the compressed block as a non-differential.

[0207] In steps S8101 and S8102, the input block is successfully compressed, but no dictionary block is used for input block compression. The matching length of the dictionary block may not exceed the threshold, therefore the input block is compressed independently. The key-value store needs to be updated using the input block.

[0208] S811: Discard unused dictionary blocks.

[0209] S812: Store the compressed block as a differential and use the block ID used as the prefix of the input block after differential compression.

[0210] In this step, the input block is successfully compressed using one or more candidate dictionary blocks retrieved in step S806, where the candidate dictionary blocks have matching strings longer than a threshold. Unused blocks can then be discarded to improve the quality of the dictionary blocks. The compressed blocks are prefixed with the IDs of the used dictionary blocks.

[0211] Figure 10 This is a schematic block diagram of the electronic device 900 provided in an embodiment of this application. Figure 10 As shown, the electronic device 900 includes: a partitioning module 901, used to partition an input block into multiple input blocks, wherein each input block includes X bytes; a selection module 902, used to select one or more byte sets from the X bytes of each input block according to a preset rule, wherein the one or more byte sets include Y bytes selected from the X bytes that are not contiguous in the original order; a calculation module 903, used to calculate one or more input hash values ​​for each input block based on the one or more byte sets; and a compression module 904, used to compress the input block according to the input hash values ​​calculated by the calculation module for the multiple input blocks.

[0212] In the technical solution, the division module 901 is configured to divide the input block into one or more sub-blocks. The selection module 902 is configured to select one or more byte sets for the hash calculation of the calculation module 903. The hash calculation result is used by the compression module 904 to compress the input block. Selecting a small number of bytes instead of the entire sub-block for hash calculation reduces the calculation cost involved in the compression process and improves the compression efficiency.

[0213] In some embodiments, when the number of byte sets is multiple for each input sub-block, the difference between the offset0 position of a first byte set and any of the other byte sets is 2m, where m is a natural number unique to the different byte sets, the first byte set is one of the multiple byte sets, and the offset0 position is the position of the first byte of each of the one or more byte sets.

[0214] In some embodiments, the preset rule includes that the difference between the byte position of the discontinuous Y bytes and the offset0 position satisfies a modified Fibonacci sequence or a logarithmic function sequence.

[0215] In some embodiments, the compression module 904 is specifically configured to: determine one or more first blocks according to the input hash values, where the one or more first blocks are processed blocks, and each of the one or more first blocks includes one or more first sub-blocks with the same hash value as one of the input hash values; and compress the input block based on the one or more first blocks.

[0216] In some embodiments, the compression module 904 is configured to: determine one or more first sub-blocks based on the input hash values, where each of the one or more first sub-blocks is a processed sub-block and has the same hash value as one of the input hash values; and determine one or more first blocks based on the one or more first sub-blocks, where the one or more first blocks are blocks divided into the one or more first sub-blocks.

[0217] In some embodiments, the one or more first sub-blocks are determined based on a key-value store, where the key-value store stores the association between the hash values of the processed sub-blocks and the corresponding sub-block identifiers and block identifiers.

[0218] In some embodiments, the number of the one or more first sub-blocks is greater than a first preset threshold.

[0219] In some embodiments, the electronic device further includes a determination module configured to: determine one or more second blocks from the one or more first blocks according to the length of the matching string between the one or more first blocks and the input block, where the length of the matching string is greater than a second preset threshold.

[0220] In some embodiments, the result of the compressed input block includes block identifiers for one or more second blocks.

[0221] In some embodiments, the electronic device further includes an update module for updating the key-value store using the block identifier, block identifier, and corresponding input hash value of the input block.

[0222] like Figure 11 As shown, the electronic device 1000 may include a transceiver 1001, a processor 1002, and a memory 1003. The memory 1003 may be used to store code, instructions, etc., executed by the processor 1002.

[0223] It should be understood that the processor 1002 can be an integrated circuit chip with signal processing capabilities. In implementation, the various steps of the above method embodiments can be completed by hardware integrated logic circuits in the processor or by software instructions. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The processor can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor, or it can be any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly executed by a hardware decoding processor, or executed using a combination of hardware and software modules in the decoding processor. The software modules can be located in storage media that are well-established in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. The storage medium is located in memory, and the processor reads information from the memory and, in conjunction with the hardware in the processor, completes the steps of the above method.

[0224] It can be understood that the memory 1003 in the embodiments of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. The non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically EPROM (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM) used as an external cache. By way of example, but not limitation, many forms of RAM can be used, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous dynamic RAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0225] The embodiments of the present application also provide a chip system, wherein the chip includes an input / output interface, at least one processor and at least one memory. The at least one memory is configured to store instructions, and the at least one processor is configured to invoke the instructions of the at least one memory to perform the operations performed by the electronic device in the methods in the above embodiments.

[0226] The embodiments of the present application also provide a computer storage medium, wherein the computer storage medium can store program instructions to perform the steps in the above methods.

[0227] Optionally, the storage medium can be specifically the memory 1003.

[0228] The embodiments of the present application also provide a computer program product, wherein when the computer program product runs on an electronic device, the electronic device is caused to perform the steps in the above methods.

[0229] Those skilled in the art can clearly understand that each example described in combination with the embodiments disclosed in the specification can be implemented by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are performed by hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0230] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above system, device and unit can refer to the corresponding processes in the above method embodiments. Here, it is not repeated.

[0231] In the embodiments of the present application, "at least one" means one or more, and "multiple" means two or more. The term "and / or" describes the association between the associated objects, which means that there can be three relationships. For example, A and / or B can represent the following three cases: only A exists, A and B exist at the same time, and only B exists, where A and B can be singular or plural. The character "I" usually represents the "or" relationship between the associated objects. "At least one of the following" and similar expressions mean any combination of these items, including one item or any combination of multiple items. For example, at least one of a, b and c can represent: a, b, c, a and b, a and c, b and c, or a, b and c, where a, b and c can be singular or plural.

[0232] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the described device embodiments are only exemplary. For example, the unit division is only a logical function division, and other division manners can be used in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not performed. In addition, the displayed or discussed mutual coupling or direct coupling or communication connection can be implemented through some interfaces. The indirect coupling or communication connection between devices or units can be implemented through electronic, mechanical or other forms.

[0233] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, which can be located in one location, or can be distributed on multiple network units. Some or all units can be selected based on actual needs to achieve the purpose of the embodiments.

[0234] In addition, the functional units in the embodiments of the present application can be integrated into one processing unit, or each unit can exist physically independently, or two or more units are integrated into one unit.

[0235] When these functions are realized in the form of software function units and sold or used as independent products, these functions can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application or the part of the technical solutions that essentially contribute to the prior art or the part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for instructing a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in the embodiments of the present application. The above storage medium includes any medium that can store program codes, such as a USB flash drive, a removable hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0236] The above description is only a specific implementation of the present application, and is not used to limit the protection scope of the present application. Changes or replacements easily thought of by those skilled in the art within the technical scope disclosed in the present application should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A data compression method characterized by, The method comprises: dividing an input block into a plurality of input chunks, wherein each input chunk of the plurality of input chunks comprises X bytes; selecting one or more byte sets from the X bytes of each input chunk of the plurality of input chunks according to a preset rule, wherein the one or more byte sets comprise Y bytes selected from the X bytes and not continuous in original order; calculating one or more input hash values for each input chunk of the plurality of input chunks based on the one or more byte sets; compressing the input chunks according to the input hash values of the plurality of input chunks; when the number of the byte sets is multiple for each input chunk, a difference value of an offsetO position between a first byte set and any one of the other byte sets is 2 m , wherein the m is a natural number unique to different byte sets, the first byte set is one byte set of the plurality of byte sets, and the offset0 position is a position of a first byte of each byte set of the one or more byte sets.

2. The method of claim 1, wherein, The preset rule comprises: a difference value of byte positions between the Y bytes not continuous and the offset0 position satisfies a modified Fibonacci sequence or a logarithmic function sequence.

3. The method of claim 1, wherein, The compressing the input block according to the input hash values of the plurality of input chunks comprises: determining one or more first blocks according to the input hash values, wherein the one or more first blocks are processed blocks, and each first block of the one or more first blocks comprises one or more first chunks whose hash values are the same as one of the input hash values; compressing the input block based on the one or more first blocks.

4. The method of claim 3, wherein, The determining one or more first blocks according to the input hash values comprises: determining the one or more first chunks based on the input hash values, wherein each first chunk of the one or more first chunks is a processed chunk, and a hash value of the processed chunk is the same as one of the input hash values; determining the one or more first blocks based on the one or more first chunks, wherein the one or more first blocks are blocks divided into the one or more first chunks.

5. The method of claim 4, wherein, The one or more first chunks are determined based on a key-value storage, wherein the key-value storage stores associations of hash values of the processed chunks and corresponding chunk identifiers and block identifiers.

6. The method of claim 3, wherein, A number of the one or more first chunks is greater than a first preset threshold.

7. The method of claim 3, wherein, After the compressing the input block based on the one or more first blocks, the method comprises: determining one or more second blocks from the one or more first blocks according to lengths of matching strings between the one or more first blocks and the input block, wherein the lengths of the matching strings are greater than a second preset threshold.

8. The method of claim 7, wherein, A result of the compressed input block comprises block identifiers of the one or more second blocks.

9. The method according to any one of claims 1 to 8, characterized in that, The method further comprises: updating the key-value storage using the chunk identifiers of the input chunks, the block identifiers, and corresponding input hash values.

10. An electronic device, comprising: The method comprises: a dividing module configured to divide an input block into a plurality of input chunks, wherein each input chunk of the plurality of input chunks comprises X bytes; The selection module is configured to select one or more byte sets from the X bytes of each of the plurality of input chunks according to a preset rule, wherein the one or more byte sets include Y bytes selected from the X bytes and not continuous in original order; The calculation module is configured to calculate one or more input hash values for each of the plurality of input chunks based on the one or more byte sets; The compression module is configured to compress the input block according to the input hash values calculated by the calculation module for the plurality of input chunks. When the number of byte sets is multiple for each input chunk, the difference of offsetO positions between the first byte set and any of the other byte sets is 2 m wherein the m is a natural number unique to different byte sets, the first byte set is one of the multiple byte sets, and the offsetO position is a position of a first byte of each of the one or more byte sets.

11. The electronic device of claim 10, wherein, The preset rule includes: The difference between the Y bytes and the offset0 position satisfies a modified Fibonacci sequence or a logarithmic function sequence.

12. The electronic device of claim 10, wherein, The compression module is configured to: determine one or more first blocks according to the input hash values, wherein the one or more first blocks are processed blocks, and each of the one or more first blocks includes one or more first chunks with the same hash value as one of the input hash values; compress the input block based on the one or more first blocks.

13. The electronic device of claim 12, wherein, The compression module is configured to: determine the one or more first chunks based on the input hash values, wherein each of the one or more first chunks is a processed chunk, and the hash value is the same as one of the input hash values; determine the one or more first blocks based on the one or more first chunks, wherein the one or more first blocks are blocks divided into the one or more first chunks.

14. The electronic device of claim 12, wherein, The one or more first chunks are determined based on a key-value storage, wherein the key-value storage stores associations between hash values of the processed chunks and corresponding chunk identifiers and block identifiers.

15. The electronic device of claim 12, wherein, The number of the one or more first chunks is greater than a first preset threshold.

16. The electronic device of claim 12, wherein, The electronic device further includes a determination module configured to determine one or more second blocks from the one or more first blocks according to lengths of matching strings between the one or more first blocks and the input block, wherein the lengths of the matching strings are greater than a second preset threshold.

17. The electronic device of claim 16, wherein, The result of the compressed input block includes block identifiers of the one or more second blocks.

18. The electronic device of any of claims 10-17, wherein, The electronic device further includes an update module configured to update the key-value storage using the chunk identifiers of the input chunks, the block identifiers, and corresponding input hash values.

19. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed on a device, cause the device to perform the method according to any one of claims 1 to 9.

20. A computer program product, characterised in that, The computer program product, when executed on a device, causes the device to perform the method according to any one of claims 1 to 9.

21. A chip system, characterized by The chip system includes a memory and a processor, wherein the memory is configured to store a computer program, and the processor is configured to invoke and execute the computer program from the memory to cause a device in which the chip system is located to perform the method according to any one of claims 1 to 9.

Images