Data compression method and electronic equipment

Abstract

The embodiment of the invention provides a similarity detection and difference compression method and electronic equipment. The method comprises the following steps: calculating hash values of one or more byte sets obtained from sub-blocks of an input block according to a preset rule; according to the calculated hash value, searching a candidate dictionary used for the difference compression of the input block in the processed block; performing priority ranking on candidate dictionaries according to the number of the hash values matched with the input block; selecting a fixed number of optimal candidate dictionaries (first blocks) for differential compression, and performing differential compression; in the process of the differential compression, determining an effective dictionary (second block) for the differential compression through the number of matching character strings between each candidate dictionary and an input block of the differential compression, the number being greater than a threshold value; and further storing or transmitting the input block subjected to the difference compression by taking the ID of the effective dictionary as a prefix. According to the technical scheme, the method can reduce the calculation cost and improve the compression efficiency.

Description

Technical Field

[0001] Embodiments of the present application relate to the field of data similarity in data storage and transmission, and more specifically, to a data compression method and an electronic device, which improves differential compression of the electronic device in data storage and data transmission. Background Art

[0002] With the development of computer technology, the amount of data that needs to be stored or transmitted has increased dramatically. Lossless compression, a process that reduces the size of data without losing information, is widely used to minimize the amount of data stored and transmitted.

[0003] Through similarity detection, similar data can be deduplicated or differentially compressed, and the efficiency of data storage and transmission can be further improved. The known similarity detection methods are very computationally expensive and time-consuming. Summary of the invention

[0004] The embodiment of the present application provides a method for data similarity detection, which improves differential compression in data storage and data transmission of electronic devices.

[0005] According to a first aspect, an embodiment of the present application provides a method for data similarity detection, comprising: dividing an input block into a plurality of input blocks, wherein each of the plurality of input blocks includes X bytes; selecting one or more byte sets from the X bytes of each of the plurality of input blocks according to a preset rule, wherein the one or more byte sets include Y bytes selected from the X bytes that are discontinuous in the original order; based on the one or more byte sets, calculating one or more input hash values ​​for each of the plurality of input blocks; matching the input block with other blocks according to the calculated hash values ​​of the one or more blocks to determine their similarity. Then, the one or more best similarity candidate blocks found in this manner are used as a dictionary to perform differential compression on the input block.

[0006] According to the above technical solution, the input block is divided into multiple input blocks, and a small number of bytes (byte sets) are selected from each input block for hash calculation. Compared with using all bytes in the block for hash calculation, selecting a byte set can reduce the computational cost involved in the hash calculation. The calculated hash value can be used to match the input block with other blocks to determine their similarity. Then, one or more best similarity candidate blocks found in this way are used as dictionaries to perform differential compression on the input block.

[0007] In an optional implementation, when there are multiple byte sets for each input block, several byte sets are selected from each block, and the corresponding number of hash values ​​is calculated for each block. The first byte of each selected byte set is offset from the beginning of the block, equal to 0, 1, 2, 4...2 m , where m is a natural number unique to different byte sets.

[0008] According to the above technical solution, if multiple byte sets are selected in an input block, or multiple input hash values ​​are calculated for an input block, the selected byte sets may be shifted by 0, 1, 2, 4, ... 2 m bytes, which may help detect insertions and deletions due to m The similarity feature of shifting by one byte each, which is quite frequent due to the ubiquitous 8-bit, 16-bit, 32-bit and 64-bit data types in modern SW, improves the detection of similarity between blocks and thus improves the differential compression ratio.

[0009] In an optional implementation, the preset rule includes: the interval between the discontinuous Y bytes is determined by a modified Fibonacci sequence, a logarithmic function or an exponential function. The modified Fibonacci sequence is a Fibonacci sequence without the first 1, such as 1, 2, 3, 5...

[0010] In an optional implementation, the preset rule includes: for any two bytes in each of the byte sets, selecting the two bytes from non-adjacent positions in the input block.

[0011] According to the above technical solution, bytes are selected at intervals, which can reduce the number of selected bytes and the computational cost involved in hash calculation and similarity detection.

[0012] In an optional implementation, performing similarity detection on the input block and other blocks according to the input hash values ​​of the multiple input sub-blocks includes: determining one or more first blocks according to the input hash values, wherein the one or more first blocks are processed blocks, and each of the one or more first (processed) blocks includes one or more first sub-blocks whose hash values ​​are the same as one of the input hash values. Then, the one or more best similarity candidate blocks found in this way are used as dictionaries to perform differential compression on the input block.

[0013] According to the above technical solution, the input hash value calculated for the input sub-block of the input block is used to find information about the processed block in the key-value storage, and the processed block includes sub-blocks with the same hash value.

[0014] The similarity detection of the input block and one or more processed data blocks is performed by comparing the input hash value with the hash values ​​calculated for the blocks of the processed data block (further referred to as the processed hash value and the processed block, respectively) to find one or more pairs, each pair including the input block and the processed block, whose input hash values ​​and processed hash values ​​are equal, and each block is associated with a corresponding data block identifier (also referred to as input block ID and processed block ID).

[0015] The found processed blocks whose hash values ​​are equal to the input hash value are sorted by processed block ID so that the corresponding processed block IDs whose hash values ​​are equal to the hash value of the input block are sorted by the number of blocks.

[0016] One or more processed blocks are selected as candidate dictionaries for differential compression of the input block, wherein the one or more processed blocks match the sorted block ID with the largest number of blocks and have a hash value equal to the hash value of the input block.

[0017] The input block is differentially compressed using the selected processed block as a candidate dictionary.

[0018] During differential compression, it is determined whether each candidate dictionary includes a byte string,equal to the byte string in the input block, so that differential compression is effective, i.e., the byte string in the input block is encoded as a reference to an equal byte string in the dictionary block, whose number of bits is less than the number of bits in the corresponding byte string.

[0019] Dictionary candidates that do not include any byte string that is valid for delta compression of the input block are discarded.

[0020] Dictionary candidates that include valid delta-compressed byte strings for the input block are retained as valid dictionaries for further reference.

[0021] Stores the delta compression result of the input block and the processed block ID of the valid dictionary.

[0022] A subset of input hash values ​​is selected to be stored as processed hash values, the subset of input hash values ​​being associated with corresponding block IDs for future similarity detection between the processed block and future input blocks, such that the subset includes only different hash values; in addition, the subset includes one or more minimum hash values ​​and one or more maximum hash values.

[0023] The selected input hash value associated with the input block ID is stored for later use in a suitable data structure for similarity detection between the processed block and future input blocks to facilitate fast lookup by using the hash as a key to obtain the processed block ID and retrieve the corresponding processed block as a candidate dictionary for differential compression of the input block.

[0024] According to a second aspect, an electronic device is provided. The electronic device comprises: a division module, configured to divide an input block into a plurality of input blocks, wherein each of the plurality of input blocks comprises X bytes; a selection module, configured to select one or more byte sets from the X bytes of each of the plurality of input blocks according to a preset rule, wherein the one or more byte sets comprise Y bytes selected from the X bytes that are discontinuous in the original order; a calculation module, configured to calculate one or more input hash values ​​for each of the plurality of input blocks based on the one or more byte sets; and a compression module, configured to compress the input block according to the input hash values ​​calculated by the calculation module for the plurality of input blocks.

[0025] According to the above technical solution, the partitioning module is used to partition the input block into one or more sub-blocks. The selection module is used to select one or more byte sets for hash calculation by the calculation module. The hash calculation result will be used by the compression module to compress the input block. Selecting a small number of bytes instead of the entire sub-block for hash calculation reduces the computational cost involved in the compression process and improves the compression efficiency.

[0026] In an optional implementation, when there are multiple byte sets for each input block, the difference in the offset0 position between the first byte set and any one of the other byte sets is 2 m , wherein m is a natural number unique to different byte sets, the first byte set is a byte set among the multiple byte sets, and the offset0 position is the position of the first byte of each byte set in the one or more byte sets.

[0027] In an optional implementation, the preset rule includes: the difference in byte position between the discontinuous Y bytes and the offset0 position satisfies a modified Fibonacci sequence or a logarithmic function sequence.

[0028] In an optional implementation, the compression module is specifically used to: determine one or more first blocks based on the input hash value, wherein the one or more first blocks are processed blocks, and each of the one or more first blocks includes one or more first sub-blocks whose hash values ​​are the same as one of the input hash values; and compress the input block based on the one or more first blocks.

[0029] In an optional implementation, the compression module is used to: determine the one or more first blocks based on the input hash value, wherein each of the one or more first blocks is a processed block and the hash value is the same as one of the input hash values; determine the one or more first blocks based on the one or more first blocks, wherein the one or more first blocks are blocks divided into the one or more first blocks.

[0030] In an optional implementation, the one or more first blocks are determined based on a key-value store, wherein the key-value store stores an association between a hash value of the processed block and a corresponding block identifier and a block identifier.

[0031] In an optional implementation, the number of the one or more first blocks is greater than a first preset threshold.

[0032] In an optional implementation, the electronic device further includes a determination module, which is used to determine one or more second blocks from the one or more first blocks based on the length of a matching string between the one or more first blocks and the input block, wherein the length of the matching string is greater than a second preset threshold.

[0033] In an optional implementation, the result of compressing the input block includes the block identifiers of the one or more second blocks.

[0034] In an optional implementation, the electronic device further includes an update module, which is used to update the key-value storage using the block identifier, the block identifier and the corresponding input hash value of the input block.

[0035] According to a third aspect, an embodiment of the present application provides a data compression method, comprising: dividing an input block into multiple input blocks, wherein each of the multiple input blocks includes X bytes; according to a preset rule, selecting one or more byte sets from the X bytes of each of the multiple input blocks, wherein the one or more byte sets include Y bytes selected from the X bytes that are discontinuous in the original order; based on the one or more byte sets, calculating one or more input hash values ​​for each of the multiple input blocks; and compressing the input block according to the input hash values ​​of the multiple input blocks.

[0036] According to the above technical solution, the input block is divided into multiple input blocks, and a small number of bytes (byte sets) are selected from the divided input blocks for hash calculation. Selecting a byte set instead of taking the block as a whole to obtain features through hash calculation can reduce the computational cost involved in the hash calculation. The calculated input hash value can be used to compress the input block.

[0037] Optionally, one or more sets of bytes may be selected for hashing of the input block as a whole, and the hash value of the processed block may then be used to compress the input block.

[0038] In an optional implementation, when there are multiple byte sets for each input block, the difference in the offset0 position between the first byte set and any one of the other byte sets is 2 m , wherein m is a natural number unique to different byte sets, the first byte set is a byte set among the multiple byte sets, and the offset0 position is the position of the first byte of each byte set in the one or more byte sets.

[0039] According to the above technical solution, if multiple byte sets are selected in an input block, or multiple input hash values ​​are calculated for an input block, the difference in the offset0 position between the first byte set and any of the other byte sets is 2 m , in other words, for different byte sets, the selected bytes may be shifted by 2 m bytes, which may help detect insertions and deletions due to m The similar feature of shifting by 100 bytes is quite frequent due to the ubiquitous 8-bit, 16-bit, 32-bit and 64-bit data types in modern SW, thus improving the compression ratio. Optionally, the offset0 position of the first byte set is 0, the offset0 position of the second byte set is 1, the offset0 position of the third byte set is 2, and so on. In other words, the difference in the offset0 position between the first byte set and the subsequent byte set is 1, 2, 4, 8, 16, ...

[0040] In an optional implementation, the preset rule includes: the difference in byte position between the discontinuous Y bytes and the offset0 position satisfies a modified Fibonacci sequence or a logarithmic function sequence.

[0041] According to the above technical solution, the difference in byte positions between byte sets satisfies a modified Fibonacci sequence or a logarithmic function sequence or an exponential function sequence. Bytes are selected at intervals so that only a small number of bytes are selected for hash calculation, thereby reducing calculation costs and increasing compression speed.

[0042] In an optional implementation, the preset rule includes: for any two bytes in each of the byte sets, selecting the two bytes from non-adjacent positions in the input block.

[0043] According to the above technical solution, bytes are selected at intervals, which can reduce the number of selected bytes and the involved computational cost.

[0044] In an optional implementation, compressing the input block according to the input hash value of the multiple input blocks includes: determining one or more first blocks according to the input hash value, wherein the one or more first blocks are processed blocks, and each of the one or more first blocks includes one or more first blocks whose hash values ​​are the same as one of the input hash values; and compressing the input block based on the one or more first blocks.

[0045] According to the above technical solution, the input hash value calculated for the input sub-blocks of the input block is used to find a block in the key-value storage, and the block includes sub-blocks with the same hash value.

[0046] For example, the first block includes all blocks corresponding to matching records in the key-value store.

[0047] As another example, a first block may be selected from the blocks corresponding to the record retrieved from the key-value store.

[0048] As another example, priority levels can be assigned to blocks corresponding to records retrieved from the key-value store based on the number of blocks each block has. Then, if two blocks in different blocks are retrieved from the key-value store based on the same hash value, the block with the lower priority in the block will be eliminated because the block with the higher priority in the block can compress a specific input block in the input block with the same hash value. The blocks are then reordered based on the number of remaining blocks. The reordered blocks can be considered as the first blocks that may be able to perform differential compression on the input block. This operation will improve the quality of the first block as a candidate dictionary block.

[0049] In an optional implementation, determining one or more first blocks based on the input hash value includes: determining the one or more first partitions based on the input hash value, wherein each of the one or more first partitions is a processed block and has a hash value that is the same as one of the input hash values; determining the one or more first blocks based on the one or more first partitions, wherein the one or more first blocks are blocks divided into the one or more first partitions.

[0050] In an optional implementation, the one or more first blocks are determined based on a key-value store, wherein the key-value store stores an association between a hash value of the processed block and a corresponding block identifier and a block identifier.

[0051] In an optional implementation, the number of the one or more first blocks is greater than a first preset threshold.

[0052] Through the above technical solution, the number of sub-blocks included in the first block is greater than the first preset threshold, which increases the possibility that the first block can be used to compress the input block and improves the compression ratio.

[0053] In an optional implementation, after compressing the input block based on the one or more first blocks, the method includes: determining one or more second blocks from the one or more first blocks based on the length of the matching string between the one or more first blocks and the input block, wherein the length of the matching string is greater than a second preset threshold.

[0054] According to the above technical solution, the second block is selected from the first block according to the string matching result, and only the first block whose matching string length is greater than the second preset threshold is considered as the second block. The second block is a block that can be used to compress the input block.

[0055] In an optional implementation, the result of compressing the input block includes the block identifiers of the one or more second blocks.

[0056] According to the above technical solution, the compressed input block includes the block identifier of the second block. For example, the compressed input block is prefixed with the block identifier of the second block, which can be used for compression of the input block.

[0057] In an optional implementation, the method further includes: updating the key-value storage using the chunk identifier, the chunk identifier and the corresponding input hash value of the input chunk.

[0058] According to the above technical solution, after compressing the input block, the block identifier, block identifier and corresponding input hash value of the input block are used to update the key-value storage, so that when compressing subsequent blocks, the block identifier and block identifier of the input block can be used for the similarity detection process.

[0059] According to a fourth aspect, an embodiment of the present application provides a computer-readable storage medium including instructions. When the instructions are executed on a computer, the computer is caused to execute the method in the first aspect or any optional implementation of the first aspect.

[0060] According to a fifth aspect, an electronic device is provided, comprising a processor and a memory. The processor is connected to the memory. The memory is used to store instructions, and the processor is used to execute the instructions. When the processor executes the instructions stored in the memory, the processor executes the method in the first aspect or any optional implementation of the first aspect.

[0061] According to the sixth aspect, a chip system is provided, wherein the chip system includes a memory and a processor, the memory is used to store a computer program, and the processor is used to call the computer program from the memory and run the computer program so that the electronic device where the chip system is located executes the method in the first aspect or any optional implementation of the first aspect.

[0062] According to a seventh aspect, a computer program product is provided, wherein when the computer program product is run on an electronic device, the electronic device is caused to execute the method in the first aspect or any optional implementation manner of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 A flow chart of an embodiment of a method 100 for data compression is shown.

[0064] Figure 2 An embodiment for selecting a set of bytes for each chunk in an input block is shown.

[0065] Figure 3 Shown is the Figure 1 Block 0 in selects an embodiment of more than one set of bytes.

[0066] Figure 4 It is a schematic diagram of key-value storage.

[0067] Figure 5 is a schematic diagram describing the differential compression process provided by the present application.

[0068] Figure 6 is a schematic diagram describing the differential compression process.

[0069] Figure 7 The key-value store update result based on the input block is shown.

[0070] Figure 8 is a flow chart of an embodiment of a method 700 for data compression.

[0071] Fig. 9 is a flow chart of an embodiment of a method 800 for data compression.

[0072] Fig.10 is a schematic block diagram of an electronic device 900 .

[0073] Fig.11 is a schematic block diagram of an electronic device 1000 . DETAILED DESCRIPTION

[0074] The technical solution in this application is described below with reference to the accompanying drawings.

[0075] definition:

[0076] Unless otherwise stated or implicit from context, the following terms and phrases have the meanings provided below.

[0077] The term "block" refers to a sequence of bytes having a predetermined length.

[0078] When a chunk is divided from a block, the term "chunk" refers to a sequence of bytes whose predetermined length is less than the predetermined length of the block. In some cases, when the entire block is considered as one chunk, the chunk is a sequence of bytes whose length is the same as the length of the block.

[0079] The term "byte" refers to a sequence of 8 bits, where a bit is the smallest unit in computer science.

[0080] The phrase "hash value" refers to the output returned after applying a hash function algorithm. The phrases "hash function algorithm" and "hash function" refer to an algorithm or subroutine that maps a large data set (optionally, of variable length) to a smaller data set of a fixed size for a particular hash function. The value returned by an algorithm may also be called a hash function value, hash code, hash sum, checksum, or hash value.

[0081] Data compression, source coding, or bit rate reduction is the process of encoding information using fewer bits than the original representation. Any particular compression is either lossy or lossless. Lossless compression reduces bits by identifying and eliminating statistical redundancy. Lossless compression does not lose any information. Lossy compression reduces bits by removing unnecessary or less important information. Typically, a device that performs data compression is called an encoder, and a device that performs the inverse process of said data compression (decompression) is called a decoder.

[0082] Compression is useful because it reduces the resources required to store and transmit data. The compression and decompression process consumes computing resources. There is a space-time complexity trade-off for data compression. The design of a data compression scheme involves a balance between various factors, including the degree of compression, the amount of distortion introduced (when using lossy data compression), and the computing resources required to compress and decompress the data.

[0083] Differential encoding is a lossless compression method, which is a way of storing or transmitting data as differences (Δ) between sequential data rather than complete files; more generally, this is called data differencing. Differential encoding is sometimes called differential compression, especially in situations where a history of changes is required (for example, in revision control software).

[0084] Data compression ratio or compression ratio (CR), also known as compression power, is a measure of the relative reduction in the size of the data representation produced by a data compression algorithm. It is usually expressed as the uncompressed size divided by the compressed size, expressed as:

[0085]

[0086] When differential compression is applied, the input data is divided into smaller units and hash values ​​are calculated for these units. The hash values ​​are considered to be the fingerprints of these units. Therefore, for units with the same hash value, one original unit and one differential value can be stored instead of two complete units.

[0087] However, there is a balance between the number of hash values ​​that need to be calculated and the computational cost of each calculated hash value. That is, when the bytes / bits of the divided units are small, the input data is divided into more units, thereby increasing the overall computational cost of the hash calculation, while when the bytes / bits of the divided units are large, each unit is longer, so the computational cost of each hash value is higher. It is important to reduce the computational cost of compression.

[0088] The embodiments of the present application address the challenges of reducing the computational cost of compression algorithms and accelerating compression algorithms.

[0089] Figure 1 1 is a flow chart of a data compression method 100 provided in an embodiment of the present application. The method 100 includes the following steps.

[0090] S102: Divide the input block into a plurality of input blocks, wherein each of the plurality of input blocks includes X bytes.

[0091] An input block may be received from a host computer via a wired or wireless network. The data to be compressed may be transmitted in multiple blocks, or the blocks may be defined by a device that performs a data compression method. A block refers to a sequence of bytes having a predetermined byte length. An input block is a block being processed. Before the input block, several blocks may have been processed. After the input block, there may be several blocks to be processed. The block size is not fixed. For example, the size of a block may be 4 kilobytes (KB). A block may include a header to identify the block. The header may include information such as, but not limited to, a block identifier or block ID associated with the block. A block with a block ID of 2 may be represented as block 2.

[0092] The input block can be divided into multiple input sub - blocks. A sub - block refers to a byte sequence whose predetermined length is less than that of the block. For example, a 4KB input block can be divided into 4 * 1024B input sub - blocks. In addition, each sub - block has a sub - block ID for identifying a unique sub - block. Therefore, the four input sub - blocks of the input block can be represented as sub - block 0, sub - block 1, sub - block 2, and sub - block 3. To distinguish sub - blocks in different blocks, (block ID, sub - block ID) is introduced as the unique ID (UID) of the sub - block to help illustrate this embodiment.

[0093] S104: According to a preset rule, select one or more byte sets from X bytes of each input sub - block among multiple input sub - blocks, where the one or more byte sets include Y non - consecutive bytes selected from the X bytes in the original order.

[0094] In some embodiments, one byte set is selected from each sub - block. For all sub - blocks in the data stream, bytes are selected according to a preset rule. The byte set includes Y non - consecutive bytes from the X bytes. In other words, the selected bytes are not consecutive. The byte set is selected starting from the offset0 position with an increasing step size, where the selected bytes are separated by a predefined pattern. For example, the preset rule includes: the difference between the byte positions of the Y non - consecutive bytes and the offset0 position satisfies a modified Fibonacci sequence or a logarithmic function sequence. The logarithmic sequence is a set of real numbers a[i], i = 1, 2..., such that

[0095] (1) 0 <= a[i] <= 1,

[0096] (2) a[k]+a[i] < a[r]+a[s], if k * l < r * s,

[0097] (3) a[k]+a[i] = a[r]+a[s], if k * l = r * s.

[0098] In other words, the step size can increase in a logarithmic, exponential, or linear manner through a modified Fibonacci sequence or even through a self - designed sequence.

[0099] The difference between the byte positions of the Y non - consecutive bytes and the offset0 position is the result of subtracting the offset0 position from the byte position or byte ID of one of the Y bytes. Take the case where the difference satisfies a modified Fibonacci sequence as an example. The input block is a 64 - byte string divided into 4 input sub - blocks, respectively represented as sub - block 0, sub - block 1, sub - block 2, and sub - block 3. As Figure 2 shown, each input sub - block is a 16 - byte string. Figure 2Each number in the box represents a byte ID or byte position. In this application, "offset0 position", "byte number", "byte ID", "byte position" all refer to the position of the byte in the original input block from which the byte was selected. Block 0 includes bytes 0 to byte 15, block 1 includes bytes 16 to byte 31, block 2 includes bytes 32 to byte 47, and block 3 includes bytes 48 to byte 63.

[0100] The Fibonacci sequence is [1,1,2,3,5,8,13,21…]. The first 1 will be replaced with 0 to avoid repetition. Then the modified Fibonacci sequence will be [0,1,2,3,5,8,13,21…].

[0101] If the offset0 position is the starting position of the input block when selecting the first byte of Y bytes, or the byte ID of the offset0 position is 0, since the position difference satisfies the modified Fibonacci sequence, the byte ID of the selected byte for block 0 is [0,1,2,3,5,8,13], which is Figure 2 According to the modified Fibonacci sequence, the byte IDs of the selected bytes of block 1, block 2, and block 3 are [16, 17, 18, 19, 21, 24, 29], [32, 33, 34, 35, 37, 40, 45], and [48, 49, 50, 51, 53, 56, 61], respectively.

[0102] If the offset0 position becomes m, since the position difference satisfies the modified Fibonacci sequence, for block 0, the byte IDs of the selected bytes are [m, 1+m, 2+m, 3+m, 5+m, 8+m, 13+m]. The case where the byte ID exceeds the maximum byte ID in block 0 is not considered here.

[0103] It should be noted that in the above example, for each block, the length of the byte set is 7 bytes. Optionally, the selected bytes can follow any part of the modified Fibonacci sequence. The length can be limited by a small constant. For example, if the length of the selected bytes is 5 bytes, the byte IDs of the selected bytes of block 0 can be [0,1,2,3,5], [1,2,3,5,8] or [2,3,5,8,13]. In addition, the first 1 can be deleted to obtain the modified Fibonacci sequence, and the modified Fibonacci sequence will be [1,2,3,5,8,13,21...].

[0104] It should also be noted that the preset rules may be: for any two bytes in each byte set, the two bytes are selected from non-adjacent positions in the block, or the number of consecutive bytes selected from the block does not exceed n bytes, where n is a preset value. These preset rules will reduce the number of bytes selected and expand the selection range in the block, thereby detecting more features and reducing the costs involved.

[0105] According to a third aspect, an embodiment of the present application provides a computer-readable storage medium including instructions. When the instructions are executed on a computer, the computer is caused to execute the method in the first aspect or any optional implementation of the first aspect.

[0106] According to a fourth aspect, an electronic device is provided, comprising a processor and a memory. The processor is connected to the memory. The memory is used to store instructions, and the processor is used to execute the instructions. When the processor executes the instructions stored in the memory, the processor executes the method in the first aspect or any optional implementation of the first aspect.

[0107] According to the fifth aspect, a chip system is provided, wherein the chip system includes a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to call the computer program from the memory and run the computer program so that the electronic device where the chip system is located executes the method in the first aspect or any optional implementation of the first aspect.

[0108] According to a sixth aspect, a computer program product is provided, wherein when the computer program product is run on an electronic device, the electronic device is caused to execute the method in the first aspect or any optional implementation of the first aspect.

[0109] In an optional implementation, if only one byte set is selected for a block, the offset0 position, i.e., the starting position of the byte set, can be randomly selected. In this implementation, the offset0 position does not necessarily represent the first byte position of the byte set. For example, if the modified Fibonacci sequence is [0,1,2,3,5,8,13,21…], the offset0 position is the position of the first byte in a byte set. However, if the modified Fibonacci sequence is [1,2,3,5,8,13,21…], the first byte of the byte set is the byte after the offset0 position.

[0110] In another optional implementation, when the number of byte sets for each input block is multiple, the difference in offset0 position between the first byte set and any one of the other byte sets is 2m, where m is a natural number unique to different byte sets, the first byte set is one of the multiple byte sets, and the offset0 position is the position of the first byte of each of the one or more byte sets. In other words, the difference follows an exponential function with base 2, that is, the difference in offset0 position of different byte sets of a block can be 1 byte, 2 bytes, 4 bytes, 8 bytes, 16 bytes, 32 bytes, ... For different byte sets in a block, the value of m can be a continuous integer or a discontinuous integer. The offset0 position of different byte sets in a block is intended to detect similar content that is shifted by 2m bytes due to insertion and deletion, which is very frequent due to the ubiquitous 8-bit, 16-bit, 32-bit, and 64-bit data types in modern SW. For example, if both chunk 0 in chunk 3 and chunk 1 in chunk 5 have 16 bytes, where chunk 1 can be implemented by inserting one byte before chunk 0 and deleting the last byte in chunk 0, then the first fifteen bytes in chunk 0 are the same as the last fifteen bytes in chunk 1. If only one byte set is selected for each chunk in the same way, the matching fifteen bytes will not be detected by the selected byte set. However, if two byte sets are selected and the two byte sets are shifted by one byte, the matching fifteen bytes will be detected by the selection method proposed in the present application.

[0111] If an offset0 position (represented as reference offset0 position) is the first byte of a block, then the offset0 position of the byte set is 0 bytes, 1 byte, 2 bytes, 4 bytes, 8 bytes, 16 bytes, etc. away from the first byte of the block, or the offset0 position is the first byte, the second byte, the third byte, the fifth byte, the ninth byte, the seventeenth byte, etc. of the block. Figure 3As shown, for example, for each input block, three byte sets are selected. For block 0, the byte IDs at the offset0 position are 0, 1, and 2, which are the first byte, the second byte, and the third byte of the block, respectively. The offset0 position is equal to 0 bytes, 1 bytes, 2 bytes, 4 bytes, 8 bytes, 16 bytes, 32 bytes, etc. starting from the reference offset0 position. The byte IDs of the three byte sets are [0,1,2,3,5,8,13], [1,2,3,4,6,9,14], and [2,3,4,5,7,10,15], respectively. Multiple byte sets are selected for hash calculation to detect the characteristics of the blocks (discussed later). In this way, several byte sets are selected for subsequent hash calculations.

[0112] S106: Calculate one or more input hash values ​​for each input block in the plurality of input blocks based on the one or more byte sets.

[0113] Hash calculation can be interpreted as mapping a set of bytes to a scalar value (a number). The calculated scalar value is used as an index to determine the bucket in the key-value store (discussed later). Hash calculation is based on the selected set of bytes. Therefore, if a block selects only one set of bytes, a block will calculate one hash value, and if a block selects multiple sets of bytes, a block will calculate multiple hash values. However, it should be noted that multiple hash values ​​of a block are not necessarily different. In addition, if different blocks have similar characteristics, their hash values ​​may also be the same.

[0114] The embodiment of the present application does not limit the algorithm of hash calculation. For example, hash calculation can use Rabin-Karp, MD, SHA algorithm (secure hash algorithm), crc 32, etc.

[0115] S108: Compress the input block according to the input hash values ​​of the multiple blocks.

[0116] In some embodiments, compressing an input block according to input hash values ​​of multiple input blocks includes: determining one or more first blocks according to one or more input hash values, wherein the one or more first blocks are processed blocks, and each of the one or more first blocks includes one or more first blocks whose hash values ​​are the same as one of the one or more input hash values; and compressing the input block based on the one or more first blocks.

[0117] In order to clearly describe the technical solutions in the embodiments of the present application, in the embodiments of the present application, the words "first" and "second" are used to distinguish the same items or similar items with substantially the same functions and effects. Those skilled in the art can understand that the terms "first" and "second" do not limit the quantity or execution order, and the terms "first" and "second" do not indicate a definite difference.

[0118] In order to obtain the one or more first blocks, it is necessary to use the input hash value as a key to look up the key-value storage (or hash table), thereby obtaining the one or more first blocks.

[0119] In this implementation, determining one or more first blocks based on the input hash value includes: determining one or more first partitions based on the input hash value, wherein each of the one or more first partitions is a processed block and the hash value is the same as one of the input hash values; determining one or more first blocks based on the one or more first partitions, wherein the one or more first blocks are blocks divided into one or more first partitions.

[0120] The key-value store stores the association of the hash values ​​of processed chunks with the corresponding (chunk ID, chunk ID) records, using the input hash values ​​as keys and (chunk ID, chunk ID) records as values. The data stream to be compressed consists of multiple chunks, and the processed chunks come from the processed chunks that precede the input chunks. The processed chunks have been divided into one or more chunks of X bytes, and a set of bytes of Y bytes are selected from the chunks for hash calculation in the same way as the set of bytes were selected for the input chunks. Assuming the first chunk of the data stream to be compressed is denoted chunk 0, and the input chunk is chunk 5, the key-value store stores the association of the records from chunks of chunk 0, chunk 1, chunk 2, chunk 3, and chunk 4. Obviously, the key-value store is dynamically updated as chunks are processed.

[0121] For example, a key-value store may have multiple buckets that hold records for chunks with different hash values. Each bucket will hold records for chunks with the same hash value. So if a hash value is calculated for a chunk, records for 4 chunks in a processed block can be stored in at most 4 buckets of the key-value store.

[0122] The key-value store is looked up using the hash value of the input chunk as the key, which may result in multiple buckets. All records in these buckets are then used for the input chunk. The chunk corresponding to the record determined based on the key-value store is the first chunk mentioned above. The chunk mentioned above is the first chunk that can be used for the compression process of the input chunk.

[0123] In some embodiments, the record of the block is used as the first block to be used as a dictionary block for compressing the input block. In other embodiments, the dictionary block is also selected from the first block. For example, the number of one or more first sub-blocks is greater than a first preset threshold, which indicates that the block with more similar sub-blocks to the input block is qualified as the first sub-block.

[0124] To obtain one or more first blocks of an input block, the method may comprise the following steps:

[0125] (a) Sort the records by block ID.

[0126] After the first sort, records with the same block ID are grouped together, and any group includes records with the same block ID. Among the records in each group, there may be records with the same split block ID. Duplicate records are removed, and then each group includes records with the same block ID and unique split block IDs.

[0127] (b) Sort the groups by the number of records (number of chunks).

[0128] The blocks corresponding to the groups with more records or more blocks include more features (hash values) of the input blocks. Therefore, after the second sorting, N groups can be selected from the M groups as the first blocks of the input blocks, where N≤M, M is the number of blocks including at least one record, and N is the number of blocks determined from the M blocks based on certain conditions (for example, the first block includes at least 2 records).

[0129] After the two sorts, the first block is selected as the dictionary block of the input block, and then the input block is compressed based on the first block.

[0130] If it is necessary to explain, the two sorting processes provided here are only one implementation method of determining the first block, and there may be other means to determine the first block, for example, the number of records corresponding to each block may be recorded in a statistical table. In any case, the first block will be implemented.

[0131] If the number of the first block is zero, the input block is compressed separately and stored or transmitted over the network. The following process of this technical solution is: update the key value storage using the block identifier, block identifier and corresponding input hash value of the input block. In other words, insert the calculated hash value and corresponding record of (block ID, block ID) into the key value storage for further searching for subsequent input blocks in the first block.

[0132] It should be noted that if at least two hash values ​​are calculated for a single block, a block having one of the at least two hash values, rather than a block having all of the at least two hash values, is eligible as a candidate dictionary block, which can reduce the overhead computational cost of the determination process of the first block.

[0133] It should also be noted that if at least two byte sets are selected for each input block, then in a possible implementation, a hash value set may be selected based on a byte set to be inserted into the key-value storage.

[0134] In an optional implementation, after compressing the input block based on one or more first blocks, the method includes: determining one or more second blocks from the one or more first blocks according to the length of the matching string between the one or more first blocks and the input block, wherein the length of the matching string is greater than a second preset threshold.

[0135] In this implementation, the first block is a block that has the potential to be a dictionary block, but is not necessarily a useful dictionary block for compressing the input block. Therefore, the efficiency of the first blocks for compressing the input block is evaluated, and unused first blocks are discarded to reduce storage costs. The number of first blocks is variable. For example, there may be one or more first blocks after a hash match. The one or more first blocks are then copied into a buffer, which is transmitted as input to the differential compressor, followed by the input block. The output of the differential compressor includes the differentially compressed input blocks and statistics on how multiple substrings ("words") from the input block are found as duplicates in each dictionary block.

[0136] Statistics can be collected via further pseudo-code:

[0137]

[0138] input_word_in_dict is a pseudocode function that searches for a word (or substring) from an input block in a dictionary block (dictZ, dictY, dictA) and takes two parameters, w (the word or substring) and the dictionary block (dictZ, dictY, dictA). According to the pseudocode, the statistics of how many substrings ("words") in the input block are found as duplicates in each dictionary block are stats[dictZ], stats[dictY], and stats[dictA].

[0139] The second block selected from the first block based on the specified condition is a block that is effective for performing differential compression on the input block. The first dictionary block that qualifies as the second block can be determined by certain rules. For example, the total length of the matching strings in the first block and the input block is greater than a certain threshold. By using the above pseudo code, the string matching results (stats[dictZ], stats[dictY], and stats[dictA]) between the input block and the dictionary blocks (dictZ, dictY, and dictA) can be obtained, and the second block can be determined based on the matching results.

[0140] In some embodiments, the number of second blocks of the input block is 0 and the input block is compressed independently.

[0141] For example, the number of first blocks is one or more, but no first block meets the requirement of becoming a second block, that is, the length of the matching string between all first blocks and the input block is zero or less than a certain threshold. Therefore, all first blocks will be discarded.

[0142] In other embodiments, one or more second blocks are effectively used to compress the input block. One or more second blocks are selected from the first block, and the length of the matching string of the one or more second blocks and the input block is greater than a threshold. In this case, the second block can be used to compress the input block, and the compression result is prefixed with the ID of the second block (second block ID).

[0143] It should be noted that the second block is a block used to compress the input block. The input block has been compressed in the previous process. The second block is determined to evaluate the efficiency of the first block, thereby discarding the unused first block and saving storage costs.

[0144] After determining the second blocks evaluated as being useful for compressing the input block, the result of the compressed input block is prefixed with one or more block identifiers of the second blocks.

[0145] Assuming that the ID of the input block is block 10, and only one hash value is calculated for a block, the following example illustrates the process of determining the first block and the second block.

[0146] The input block is divided into 4 input blocks, and the block IDs are block 0, block 1, block 2 and block 3 respectively.

[0147] As mentioned above, the key-value store stores the association between the hash values ​​of the blocks divided from block 0 to block 9 and the corresponding (block ID, block ID) records. Figure 4 As shown, the current key-value storage includes multiple buckets corresponding to records of (block ID, block ID) with unique hash values, where the block ID is between 0 and 9 and the block ID is between 0 and 3. For example, bucket 401 stores records of (4, 2), (1, 3), and (5, 0), and the corresponding blocks share a common hash value X. The hash values ​​of bucket 402, bucket 403, bucket 404, bucket 405, and bucket 406 are U, W, Z, V, and M, respectively. It should be understood that since the 10 blocks from block 0 to block 9 are divided into 40 blocks in total, and only one hash value is calculated for each block, there are 40 records of (block ID, block ID) in the current key-value storage. In other words, there are multiple buckets and corresponding records ( Figure 4 Since these records do not affect the determination of the first block and the second block, Figure 4 These records are omitted.

[0148] Then, the first block can be selected from the processed blocks according to the hash value calculated for the input block. Specifically, for block 0, hash value X matches bucket 401, and then all records in bucket 401 are taken out from bucket 401, including (4, 2), (1, 3), and (5, 0). For block 1, no bucket matches hash value Y (assuming Figure 4 4, the hash value of the bucket not shown in 40 is not Y). For chunk 2 and chunk 3, the hash values ​​are Z and W, and the records (3, 3), (3, 1), (5, 3) are taken out from bucket 404, and the records (3, 2), (1, 1), (3, 0), (5, 2), (6, 3) are taken out from bucket 403. For these records, chunk 6, chunk 4, chunk 5, chunk 3, and chunk 1 can be used as dictionary chunks to compress the input chunks.

[0149] For more efficient compression, several blocks can be selected from the above five blocks as the first block. The results after sorting the records twice are as follows:

[0150] Group 1: (3, 3), (3, 1), (3, 2), (3, 0);

[0151] Group 2: (5, 0), (5, 2), (5, 3);

[0152] Group 3: (1, 3), (1, 1);

[0153] Group 4: (4, 2);

[0154] Group 5: (6, 3).

[0155] Group 1, Group 2, Group 3, Group 4, and Group 5 represent chunk 3, chunk 5, chunk 1, chunk 4, and chunk 6, respectively, where both Group 4 and Group 5 have one record. Chunk 3 is likely to be a dictionary chunk based on the hash values ​​calculated for chunk 2 and chunk 3 of the input chunk. Chunk 5 can help compress chunk 0, chunk 2, and chunk 3. Chunk 1 is expected to be a dictionary chunk to compress chunk 0 and chunk 3. Chunk 4 and Chunk 6 are used to compress chunk 0 and chunk 3, respectively.

[0156] As explained above, these 5 blocks can be used as dictionary blocks to compress input blocks. More ideally, several blocks that qualify as dictionary blocks or first blocks are selected according to certain rules. For example, the corresponding group of the first block needs to have at least 2 records, and the first block will be block 3, block 5, and block 1. Or the first block is determined as three largest groups, and then block 3, block 5, and block 1 are determined.

[0157] Since block 3 (the largest group) matches two blocks in total (matches block 2 twice and matches block 3 twice), in some embodiments, these matching blocks may be deleted in other groups, so the record (5, 2) corresponding to block 3 and the record (5, 3) corresponding to block 2 are deleted from group 2 to avoid duplication. In other words, the matching groups sorted by the number of matching blocks are assigned corresponding priorities, wherein the group with more matching blocks has a higher priority. Blocks with the same hash value are cleared, and then the groups with the remaining number of matching blocks are reordered. After performing similar steps on group 3, group 4, and group 5, the new groups are as follows:

[0158] Group 1: (3, 3), (3, 1), (3, 2), (3, 0);

[0159] Group 2: (5, 0).

[0160] Then after this step, groups 3, 4, and 5 are empty, and the first blocks are blocks 3 and 5. In addition, since block 5 only matches a single block in the input block, only block 3 may be considered as the first block for compressing the input block.

[0161] In an optional implementation, when evaluating the efficiency of the first block as a dictionary block for use, a plurality of second blocks are also selected from the first block. As described above, while compressing the input block, string matching is performed between the first block and the input block, wherein the block with the longer matching string is determined as the second block.

[0162] For example, block 1, block 3, and block 5 all meet the requirements to be the second block. Then these three blocks are useful dictionary blocks for compressing the input block, but only block 3 and block 5 may be effectively used to compress the input block, where block 3 and block 5 include all the strings required to compress the input block.

[0163] The unused first block will be discarded and the compressed result of the input blocks can be prefixed with the block identifiers of blocks 3 and 5.

[0164] Figure 5 is a schematic diagram describing the differential compression process provided by the present application.

[0165] The input block 504 is divided into an input block, or is not divided, and the input block 504 is divided into an input block 504 ... and the input block 504 is divided into an input block 504, and the input block 504 Figure 5The bytes shown are used to calculate two input hash values. After using the hash values ​​as keys to look up in the key-value store, three dictionary blocks (501, 502, 503) are determined. The string matching result is: blocks 501 and 502 have 1 word matched to input block 504, and block 503 has 3 words matched to input block 504. Then, the three dictionary blocks can be used to compress input block 504. Optionally, blocks 501 and 502 can be discarded because they only have 1 word matched to input block 504.

[0166] In the technical solution provided in the embodiment of the present application, the input block can be compressed according to multiple dictionary blocks (first block or second block), which improves the overall compression ratio of the input block.

[0167] After completing all the above steps, the input block can be compressed according to the dictionary block (first block or second block). If the number of dictionary blocks is zero, the input block is compressed independently. If at least one dictionary block is effectively used, the compressed data obtained with the block ID of the effectively used dictionary block as a prefix is ​​stored or transmitted by the network. The matching part of the input block and the dictionary block can be expressed as (a, b), where a is the offset of the first byte of the matching string and b is the total length of the matching string. The encoding results are different in different compression algorithms, such as Huffman coding or arithmetic coding.

[0168] Figure 6 is a schematic diagram describing the differential compression process. Figure 6 As shown, for input block 604, blocks 601, 602 and 603 are candidate dictionary blocks (may be the first block), and block 603 is a dictionary block (may be the second block). Then, the input blocks prefixed with the block ID of block 603 are transmitted to the compression area for compression.

[0169] For example, if more than one dictionary is selected as a dictionary block to compress input block 604, block 601 and block 603 are selected as dictionary blocks to compress input block 604. However, when performing string matching, block 601 as a dictionary block can be discarded because the matching string "ZETTA" already exists in block 603.

[0170] Since the input block has been compressed, the key-value store is updated with the input block hash calculation result. Figure 7 The key value storage update result is shown. Bucket 407 is a new bucket that stores the association between hash value Y and record (10, 1). Buckets 401, 403, and 404 are updated with records (10, 0), (10, 3), and (10, 2) respectively.

[0171] Similar steps are then repeated on the new input block (block 11). After all blocks have been processed, the data compression process ends.

[0172] It should be noted that the key-value store may not save all records of the processed blocks in the following cases:

[0173] (a) When a block is deleted, information about the block is deleted from the key-value store.

[0174] (b) If there is not enough memory (RAM) for the key-value storage, some information may be cleared, such as, but not limited to, information about blocks that have not been selected as dictionary blocks for a long time.

[0175] (c) If recursive dictionary blocks are not needed, no information about the differentially compressed blocks is stored in the key-value store. Recursive dictionary blocks will affect the speed of differential decompression.

[0176] Figure 8 is a flow chart of an embodiment of a method 700 for data compression. The method comprises the following steps.

[0177] S701: Calculate hash values ​​of one or more byte sets obtained from blocks from an input block according to a preset rule.

[0178] S702: Search one or more candidate dictionaries for the input block according to the calculated hash value.

[0179] S703: Prioritize the candidate dictionaries according to the number of hash values ​​that match the input block.

[0180] S704: Select one or more best candidate dictionaries to perform differential compression, and then perform differential compression on the input block.

[0181] S705: Determine one or more valid dictionaries from the candidate dictionaries according to the length of the matching string.

[0182] In S705, the length of the matching character string is greater than a threshold.

[0183] S706: Store or transmit the differentially compressed input block prefixed with the ID of the valid dictionary.

[0184] Fig. 9 is a flow chart of an embodiment of a method 800 for data compression. The method 800 includes the following steps.

[0185] S801: Divide the input block into blocks of fixed size.

[0186] In this step, the input block may be divided into a plurality of fixed-size blocks. S802: Select a byte set for hash calculation.

[0187] In this step, for a block, one or more sets of bytes can be selected for hashing. Selecting a small number of bytes instead of using the entire block for hashing can reduce the computational cost involved in the compression process.

[0188] S803: Calculate one or more hash values ​​for each byte set.

[0189] S804: Use the hash value as a key to search the key-value storage, and collect (block ID, segment ID) records with the same key.

[0190] In this step, the hash value is used as the key to find the first block in the key-value store, which was mentioned earlier.

[0191] S805: Sort the (block ID, split block ID) records to find the largest number of records with the same block ID1, the smaller number of records with other block ID2, and so on.

[0192] In this step, the blocks found in step S804 may be sorted by the number of blocks.

[0193] In an optional embodiment, blocks with a block number less than a threshold may be discarded to improve the quality of the dictionary block.

[0194] In another optional embodiment, a higher priority may be given to a block with a larger number of blocks. Similar blocks (blocks with the same hash value) may be eliminated from blocks with lower priority, which will improve the quality of candidate dictionary blocks.

[0195] S806: Retrieve the block as a dictionary block to compress the input block.

[0196] S807: Prepare a buffer for differential compression: copy the found dictionary block and copy the input block.

[0197] In this step, both the dictionary blocks and the input blocks are put into a buffer for compression.

[0198] S808: Compress the input block relative to the dictionary block.

[0199] S809: Is the compression successful?

[0200] S8091: Update the key-value store using the hash value of the input block.

[0201] S8092: Store incompressible input blocks.

[0202] In step S8091 and step S8092, the input block is not compressible. For example, it may include image data that has been compressed, so the input block is not successfully compressed. However, it can still be a candidate dictionary block for other blocks, so it is necessary to update the key value storage using the (block ID, chunk ID) record according to the hash value of the input block.

[0203] S810: Is the dictionary block used?

[0204] In this step, the input block is compressed successfully, but the dictionary block may be useful for compression of the input block or the dictionary block may not be used for compression of the input block.

[0205] S8101: Update the key-value storage according to the hash value of the input block.

[0206] S8102: Store the compressed block as a non-difference.

[0207] In step S8101 and step S8102, the input block is compressed successfully, but no dictionary block is used for compression of the input block. The matching length of the dictionary block may not exceed the threshold, so the input block is compressed independently. The key-value store needs to be updated with the input block.

[0208] S811: Discard unused dictionary blocks.

[0209] S812: Store the compressed block as a difference, and use the used block ID as a prefix of the input block after the difference compression.

[0210] In this step, the input block is successfully compressed using one or more candidate dictionary blocks retrieved in step S806, which have matching strings longer than a threshold. Unused blocks can then be discarded to improve dictionary block quality. The compressed block is prefixed with the used dictionary block ID.

[0211] Fig.10 is a schematic block diagram of an electronic device 900 provided in an embodiment of the present application. Fig.10 As shown, the electronic device 900 includes: a division module 901, which is used to divide an input block into multiple input blocks, wherein each of the multiple input blocks includes X bytes; a selection module 902, which is used to select one or more byte sets from the X bytes of each input block in the multiple input blocks according to a preset rule, wherein the one or more byte sets include Y bytes selected from the X bytes that are discontinuous in the original order; a calculation module 903, which is used to calculate one or more input hash values ​​for each input block in the multiple input blocks based on the one or more byte sets; and a compression module 904, which is used to compress the input block according to the input hash values ​​calculated by the calculation module for the multiple input blocks.

[0212] In the above technical solution, the partitioning module 901 is used to partition the input block into one or more sub-blocks. The selection module 902 is used to select one or more byte sets for hash calculation by the calculation module 903. The hash calculation result will be used by the compression module 904 to compress the input block. Selecting a small number of bytes instead of the entire sub-block for hash calculation reduces the computational cost involved in the compression process and improves the compression efficiency.

[0213] In some embodiments, when there are multiple byte sets for each input block, the difference in offset0 position between the first byte set and any one of the other byte sets is 2m, where m is a natural number unique to different byte sets, the first byte set is one of the multiple byte sets, and the offset0 position is the position of the first byte of each of the one or more byte sets.

[0214] In some embodiments, the preset rule includes: the difference in byte position between the discontinuous Y bytes and the offset0 position satisfies a modified Fibonacci sequence or a logarithmic function sequence.

[0215] In some embodiments, the compression module 904 is specifically used to: determine one or more first blocks based on the input hash value, wherein the one or more first blocks are processed blocks, and each of the one or more first blocks includes one or more first sub-blocks whose hash value is the same as one of the input hash values; and compress the input block based on the one or more first blocks.

[0216] In some embodiments, the compression module 904 is used to: determine one or more first blocks based on the input hash value, wherein each of the one or more first blocks is a processed block and the hash value is the same as one of the input hash values; determine one or more first blocks based on the one or more first blocks, wherein the one or more first blocks are blocks divided into one or more first blocks.

[0217] In some embodiments, the one or more first chunks are determined based on a key-value store, wherein the key-value store stores an association between hash values ​​of the processed chunks and corresponding chunk identifiers and chunk identifiers.

[0218] In some embodiments, the number of the one or more first blocks is greater than a first preset threshold.

[0219] In some embodiments, the electronic device further includes a determination module, which is used to determine one or more second blocks from one or more first blocks based on the length of a matching string between the one or more first blocks and the input block, wherein the length of the matching string is greater than a second preset threshold.

[0220] In some embodiments, the result of compressing the input block includes block identifiers of one or more second blocks.

[0221] In some embodiments, the electronic device further comprises an update module for updating the key-value storage using the chunk identifier, the chunk identifier and the corresponding input hash value of the input chunk.

[0222] like Fig.11 As shown, the electronic device 1000 may include a transceiver 1001, a processor 1002, and a memory 1003. The memory 1003 may be used to store codes, instructions, etc. executed by the processor 1002.

[0223] It should be understood that the processor 1002 can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method embodiment can be completed by a hardware integrated logic circuit in the processor or an instruction in the form of software. The above processor can be a general processor, a digital signal processor (digital signal processor, DSP), an application specific integrated circuit (application specific integrated circuit, ASIC), a field programmable gate array (field programmable gate array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The processor can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general processor can be a microprocessor, or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application can be directly executed by a hardware decoding processor, or can be executed using a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory or a register. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with the hardware in the processor.

[0224] It can be understood that the memory 1003 in the embodiment of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. The non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM) used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0225] The embodiment of the present application also provides a chip system, wherein the chip includes an input / output interface, at least one processor and at least one memory. The at least one memory is used to store instructions, and the at least one processor is used to call the instructions of the at least one memory to execute the operation performed by the electronic device in the method of the above embodiment.

[0226] An embodiment of the present application also provides a computer storage medium, wherein the computer storage medium can store program instructions to execute the steps in the above method.

[0227] Optionally, the storage medium may specifically be the memory 1003 .

[0228] An embodiment of the present application further provides a computer program product, wherein when the computer program product is run on an electronic device, the electronic device is enabled to execute the steps in the above method.

[0229] Those skilled in the art will appreciate that, in conjunction with the various examples described in the embodiments disclosed in this specification, the units and algorithm steps can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed by hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0230] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-mentioned systems, devices and units can refer to the corresponding processes in the above-mentioned method embodiments, which will not be described in detail here.

[0231] In the embodiments of the present application, "at least one" means one or more, and "plurality" means two or more. The term "and / or" describes the association relationship between associated objects, indicating that there may be three relationships. For example, A and / or B can represent the following three situations: only A exists, A and B exist at the same time, and only B exists, where A and B can be singular or plural. The character "I" usually represents the "or" relationship between associated objects. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of one item or multiple items. For example, at least one of a, b and c can represent: a, b, c, a and b, a and c, b and c, or a, b and c, where a, b and c can be singular or plural.

[0232] In several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the described device embodiments are merely exemplary. For example, unit division is merely a logical function division, and other division methods may be used in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not performed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be implemented through some interfaces. The indirect coupling or communication connection between devices or units may be implemented in electronic, mechanical or other forms.

[0233] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, may be located in one location, or may be distributed over multiple network units. Some or all of the units may be selected based on actual needs to achieve the purpose of the solution of the embodiment.

[0234] In addition, the functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0235] When these functions are implemented in the form of software functional units and sold or used as independent products, these functions can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or a part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for instructing a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in the embodiment of the present application. The above-mentioned storage medium includes: any medium that can store program code, such as a USB flash drive, a removable hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk.

[0236] The above description is only a specific implementation of the present application and is not intended to limit the scope of protection of the present application. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application shall be subject to the scope of protection of the claims.

Claims

1. A data compression method, characterized in that: include: Divide the input block into a plurality of input blocks, wherein each input block in the plurality of input blocks includes X bytes; According to a preset rule, one or more byte sets are selected from the X bytes of each input block in the multiple input blocks, wherein the one or more byte sets include Y bytes selected from the X bytes that are not consecutive in the original order; Based on the one or more sets of bytes, calculating one or more input hash values ​​for each input block in the plurality of input blocks; The input block is compressed according to the input hash values ​​of the plurality of input blocks.

2. The method according to claim 1, characterized in that When the number of byte sets is multiple for each input block, the difference in the offset0 position between the first byte set and any of the other byte sets is 2 m , wherein m is a natural number unique to different byte sets, the first byte set is a byte set among the multiple byte sets, and the offset0 position is the position of the first byte of each byte set in the one or more byte sets.

3. The method according to claim 2, characterized in that The preset rules include: The difference in byte position between the discontinuous Y bytes and the offset0 position satisfies a modified Fibonacci sequence or a logarithmic function sequence.

4. The method according to any one of claims 1 to 3, characterized in that The compressing the input block according to the input hash values ​​of the plurality of input blocks comprises: Determine one or more first blocks according to the input hash value, wherein the one or more first blocks are processed blocks, and each of the one or more first blocks includes one or more first sub-blocks whose hash value is the same as one of the input hash values; The input block is compressed based on the one or more first blocks.

5. The method according to claim 4, characterized in that Determining one or more first blocks according to the input hash value comprises: Determining the one or more first blocks based on the input hash value, wherein each of the one or more first blocks is a processed block and has a hash value that is the same as one of the input hash values; The one or more first blocks are determined based on the one or more first partitions, wherein the one or more first blocks are blocks divided into the one or more first partitions.

6. The method according to claim 4 or 5, characterized in that: The one or more first segments are determined based on a key-value store that stores an association between a hash value of the processed segment and a corresponding segment identifier and a segment identifier.

7. The method according to any one of claims 4 to 6, characterized in that The number of the one or more first blocks is greater than a first preset threshold.

8. The method according to any one of claims 4 to 7, characterized in that After compressing the input block based on the one or more first blocks, the method comprises: One or more second blocks are determined from the one or more first blocks according to the length of the matching string between the one or more first blocks and the input block, wherein the length of the matching string is greater than a second preset threshold.

9. The method according to claim 8, characterized in that The result of compressing the input block includes block identifiers of the one or more second blocks.

10. The method according to any one of claims 1 to 9, characterized in that The method further comprises: The key-value store is updated using the chunk identifier, the chunk identifier, and the corresponding input hash value of the input chunk.

11. An electronic device, characterized in that: include: A partitioning module, configured to partition an input block into a plurality of input sub-blocks, wherein each of the plurality of input sub-blocks comprises X bytes; A selection module, configured to select one or more byte sets from the X bytes of each input block in the multiple input blocks according to a preset rule, wherein the one or more byte sets include Y bytes selected from the X bytes that are discontinuous in original order; a calculation module, configured to calculate one or more input hash values ​​for each of the plurality of input blocks based on the one or more sets of bytes; A compression module is used to compress the input block according to the input hash value calculated by the calculation module for the multiple input blocks.

12. The electronic device according to claim 11, characterized in that: When the number of byte sets is multiple for each input block, the difference in the offset0 position between the first byte set and any of the other byte sets is 2 m , wherein m is a natural number unique to different byte sets, the first byte set is a byte set among the multiple byte sets, and the offset0 position is the position of the first byte of each byte set in the one or more byte sets.

13. The electronic device according to claim 11, characterized in that: The preset rules include: The difference in byte position between the discontinuous Y bytes and the offset0 position satisfies a modified Fibonacci sequence or a logarithmic function sequence.

14. The electronic device according to any one of claims 11 to 13, characterized in that: The compression module is used to: Determine one or more first blocks according to the input hash value, wherein the one or more first blocks are processed blocks, and each of the one or more first blocks includes one or more first sub-blocks whose hash value is the same as one of the input hash values; The input block is compressed based on the one or more first blocks.

15. The electronic device according to claim 14, characterized in that: The compression module is used to: Determining the one or more first blocks based on the input hash value, wherein each of the one or more first blocks is a processed block and has a hash value that is the same as one of the input hash values; The one or more first blocks are determined based on the one or more first partitions, wherein the one or more first blocks are blocks divided into the one or more first partitions.

16. The electronic device according to claim 14 or 15, characterized in that: The one or more first segments are determined based on a key-value store that stores an association between a hash value of the processed segment and a corresponding segment identifier and a segment identifier.

17. The electronic device according to any one of claims 14 to 16, characterized in that: The number of the one or more first blocks is greater than a first preset threshold.

18. The electronic device according to any one of claims 14 to 17, characterized in that: The electronic device also includes a determination module, which is used to determine one or more second blocks from the one or more first blocks according to the length of a matching string between the one or more first blocks and the input block, wherein the length of the matching string is greater than a second preset threshold.

19. The electronic device according to claim 18, characterized in that: The result of compressing the input block includes block identifiers of the one or more second blocks.

20. The electronic device according to any one of claims 11 to 19, characterized in that: The electronic device further comprises an update module for updating the key-value storage using the chunk identifier, the chunk identifier and the corresponding input hash value of the input chunk.

21. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, which, when executed on a device, cause the device to perform the method according to any one of claims 1 to 10.

22. A computer program product, characterized in that When the computer program product is run on a device, the device is caused to perform the method according to any one of claims 1 to 10.

23. A chip system, characterized in that: It comprises a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to call the computer program from the memory and run the computer program, so that the device where the chip system is located executes the method according to any one of claims 1 to 10.

Images