Data lossless compression and decoding method combining rANS and LZ4 encoding

Abstract

The present invention provides a data lossless compression method combining rANS and LZ4 encoding, comprising: using rANS to calculate the compressed data state value of input original data; converting the compressed data state value into a character state value; using the LZ4 block structure to match and compress the character set state value to obtain compressed data. The present invention utilizes the advantages of high compression efficiency of rANS and fast compression speed of LZ4 to achieve a higher compression rate at a faster compression speed, thereby improving the performance of the lossless data compression algorithm.

Description

Technical Field

[0001] The present invention relates to the technical field of data lossless compression, and in particular to a data lossless compression and decoding method, system and medium combining rANS and LZ4 encoding. Background Art

[0002] With the development of digital communication, data has shown explosive growth, and it takes a considerable cost to store and transmit data. The large amount of diversified big data that has emerged at high speed in modern times has brought great difficulties to the storage and transmission of data. Reducing the size of data can store more data in the same storage space and transmit more data faster in limited bandwidth. Therefore, efficient data compression technology is crucial.

[0003] Data compression includes lossy compression and lossless compression. Lossy compression cannot accurately restore the original data, but has a high compression rate. It is widely used to store images, videos and audio files because partial loss of these data is acceptable and the human visual system cannot perceive such distortion. Lossless compression can accurately restore the original data from the compressed data without involving data loss, thus maintaining the integrity of the data. Lossless compression is usually used to compress text content, medical images, military images, satellite remote sensing images and other data that require high accuracy and do not allow errors. Taking medical images as an example, modern medical diagnosis relies heavily on digital images. In order to accurately analyze human health problems, these medical images should be saved in a lossless format without losing any key parts. For remote medical diagnosis, efficient lossless compression technology can save image transmission time and improve treatment efficiency. Therefore, it is of great significance to improve the compression performance of lossless compression technology.

[0004] Common lossless compression methods are generally based on entropy coding or dictionary coding. Entropy coding is to use the statistical probability of a symbol sequence for compression so that the compression result is as close to its entropy value as possible. It is commonly used in image compression and video compression methods. The current mainstream entropy coding techniques include the widely used Huffman coding and arithmetic coding, as well as the asymmetric numeral system (ANS) proposed in recent years. Dictionary-based coding methods are useful when the original data contains more repeated patterns. When a repeated pattern appears in the input sequence, it is encoded using a dictionary index. The Lempel-Ziv method (LZ) is a dictionary-based coding technique that is often used for lossless file compression. There are a large number of variants of the LZ method. In addition to LZ77 and LZ78, there are many other variants, such as LZW, LZMA, LZSS, LZFSE, LZ4, Snappy, and Zstandard. There are other lossless compression methods, including run-length coding and BWT transform.

[0005] However, most lossless compression methods cannot achieve both compression speed and compression rate. To achieve a high compression rate, the compression speed is generally slower, such as Huffman coding and arithmetic coding. If you want a faster compression speed, you need to compromise on the compression rate. Summary of the invention

[0006] In view of the defects in the prior art, the object of the present invention is to provide a data lossless compression and decoding method, system and medium combining rANS and LZ4 encoding.

[0007] According to one aspect of the present invention, there is provided a data lossless compression method combining rANS and LZ4 encoding, comprising:

[0008] Utilize rANS to calculate the compressed data state value of the input original data;

[0009] Converting the compressed data status value into a character type status value;

[0010] The character state value is matched and compressed using the LZ4 block structure to obtain compressed data.

[0011] Preferably, the step of calculating the compressed data state value of the input original data by using rANS includes:

[0012] Initialize rANS encoder;

[0013] Limiting the compressed data state value to be within a normalized interval I,

[0014] ,

[0015] Where L is an arbitrary positive integer, b is an integer greater than or equal to 2;

[0016] Calculate the probability of occurrence of the symbol of the input original data , and limiting the sum of the occurrence probabilities of the original data symbols to 1;

[0017] Assume that the original data consists of different symbols If the previous Symbols After rANS encoding, continue to input symbols Perform rANS encoding and set the current state value to , according to the state transition formula Loop iterative calculation outputs the next state value :

[0018] ,

[0019] Where n is the set quantization coefficient, , express Modulus , and get the remainder after dividing the two.

[0020] Preferably, the rANS encoder stores the encoded symbols in reverse order.

[0021] Preferably, converting the compressed data status value into a character type status value comprises:

[0022] After rANS encoding is completed, the final compressed data state value is obtained;

[0023] According to the size of the final compressed data state value, providing a corresponding maximum memory space for the LZ4 block structure;

[0024] The final compressed data state value is converted into a character type.

[0025] Preferably, the LZ4 block structure consists of sequences, each sequence sequentially comprising:

[0026] Token block, each sequence starts with a one-byte Token; the high 4 bits of each Token record are not matched to the uncompressed literal sequence length, and the low 4 bits record the matching sequence length;

[0027] Literal sequence length block. If the length of the sequence exceeds the upper 4 bits of the Token, any number of bytes will be extended after the Token.

[0028] A literal sequence block, wherein the literal sequence block stores uncompressed literals;

[0029] An offset block, wherein the offset block records the distance from the current character to the matching item;

[0030] Match the sequence length block. If the sequence length exceeds the lower 4 bits of the Token, any number of bytes will be extended after the offset block.

[0031] Using the offset and the matching sequence length, the decoder copies the repeated data from the decoded buffer.

[0032] Preferably, the matching and compressing the character state value using the LZ4 block structure to obtain compressed data includes:

[0033] Read the character data state value obtained by the final rANS encoding;

[0034] Each byte is traversed, 4 bytes are read to calculate the hash value, and a match is found in the hash table;

[0035] If it matches, go to the next step; if it does not match, store the hash value and the current traversal position in the hash table and continue traversing the bytes;

[0036] When a match is found in the hash table, a one-byte Token is constructed in the output buffer, and the literal length is stored in the high 4 bits of the Token. If the literal length exceeds the high 4 bits of the Token, the literal sequence length block after the Token block continues to extend the bytes that exceed the limit until all literal sequence lengths are stored.

[0037] Subtract the current position from the matching position to get the offset, and store it in the 2 bytes after the literal sequence length block;

[0038] Continue to compare bytes from the matching position to get the longest matching sequence, and store the length of the matching sequence in the lower 4 bits of the Token. If there is a part exceeding the 4-bit length, it will be stored in the bytes after the offset block until all matching sequence lengths are stored;

[0039] Repeat the traversal of the bytes until all the data is compressed to obtain compressed data.

[0040] According to a second aspect of the present invention, a data lossless decoding method combining rANS and LZ4 encoding is provided, wherein the decoding process is opposite to the compression process in the data lossless compression method combining rANS and LZ4 encoding, and the compression process is performed in reverse order.

[0041] Preferably, the decoding process includes:

[0042] Read the compressed data to be decompressed;

[0043] Reading a first sequence from the LZ4 compressed data block as a current operation position;

[0044] The current operation position is regarded as Token, and the high 4 bits of data are read as the length of the literal sequence; if the length is 15, it means that the length of the literal sequence exceeds the high 4 bits of Token, then continue to read the bytes after Token and add them to the length of the literal sequence;

[0045] Read the literal sequence according to the length of the literal sequence and output it to the cache;

[0046] Read the 2-byte offset and trace back the offset bytes from the current position of the output buffer to get the matching position;

[0047] Read the matching length, obtain the matching sequence according to the matching position, and output it to the cache;

[0048] Continue decompression until all LZ4 blocks are decompressed and the character-type rANS status value is stored in the cache;

[0049] The character-type rANS status value in the buffer is converted into an integer type, and rANS decoding is performed on it to obtain the original data.

[0050] According to a third aspect of the present invention, there is provided a data lossless compression and decoding system combining rANS and LZ4 encoding, comprising:

[0051] Compression system: compressing the input original data using the data lossless compression method combining rANS and LZ4 encoding to obtain compressed data;

[0052] Decompression system: Decompress the compressed data using the data lossless compression decoding method combining rANS and LZ4 encoding to obtain the input original data.

[0053] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, wherein the storage medium stores at least one instruction, at least one program, a code set or an instruction set, wherein the at least one instruction, the at least one program, the code set or the instruction set is loaded by a processor and executes any one of the following methods:

[0054] - the data lossless compression method combining rANS and LZ4 encoding, or,

[0055] - The data lossless compression and decoding method combining rANS and LZ4 encoding.

[0056] Compared with the prior art, the present invention has the following beneficial effects:

[0057] The embodiment provided by the present invention combines the characteristics of rANS and LZ4 for lossless data compression. The dictionary encoding method LZ4 changes the disadvantage of the slow compression speed of traditional dictionary encoding in the past, and focuses on improving the compression speed. Its compression speed is currently the fastest among all lossless compression encodings. Therefore, the data is compressed using the block structure of LZ4 with extremely fast compression speed.

[0058] The rANS in the embodiment provided by the present invention is a variant of the new generation entropy coding asymmetric digital system ANS, which has a compression speed close to Huffman coding and a compression efficiency of arithmetic coding, and can compress and process large-scale data. After using rANS to calculate the data to obtain the compressed data state character set, it is matched and compressed using the LZ4 block structure, and a higher compression rate can be obtained at a faster speed.

[0059] The embodiment provided by the present invention utilizes the low algorithm complexity of rANS and LZ4, and is convenient for compression and decompression. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0061] Figure 1 A flowchart of a compression process according to an embodiment of the present invention;

[0062] Figure 2 A block structure diagram of LZ4 according to a preferred embodiment of the present invention;

[0063] Figure 3 A flowchart of the LZ4 compression algorithm process of a preferred embodiment of the present invention;

[0064] Figure 4 A histogram comparing the compression ratios of the preferred embodiment of the present invention with various lossless compression methods on the Silesia Corpus dataset. DETAILED DESCRIPTION

[0065] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several variations and improvements may be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0066] The present invention provides a preferred embodiment, a data lossless compression method combining rANS and LZ4 encoding, comprising:

[0067] S1, using the rANS compressor to calculate the compressed data state value of the input original data;

[0068] S2, converting the compressed data state value into a character state value;

[0069] S3. Use the LZ4 block structure to match and compress the character state value to obtain compressed data.

[0070] LZ4 coding changes the disadvantage of slow compression speed of traditional dictionary coding in the past, and focuses on improving compression speed. Its compression speed is currently the fastest among all lossless compression coding. rANS is a variant of the new generation of entropy coding asymmetric digital system ANS, which has a compression speed close to Huffman coding and compression efficiency of arithmetic coding, and can compress and process large-scale data. This embodiment combines the advantages of high compression efficiency of rANS and fast compression speed of LZ4, achieves a higher compression rate at a faster compression speed, and improves the performance of the lossless data compression algorithm.

[0071] In order to improve the compression rate, the present invention provides a preferred embodiment, a data lossless compression method combining rANS and LZ4 encoding, referring to Figure 1 ,include:

[0072] S101, read original data;

[0073] S102, initializing the rANS encoder; in order to prevent the state value of the rANS encoding result from exceeding the range, the state value is limited to the normalized interval I:

[0074] ,

[0075] Wherein L is an arbitrary positive integer, and b is an integer greater than or equal to 2. In the present invention, the values ​​of L and b are , so the value range of the state value x is { , , }.

[0076] S103, calculate the probability of occurrence of symbols of input data , and limit the sum of the input data symbol probabilities to 1;

[0077] S104, rANS encoding obtains an integer status value, including:

[0078] S1041, assuming that the original data consists of different symbols If the previous Symbols After rANS encoding, continue to input symbols Perform rANS encoding, assuming the current state value is , according to the state transition formula Loop iterative calculation outputs the next state value :

[0079] ,

[0080] Where n is the set quantization coefficient, and the value of n in this embodiment is 14. , which is essentially the cumulative probability of a given symbol sequence; express Modulus In this embodiment, , Set to , 0.

[0081] S1042, since the rANS encoder is similar to a stack with LIFO, the last value after encoding will be the first value returned by the decoder. Therefore, the encoded symbols need to be stored in reverse order, that is, the encoder processes the stored data in reverse order (from the end to the front end), and the decoder processes the read data in forward order (from the front end to the end).

[0082] S1043, after the input data rANS encoding is completed to obtain the final state value, the state value size is calculated and the corresponding maximum memory space is allocated for the LZ4 block structure.

[0083] S105, converting the final state integer value of rANS into a character type;

[0084] S106, performing LZ4 block structure compression encoding on the character type;

[0085] S107, obtain compressed data.

[0086] In order to improve the compression speed, the present invention provides a preferred embodiment. Figure 2 As shown, it is a schematic diagram of the structure of the LZ4 compression block in this embodiment. Each sequence starts with a one-byte token (Token). The upper 4 bits record the length of the uncompressed literal sequence (Literal length) that is not matched, and the lower 4 bits record the length of the matching sequence (Matchlength). If the length exceeds 4 bits, any number of bytes can be extended after the Token. After recording the literal length, the uncompressed literals (Literals) are stored. The offset (Offset) records the distance from the current character to the matching item. Using the offset and the matching sequence length, the decoder can copy repeated data from the decoded buffer.

[0087] Further, compression is performed based on the LZ4 compression block in the above preferred embodiment, such as Figure 3 As shown, including:

[0088] S201, read the final state value of rANS converted into character type in the previous step;

[0089] S202, traverse each byte, read 4 bytes and calculate the hash value;

[0090] S203, search for a match in the hash table; if there is a match, proceed to the next step S204; if there is no match, store the hash value and the current traversed position in the hash table and continue traversing the bytes;

[0091] S204, if a match is found in the hash table, construct an LZ4 block structure, including:

[0092] S2041, construct a byte Token in the output buffer, store the literal length in the high 4 bits of the Token, if the literal length exceeds the high 4 bits of the Token, then the literal sequence length block after the Token block continues to extend the bytes exceeding the portion until all literal sequence lengths are stored;

[0093] S2042, subtract the current position from the matching position to obtain an offset, and store it in the 2 bytes after the literal sequence;

[0094] S2043, continue to compare bytes from the matching position to obtain the longest matching sequence, and store the length of the matching sequence in the lower 4 bits of the Token. If there is a part exceeding the 4-bit length, it is stored in the bytes after the offset block until all matching sequence lengths are stored;

[0095] S205, repeat S202 until all data are compressed to obtain the final compressed data.

[0096] The Silesia Corpus dataset is compressed by combining the rANS and LZ4 data lossless compression methods of the above embodiments. The compression results are compared with other different lossless compression algorithms (including arithmetic coding, Huffman coding, rANS and LZ4). The compression result index is expressed by compression ratio:

[0097] ,

[0098] The smaller the compression ratio, the higher the compression rate and the better the compression effect. Figure 4 As shown, it can be seen that the compression ratio of the rANS+LZ4 algorithm proposed in the present invention is improved by an average of 73.65%, 75.03%, 74.92% and 68.65% respectively compared with arithmetic coding, Huffman coding, rANS and LZ4 in terms of compression rate. The compression ratio is the smallest and the compression effect is the best.

[0099] This embodiment also calculates the time taken by the above-mentioned data lossless compression algorithm combining rANS and LZ4 and other lossless compression algorithms to compress the Silesia Corpus dataset. The experimental results are shown in Table 1. It can be seen that although the compression time is slower than the fastest LZ4 method, it is reduced by 83.98%, 81.26% and 4.86% on average compared with arithmetic coding, Huffman coding and rANS, respectively.

[0100]

[0101] In summary, it can be seen that the advantages of the high compression efficiency of the new generation entropy coded asymmetric digital system rANS and the fast compression speed of LZ4 are combined to achieve a higher compression rate at a faster compression speed and improve the performance of the lossless data compression algorithm.

[0102] Based on the same concept, the present invention provides a data lossless decoding method combining rANS and LZ4 encoding, wherein the decoding process is opposite to the compression process, and each compression step is performed in reverse order.

[0103] The present invention provides a preferred embodiment for decoding compressed data, comprising:

[0104] S301, reading compressed data to be decompressed;

[0105] S302, reading the first sequence from the LZ4 compressed data block as the current operation position;

[0106] S303, consider the current operation position as Token, take the high 4 bits of data as the length of the literal sequence. If the length is 15, it means that the length of the literal sequence exceeds the high 4 bits of Token, continue to read the bytes after Token, and add them to the length of the literal sequence;

[0107] S304, reading the literal sequence according to the length of the literal sequence and outputting it to the cache;

[0108] S305, read the offset of 2 bytes, trace back the offset bytes from the current position of the output buffer to obtain the matching position;

[0109] S306, read the matching length in a similar manner, obtain a matching sequence according to the matching position, and output it to the cache;

[0110] S307, continue decompression until all LZ4 blocks are decompressed, and then the character-type rANS status value is stored in the cache;

[0111] S308, converting the character-type rANS status value in the cache into an integer type, performing rANS decoding on the integer to obtain the original data.

[0112] Based on the same concept, the present invention also provides a data lossless compression system combining rANS and LZ4 encoding, comprising:

[0113] Compression system: compressing the input raw data using the above-mentioned data lossless compression method combining rANS and LZ4 encoding to obtain compressed data;

[0114] Decompression system: Decompress the compressed data using the above-mentioned data lossless compression decoding method combining rANS and LZ4 encoding to obtain the input original data.

[0115] The present invention also provides a computer-readable storage medium, wherein the storage medium stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded by a processor and executes any of the following methods:

[0116] - the above lossless data compression method combining rANS and LZ4 encoding, or,

[0117] - The above-mentioned data lossless compression and decoding method combining rANS and LZ4 encoding.

[0118] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various modifications or variations within the scope of the claims, which does not affect the essence of the present invention. The above preferred features can be used in any combination without conflicting with each other.

Claims

1. A data lossless compression method combining rANS and LZ4 encoding, characterized in that: include: Calculate the compressed data state value of the input raw data using the rANS encoder; Converting the compressed data status value into a character type status value; Using the LZ4 block structure to match and compress the character state value to obtain compressed data; The method of calculating the compressed data state value of the input original data by using the rANS encoder includes: Initialize rANS encoder; Limiting the compressed data state value to be within a normalized interval I, ， Where L is an arbitrary positive integer, b is an integer greater than or equal to 2; Calculate the probability of occurrence of the symbol of the input original data , and limiting the sum of the occurrence probabilities of the original data symbols to 1; The original data consists of different symbols constitute; if the previous Symbols After rANS encoding, continue to input symbols Perform rANS encoding and set the current compressed data state value to , according to the state transition formula Loop iterative calculation outputs the next compressed data state value : ， Where n is the set quantization coefficient, , express Modulus , get the remainder after dividing the two; The LZ4 block structure consists of sequences, each of which includes: Token block, each sequence starts with a one-byte Token; the high 4 bits of each Token block record are not matched to the uncompressed literal sequence length, and the low 4 bits record the matching sequence length; Literal sequence length block. If the length of the sequence exceeds the upper 4 bits of the Token, any number of bytes will be extended after the Token block. A literal sequence block, wherein the literal sequence block stores uncompressed literals; An offset block, wherein the offset block records the distance from the current character to the matching item; Match the sequence length block. If the sequence length exceeds the lower 4 bits of the Token, any number of bytes will be extended after the offset block. Using the offset and the matching sequence length, the decoder copies the repeated data from the decoded buffer.

2. The method for lossless data compression combining rANS and LZ4 encoding according to claim 1, characterized in that: The rANS encoder stores the encoded symbols in reverse order.

3. The method for lossless data compression combining rANS and LZ4 encoding according to claim 1, characterized in that: The step of converting the compressed data status value into a character type status value includes: After rANS encoding is completed, the final compressed data state value is obtained; According to the size of the final compressed data state value, providing a corresponding maximum memory space for the LZ4 block structure; The final compressed data state value is converted into a character type.

4. The method for lossless data compression combining rANS and LZ4 encoding according to claim 3, characterized in that: The method of using the LZ4 block structure to match and compress the character state value to obtain compressed data includes: Read the character data state value obtained by the final rANS encoding; Each byte is traversed, 4 bytes are read to calculate the hash value, and a match is found in the hash table; If it matches, go to the next step; if it does not match, store the hash value and the current traversal position in the hash table and continue traversing the bytes; When a match is found in the hash table, a one-byte Token is constructed in the output buffer, and the literal length is stored in the high 4 bits of the Token. If the literal length exceeds the high 4 bits of the Token, the literal sequence length block after the Token block continues to extend the bytes that exceed the limit until all literal sequence lengths are stored. Subtract the current position from the matching position to get the offset, and store it in the 2 bytes after the literal sequence length block; Continue to compare bytes from the matching position to get the longest matching sequence, and store the length of the matching sequence in the lower 4 bits of the Token. If there is a part exceeding the 4-bit length, it will be stored in the bytes after the offset block until all matching sequence lengths are stored; Repeat the traversal of the bytes until all the data is compressed to obtain compressed data.

5. A data lossless decoding method combining rANS and LZ4 encoding, characterized in that: The decoding process is opposite to the compression process in the data lossless compression method combining rANS and LZ4 encoding as described in any one of claims 1 to 4, and the compression process is performed in reverse order.

6. The method for lossless decoding of data combining rANS and LZ4 encoding according to claim 5, characterized in that: include: Read the compressed data to be decompressed; Reading a first sequence from the LZ4 compressed data block as a current operation position; The current operation position is regarded as a token, and the high 4 bits of data are read as the length of the literal sequence; If the length is 15, it means that the length of the literal sequence exceeds the high 4 bits of the Token. Continue to read the bytes after the Token and add them to the length of the literal sequence. Read the literal sequence according to the length of the literal sequence and output it to the cache; Read the 2-byte offset and trace back the offset bytes from the current position of the output buffer to get the matching position; Read the matching length, obtain the matching sequence according to the matching position, and output it to the cache; Continue decompression until all LZ4 blocks are decompressed and the character-type rANS status value is stored in the cache; The character-type rANS status value in the buffer is converted into an integer type, and rANS decoding is performed on it to obtain the original data.

7. A data lossless compression and decompression system combining rANS and LZ4 encoding, characterized in that: include: Compression system: compressing the input original data by using the data lossless compression method combining rANS and LZ4 encoding as described in any one of claims 1 to 4 to obtain compressed data; Decompression system: Decode the compressed data using the data lossless compression decoding method combining rANS and LZ4 encoding as described in any one of claims 5-6 to obtain the input original data.

8. A computer-readable storage medium, characterized in that: The storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded by the processor and executes any one of the following methods: - the data lossless compression method combining rANS and LZ4 encoding as claimed in any one of claims 1 to 4, or, - The data lossless decoding method combining rANS and LZ4 encoding as described in any one of claims 5-6.

Images