Data compression method and device based on arithmetic coding
By using technologies such as chunking sorting, bitmap encoding and arithmetic encoding in data compression, the existing lossless compression algorithm fails to make full use of adjacent character dependencies and the minimum encoding length limit of Hoffman encoding, achieving more efficient data compression effects.
Patent Information
- Application Number
- CN202411972369.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-30
- Publication Date
- 2025-04-25
AI Technical Summary
The existing lossless compression algorithms fail to fully consider the dependencies between adjacent characters when processing data, and the minimum encoding length of Hoffman encoding is 1, resulting in the failure to achieve the optimal compression effect when the data information entropy is small.
The data compression method based on arithmetic encoding is adopted, and the data is decomposed into different parts through steps such as chunking sorting, bitmap encoding, character encoding and length encoding, and the data is compressed using context-based adaptive binary arithmetic encoding and finite state entropy algorithm.
The data compression rate is improved, and data with information entropy less than 1 can be effectively processed, which overcomes the limitation of the minimum encoding length of Hoffman encoding of 1, and reduces the overhead of data storage and transmission.
Smart Images
Figure CN119892110A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data compression, and in particular to a data compression method and device based on arithmetic coding. Background Art
[0002] With the development of information technology, all kinds of data have shown explosive growth, and the cost of data storage and transmission has become increasingly large. How to design a more efficient data compression algorithm has become one of the urgent problems to be solved in the field of information technology.
[0003] Data compression algorithms are mainly divided into two categories: lossy compression algorithms and lossless compression algorithms. The main idea of lossy compression algorithms is to reduce data quality in exchange for compression rate, but such algorithms cannot restore compressed data to original data, and are usually only applicable to the compression of audio, video, photo and other data. Lossless compression can restore compressed data to original data, and is mainly used in databases, electronic documents and various communication data.
[0004] The existing lossless compression algorithms mainly include LZ77, ZSTD, Gzip, Snappy, etc. The main idea of these algorithms is to maintain a dictionary and a sliding window, use the dictionary to replace the duplicate data in the sliding window, and then encode and compress the processed data through Huffman coding; however, these methods of replacing duplicate data through dictionaries do not take into account the dependency between adjacent characters. In addition, these methods usually compress data based on Huffman coding; however, the minimum coding length of Huffman coding is 1. According to the information entropy theory, when the probability of a certain data appearing is large, it means that the minimum coding length of the data is less than 1, so Huffman coding cannot achieve the best compression effect.
[0005] In order to solve the above problems, the Bzip2 data compression algorithm uses the Burrows Wheeler Transform (BWT) algorithm to perform a reversible transformation on the data. After shifting the characters in the data right one by one, the starting characters are sorted in order of value, thereby converting the data into a large number of continuous repeated characters based on the dependency relationship between adjacent characters. Then the Move-to-front (MTF) algorithm is used to encode the transformed data, and finally "1-2 encoding" and Huffman encoding are used for data compression. In most cases, the Bzip2 algorithm has a better data compression rate than other algorithms.
[0006] Although the Bzip2 data compression algorithm obtains the dependency between data through the BWT algorithm and the MTF algorithm, the "1-2 encoding" used by the algorithm only encodes and compresses data with the same MTF encoding and the same corresponding characters, and does not encode and compress data with the same MTF encoding but different characters. In addition, when the information entropy of a certain data is small, the number of bits required to represent the data is less than 1, but the Huffman encoding used by the algorithm still does not solve the problem of low compression rate caused by the minimum data encoding length of 1. Summary of the invention
[0007] The technical problem to be solved by the present invention is to overcome the following defects of the Bzip2 data compression algorithm with the best compression rate: first, the "1-2 encoding" used by the algorithm does not consider encoding data with the same MTF encoding but different characters; second, the Huffman encoding it adopts cannot solve the encoding problem when the minimum encoding length of the data is less than 1. In order to solve the above problems, the present invention provides a data compression method based on arithmetic coding, which can improve the data compression rate and reduce the data storage and data transmission overhead. The method can be used as a data compression method for data storage centers, databases, data transmission or other applications.
[0008] A first aspect of the present invention provides a data compression method based on arithmetic coding, the method comprising:
[0009] S101, sorting the data to be compressed into blocks, and encoding the sorted data into blocks;
[0010] S102, decomposing the encoded data into character code, bitmap code and length code;
[0011] S103, using arithmetic coding to compress the character code, the bitmap code and the length code respectively;
[0012] S104, splicing the data compressed in step S103 to obtain and output a compression result.
[0013] Furthermore, step S101 specifically includes the following operations:
[0014] S201, dividing the data to be compressed into a number of data blocks of fixed sizes;
[0015] S202, sorting each data block using the BWT algorithm;
[0016] S203: Use the MTF algorithm to encode the sorted data blocks.
[0017] Furthermore, step S102 specifically includes the following operations:
[0018] S301, using bitmap coding to mark characters with the same sorting position;
[0019] S302, calculating the length of the characters in each sorting position, and recording the character length using length coding;
[0020] S303, encode non-repeating characters, repeated characters and repetition length, and record them using character encoding.
[0021] Furthermore, step S103 specifically includes the following operations:
[0022] S401, using context-based adaptive binary arithmetic coding to compress the bitmap code;
[0023] S402: compress the character code and length code using a finite state entropy algorithm.
[0024] Furthermore, step S104 specifically includes the following operations:
[0025] S501, splicing the compressed bitmap code, data code and length code to obtain a spliced data block;
[0026] S502, sequentially splicing each splicing data block and outputting the splicing result.
[0027] A second aspect of the present invention provides a data compression device based on arithmetic coding, the device comprising:
[0028] A sorting and encoding module is used to sort the compressed data in blocks and encode the sorted data in blocks;
[0029] A decomposition module, used for decomposing the encoded data into character code, bitmap code and length code;
[0030] A compression module, used for compressing character codes, bitmap codes and length codes respectively using arithmetic coding;
[0031] The splicing module is used to splice the compressed data of the compression module to obtain and output the compression result.
[0032] Compared with the prior art, the present invention has the following beneficial effects:
[0033] (1) The present invention compresses and encodes data with the same MTF code and the same corresponding characters, and also compresses and encodes data with the same MTF code but different corresponding characters, thereby improving the data compression rate;
[0034] (2) The present invention decomposes data into three parts: bitmap, character and length. According to the data features of each part, the present invention uses the context-based adaptive binary arithmetic coding (CABAC) and finite state entropy (FSE) algorithms to compress the data, thereby improving the data compression rate.
[0035] (3) The present invention uses the CABAC algorithm based on arithmetic coding and the FSE algorithm to compress the encoded data, and can encode data with a code length less than 1, thereby overcoming the defect that the minimum code length of Huffman coding is 1, thereby improving the data compression rate. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only preferred embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0037] Figure 1 It is a schematic diagram of the overall framework of a data compression method based on arithmetic coding provided by an embodiment of the present invention.
[0038] Figure 2 It is a schematic diagram of the data decomposition implementation principle provided by an embodiment of the present invention.
[0039] Figure 3 It is a schematic diagram of the implementation principle of compressing character encoding and length encoding using the FSE algorithm provided in an embodiment of the present invention.
[0040] Figure 4 It is a schematic diagram of the principle of implementing compressed data splicing and output provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0041] The principles and features of the present invention are described below in conjunction with the accompanying drawings. The enumerated embodiments are only used to explain the present invention and are not used to limit the scope of the present invention.
[0042] Reference Figure 1 One embodiment of the present invention provides a data compression method based on arithmetic coding, the method comprising the following steps:
[0043] S101, sorting the data to be compressed into blocks, and encoding the sorted data into blocks.
[0044] S102, decomposing the encoded data into character code, bitmap code and length code.
[0045] S103: Use arithmetic coding to compress the character code, the bitmap code and the length code respectively.
[0046] S104, splicing the data compressed in step S103 to obtain and output a compression result.
[0047] As a possible implementation, step S101 specifically includes the following steps:
[0048] S201, dividing the data to be compressed into a number of data blocks of fixed sizes.
[0049] S202: Use the BWT (Burrows Wheeler Transform) algorithm to sort each data block.
[0050] S203: Use the MTF (Move-to-front) algorithm to encode the sorted data blocks.
[0051] As another possible implementation, step S102 specifically includes the following operations:
[0052] S301, using bitmap coding to mark characters with the same sorting position.
[0053] S302, calculating the length of the characters in each sorting position, and recording the character lengths using length coding.
[0054] S303, encode non-repeating characters, repeated characters and repetition length, and record them using character encoding.
[0055] As another possible implementation, step S103 specifically includes the following operations:
[0056] S401 . Compress the bitmap code using context-based adaptive binary arithmetic coding.
[0057] S402: compress the character code and length code using a finite state entropy algorithm.
[0058] As another possible implementation, step S104 specifically includes the following operations:
[0059] S501: Concatenate the compressed bitmap code, data code and length code to obtain a concatenated data block.
[0060] S502, sequentially splicing each splicing data block and outputting the splicing result.
[0061] The method provided in this embodiment is described below by a specific example:
[0062] For any given data D, it is divided into data blocks of length l B = {b i |i=1,2,…,n}≡D, where the data block length is L(b i )=l,i∈[1,n-1], the length of the last data block is L(b n In this embodiment, the default value of the data block length is l=900000.
[0063] For each data block Use the BWT algorithm to sort and generate the transformed data b′ i Then use the MTF algorithm to generate data code v i , the MTF algorithm maintains a one-dimensional character array C, which is b′ i The many-to-one mapping of characters in the , i.e. b′ i The same character in is mapped to the same character in array C. For b′ i Each character in Its code v ij Represents the character b′ ij The position index in the character array C, that is, b′ ij The number of characters with small index. The MTF algorithm processes b′ in turn. i Each character b′ in ij , when two adjacent characters are the same, their position index in the character array C remains unchanged; when two adjacent characters are different, character b′ in array C is replaced ij The previous characters are moved back one position in turn, and then the character b′ is replaced by ij Move to the starting position of array C, that is, the position index becomes 0.
[0064] In this embodiment, the bitmap code is used to mark whether the current MTF code is repeated and the characters are the same, the length code is used to record the length of the MTF code with the same MTF code and the same characters, and the character code is used to record the length of other non-repeated MTF codes and MTF codes with the same MTF code but different characters.
[0065] Reference Figure 2 , for each data block MTF code v i , let v ij Indicates b′ i The character b′ ij ∈b′ i MTF coding, n v Indicates b i ' and the current character b' ij The number of consecutive identical characters, n v ={n|v ij =v ij+k , b′ij = b′ ij+k , k∈[0,n]}, that is, with the encoding v ij The same number of consecutive MTF codes; let N v Is an array used to store n v ;n c Indicates b i ' and the current character b' ij The number of characters with the same position, n c ={n|v ij =v ij+k , b′ ij ≠b′ ij+k , k∈[0,n]}, which is the same as the encoding vij but with the same characters as b′ ij Different, this variable is not cleared when the character changes; n r Indicates b i ' and the current character b' ij The number of characters with the same position, n r ={n|v ij =v ij+k , b′ ij ≠b′ ij+k , k∈[0,n]}, and n c The difference is that this variable is cleared when the character changes.
[0066] First, initialize all variables to 0, starting with v i Each MTF code is processed in turn starting from the starting position. When the last MTF code is traversed, the data decomposition is completed.
[0067] For MTF encoding If the MTF code is the same as the previous MTF code, and the corresponding character is b′ ij Same as the previous character, i.e. v ij =v ij-1 , b′ ij = b′ ij-1 , then update n v =n v +1, and moving on to the next MTF encoding.
[0068] If the MTF code is the same as the previous MTF code, but the corresponding character b′ij is different from the previous character, that is, v ij =v ij-1 , b′ ij ≠b′ ij-1 , then according to n v The value of distinguishes two cases, if n v = 0, then update n r =n r+1, and then continue to process the next MTF code; if n v ≠0, first set n v Save to array N v , then record n in the bitmap code r 0 and 1, used to mark the previous MTF code v ij-1 To repeat the encoding, finally reset n v =0,n r = 0, and update n r =n r +1.
[0069] If the MTF code is different from the previous MTF code, and the corresponding character b′ ij It is also different from the previous character, that is, v ij ≠v ij-1 , b′ ij ≠b′ ij-1 , then record the current MTF code v in the character code ij , then determine n v =0,n c =0, If the condition is met, a 0 is recorded in the bitmap code. Otherwise, the same MTF code number n is determined. c Is it greater than 2? If it is greater than 2, three consecutive v characters are recorded in the character code. ij , used to mark the next character code to indicate the MTF code length, and then record n in the character code c -3; if the same MTF code number n c If it is less than or equal to 2, then record n directly in the character code c MTF codes.
[0070] Then record the array N in length encoding v The number of all consecutive MTF codes saved in the bitmap code is recorded in n r 0, note that there is no need to record the array N v The MTF code mark corresponding to the number of consecutive MTF codes in has been recorded in the previous cycle and does not need to be recorded again here.
[0071] Next, determine n v Is it 0? If it is 0, reset all variables to 0 directly; otherwise, it means that in the previous cycle, the number of continuous MTF codes has not been recorded in the array N. v In this case, a 1 needs to be recorded in the bitmap code and n in the length code. v Then reset all variables to 0.
[0072] Finally, it is determined whether the MTF coding of the current data block has been processed. If it has been processed, the data decomposition result is output; otherwise, the subsequent MTF coding is processed according to the above steps.
[0073] Reference Figure 3 , use the FSE algorithm to compress the character code and length code respectively, and use one byte to record the data compression status. When the return value is 0, it means that the data is not compressed, so record the original data and length, and then record 0B00 in the flag bit; when the return value is 1, it means that the data is a repeated character, so directly record the original data length and the character, and then record 0B01 in the flag bit; when the return value is other values, it means that the data has been compressed by the FSE algorithm, record the compressed data and length, and then record 0B10 in the flag bit.
[0074] Reference Figure 4 For each compressed data block, the data block mark byte, the compressed character code, the length code and the bitmap code are concatenated. Then it is determined whether the current data block is the last data block. If it is not the last data block, 0B0 is recorded in the data block mark bit, and then the subsequent data blocks are processed; otherwise, 0B1 is recorded in the data block mark bit, and then the concatenated data is output, which is the compression result of the original data.
[0075] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. A data compression method based on arithmetic coding, characterized in that: The method comprises: S101, sorting the data to be compressed into blocks, and encoding the sorted data into blocks; S102, decomposing the encoded data into character code, bitmap code and length code; S103, using arithmetic coding to compress the character code, the bitmap code and the length code respectively; S104, splicing the data compressed in step S103 to obtain and output a compression result.
2. The data compression method based on arithmetic coding according to claim 1, characterized in that: Step S101 specifically includes the following operations: S201, dividing the data to be compressed into a number of data blocks of fixed sizes; S202, sorting each data block using the BWT algorithm; S203: Use the MTF algorithm to encode the sorted data blocks.
3. The data compression method based on arithmetic coding according to claim 1, characterized in that: Step S102 specifically includes the following operations: S301, using bitmap coding to mark characters with the same sorting position; S302, calculating the length of the characters in each sorting position, and recording the character length using length coding; S303, encode non-repeating characters, repeated characters and repetition length, and record them using character encoding.
4. The data compression method based on arithmetic coding according to claim 1, characterized in that: Step S103 specifically includes the following operations: S401, using context-based adaptive binary arithmetic coding to compress the bitmap coding; S402: compress the character code and length code using a finite state entropy algorithm.
5. The data compression method based on arithmetic coding according to claim 1, characterized in that: Step S104 specifically includes the following operations: S501, splicing the compressed bitmap code, data code and length code to obtain a spliced data block; S502, sequentially splicing each splicing data block and outputting the splicing result.
6. A data compression device based on arithmetic coding, characterized in that: The device comprises: A sorting and encoding module is used to sort the compressed data in blocks and encode the sorted data in blocks; A decomposition module, used for decomposing the encoded data into character code, bitmap code and length code; A compression module, used for compressing character codes, bitmap codes and length codes respectively using arithmetic coding; The splicing module is used to splice the compressed data of the compression module to obtain and output the compression result.