Lossless data compression algorithm based on adaptive instantaneous entropy

Abstract

The application belongs to the technical field of data processing, and is a lossless data compression algorithm based on adaptive instantaneous entropy, which comprises a compression end and a decompression end, the compression end is composed of five parts of input data stream, lookup table, instantaneous entropy encoding, highest flag bit and sequencer, the decompression end is composed of five parts of deserializer, highest flag bit, instantaneous entropy decoding, lookup table and output data stream, and specifically comprises a compression process, a decompression process and a lookup table updating process; the compression process of the application calculates instantaneous entropy and adaptively encodes entropy through the occupancy rate of the lookup table, supports the processing of continuous fast data stream; the decompression end starts decoding operation from the first bit of the received compressed data stream, effectively improves the compression rate; and the algorithm has small amount of calculation and can be realized under limited resources.

Description

Technical Field

[0001] This invention belongs to the field of data processing technology, specifically, it is a lossless data compression algorithm based on adaptive instantaneous entropy. Background Technology

[0002] The advent of the information age generates hundreds of millions of data points daily, posing significant challenges to data storage, transmission, and processing. To save storage space and improve data transmission speed, data compression technology is essential. By removing excessive redundant information, data compression technology can reduce data storage costs and increase data transmission speed; therefore, it has significant research and practical value. Data compression algorithms are the prerequisite and foundation for data compression and can be divided into two categories: lossless compression and lossy compression. Lossy compression involves compressing some irrelevant data while allowing for some information loss; lossless compression, on the other hand, encodes the data to preserve the original information, allowing the original value to be recovered from the compressed data without loss of data quality.

[0003] Lossless data compression algorithms are mainly divided into two categories according to their compression models: statistical compression algorithms and dictionary-based compression algorithms. Arithmetic coding is a well-known compression algorithm based on statistical models. It uses Shannon information entropy to compress the entire input string by converting it into a long numerical value, where the number of bits required to represent this value is smaller than the original data itself. Huffman coding is also a compression algorithm based on statistical models. Its working principle is to sort the frequency of data characters and build a bottom-up binary tree. Then, it traverses from the root node down until all characters appear in the nodes of the binary tree. Next, it assigns a value of 0 to all left subtrees and a value of 1 to all right subtrees to determine the codeword of the original symbol. Data compression is completed by assigning the shortest codeword to the most common data pattern. Another lossless compression algorithm based on a dictionary model replaces complex original data strings with simple codes. For example, the LZW algorithm stores the first occurrence of each string in a string table and replaces the string with a unique number. During compression, only the code is stored, not the original string, for data compression. During decompression, the string table generated during compression is used and discarded after compression or decompression, without occupying extra space. Huffman coding is computationally simple and can reduce the average length of the encoded string, but it requires calculating the frequency of symbols appearing in the data stream, resulting in a long compression time. Furthermore, this coding technique is not ideal when the probability difference between data symbols is not significant. Arithmetic coding can theoretically achieve good compression ratios, but its practical implementation is very complex, requiring large storage space and power consumption. The compression effect of dictionary-based compression algorithms is closely related to the repetition of data occurrences and the size of the dictionary, thus the compression effect is not fixed. Deep learning-based compression algorithms use a large amount of computation and are specifically designed for image data, not data streams, and therefore cannot be applied to text data stream compression. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention proposes a lossless data compression algorithm based on adaptive instantaneous entropy. The algorithm receives a continuous, fast input data stream at the compression end, calculates the instantaneous entropy based on the lookup table occupancy rate, performs adaptive entropy encoding, and outputs a compressed data stream. At the decompression end, decoding begins upon receiving the first and last bits of the compressed data stream, and the same lookup table as the compression end is consistently used to ultimately recover the original data symbols.

[0005] The specific technical solution adopted in this invention is as follows:

[0006] A lossless data compression algorithm based on adaptive instantaneous entropy is proposed. This algorithm considers lookup table matching results and occupancy rate during operation, calculates instantaneous entropy based on the received data stream, and adaptively performs entropy encoding to complete the compression. The compression end consists of five parts: input data stream, lookup table, instantaneous entropy encoding, highest flag bit, and serializer; the decompression end consists of five parts: deserializer, highest flag bit, instantaneous entropy decoding, lookup table, and output data stream.

[0007] The lookup table has a total of K rows, with K rows being occupied. These rows consist of data symbols that appear frequently in the input data stream, and each data symbol in a row is N bits long. Each row in the lookup table has a corresponding M-bit row index, where M is calculated as follows:

[0008]

[0009] The decompression end uses the same lookup table as the compression end to decode the compressed data and output the original data symbols. Instantaneous entropy coding is a process of adaptively encoding the data stream by calculating the instantaneous entropy of the original data symbols. The calculation of instantaneous entropy depends on the number of rows k occupied in the lookup table. Here, E represents the instantaneous entropy, and represents the bit length of the entropy code.

[0010]

[0011] Entropy coding reduces the data stream to E bits by removing the high-order bits of the matching row identifier strings in the lookup table. The serializer reconstructs the resulting data stream into a D-bit compressed data stream. The decompression unit receives the compressed data stream, uses instantaneous entropy decoding to deserialize and extract the compressed data, and outputs the original symbols.

[0012] A lossless data compression algorithm based on adaptive instantaneous entropy specifically includes a compression process, a decompression process, and a lookup table update process.

[0013] Compression Process: When the input data stream arrives at the compression end, it receives the first N bits of the original data symbol. This symbol is then matched against a lookup table. If no matching symbol is found, the instantaneous entropy E = M, indicating that the symbol is not yet occupied by the lookup table. The symbol is output without compression, and is stored in the least significant bit of the lookup table, updating the lookup table. If a match is found, the symbol is occupied by the lookup table. The corresponding row identifier string is used as the data to be compressed. Instantaneous entropy is calculated to reduce the entropy encoding of the compressed data, generating the compressed data. A highest-order flag is defined to distinguish whether the data is compressed. If the data is not compressed, the flag is set to "0", and 0 + the original data is output; if the data is compressed, the flag is set to "1", and 1 + the compressed data is output. After all input data has undergone the above processing, the serializer receives all the data generated by the above operations and reconstructs it into a D-bit compressed data stream for output.

[0014] Decompression process: When the decompression end receives the first bit of the D-bit compressed data stream, the decoding operation begins. If the highest flag bit is 1, it indicates that compressed data has appeared. The deserializer extracts E bits of data from the D-bit compressed data stream based on the instantaneous entropy E obtained from the compression end, expands it to M bits, decompresses the line identifier string, and outputs the data in the corresponding entry as the original symbol. If the highest flag bit is 0, it indicates that the original data symbol has appeared. N bits are extracted from the compressed data stream and output as the original data symbol.

[0015] Lookup table update process: The lookup table works similarly to a stack instruction, with the highest and lowest bits representing the lookup table from bottom to top. Lookup table occupancy begins with filling the lowest bit. For example, after the lowest bit is occupied, storing a new data symbol requires first placing the data symbol from the lowest bit into the next lowest bit, and then storing the new data symbol in the lowest bit. On the compression side, when an input data symbol fails to match the lookup table, it needs to be stored in the lowest bit of the lookup table. When an input data symbol successfully matches the lookup table, the successfully matched data symbol needs to be moved to the lowest bit of the lookup table. If all rows of the lookup table are occupied after multiple updates, when storing a new data symbol, the highest bit data symbol needs to be popped and discarded first, and other bits are moved sequentially to logically higher bits. After the lowest bit is cleared, the new data symbol is stored in the lowest bit. There is also a special case in the lookup table update process: when a successfully matched data symbol is located in a higher bit of an occupied row, and the number of occupied rows is large, the successfully matched data symbol is moved to the lowest bit.

[0016] The beneficial effects of this invention are as follows: the compression process of this invention calculates instantaneous entropy by looking up the occupancy rate of the table and performs adaptive entropy encoding, which supports the processing of continuous and fast data streams; the decompression end starts the decoding operation from the first bit of the received compressed data stream, which effectively improves the compression rate; moreover, the algorithm has a small computational load and can be implemented with limited resources. Attached Figure Description

[0017] Figure 1 This is a structural diagram of the compression end and decompression end of the present invention.

[0018] Figure 2 This is a flowchart of the compression process of this invention.

[0019] Figure 3 This is an example diagram of the compression process in an embodiment of the present invention.

[0020] Figure 4 This is a flowchart of the decompression process of this invention.

[0021] Figure 5 This is an example diagram of the decompression process in an embodiment of the present invention.

[0022] Figure 6 This is a schematic diagram illustrating a situation where a match fails during the lookup table update process of this invention.

[0023] Figure 7 This is a schematic diagram illustrating a successful match during the lookup table update process of this invention.

[0024] Figure 8 This is a schematic diagram of the lookup table being full during the lookup table update process of this invention.

[0025] Figure 9 This is the lookup representation when C=2 in this embodiment of the invention.

[0026] Figure 10 This is a schematic diagram of moving row (K-1) of the lookup table to the first row in an embodiment of the present invention.

[0027] Figure 11 This is a schematic diagram of the lookup table moving the matching data two rows towards the least significant bit when d=2 in an embodiment of the present invention. Detailed Implementation

[0028] To enhance understanding of the present invention, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. These embodiments are only used to explain the invention and do not limit the scope of protection of the invention.

[0029] Example: A lossless data compression algorithm based on adaptive instantaneous entropy, with the structure diagrams of the compression and decompression ends as shown below. Figure 1As shown: The compression end consists of five parts: input data stream, lookup table, instantaneous entropy encoding, highest flag bit, and serializer; the decompression end consists of five parts: deserializer, highest flag bit, instantaneous entropy decoding, lookup table, and output data stream. The lookup table has a total of K rows, with k rows occupied. It consists of data symbols that appear frequently in the input data stream, and each data symbol in a row is N bits. Each row of the lookup table has a corresponding M-bit row index, where M is calculated as follows:

[0030]

[0031] The decompression end uses the same lookup table as the compression end to decode the compressed data and output the original data symbols. Instantaneous entropy coding is a process of adaptively encoding the data stream by calculating the instantaneous entropy of the original data symbols. The calculation of instantaneous entropy depends on the number of rows k occupied in the lookup table. Here, E represents the instantaneous entropy, and represents the bit length of the entropy code.

[0032]

[0033] Entropy coding reduces the data stream to E bits by removing the high-order bits of the matching row identifier strings in the lookup table. The serializer reconstructs the resulting data stream into a D-bit compressed data stream. The decompression unit receives the compressed data stream, uses instantaneous entropy decoding to deserialize and extract the compressed data, and outputs the original symbols.

[0034] Compression process

[0035] Compression flowchart as follows Figure 2 As shown. When the input data stream arrives at the compression end, the compression end receives the first N bits of the original data symbol from the input data stream; it matches this data symbol in a lookup table. If no matching data symbol is found, the instantaneous entropy E = M, indicating that the symbol has not yet been occupied by the lookup table. The symbol is output without compression, and it is stored in the least significant bit of the lookup table, updating the lookup table. If a match is successful, it indicates that the symbol has been occupied by the lookup table. The corresponding row identifier string is used as the data to be compressed, and the instantaneous entropy is calculated to reduce the entropy encoding of the compressed data, generating compressed data. Here, we define a highest flag bit to distinguish whether the data is compressed. If the data is not compressed, the flag bit is set to "0", and 0 + original data is output; if the data is compressed, the flag bit is set to "1", and 1 + compressed data is output. After all the input data has undergone the above processing, the serializer receives all the data generated by the above operations and reconstructs it into a D-bit compressed data stream for output. Figure 3 Here is an example of a compression process. Let each processed data symbol be 8 bits, the total number of rows in the lookup table K be 8, the number of rows occupied k be 4, and the serializer outputs a 3-bit compressed data stream.

[0036] decompression process

[0037] Decompression flowchart as follows Figure 4 As shown. When the decompression end receives the first bit of the D-bit compressed data stream, the decoding operation begins. If the highest flag bit is 1, it indicates that compressed data has appeared. The deserializer extracts E bits of data from the D-bit compressed data stream based on the instantaneous entropy E obtained from the compression end, extends it to M bits, decompresses the line identifier string, and outputs the data in the corresponding entry as the original symbol. If the highest flag bit is 0, it indicates that the original data symbol has appeared. N bits are extracted from the compressed data stream and output as the original data symbol. Figure 5 This is an example of the decompression process corresponding to the compression process described above.

[0038] Lookup table update process

[0039] The lookup table works similarly to a stack instruction, with the bits arranged from bottom to top representing the most significant bit to the least significant bit. The lookup table is filled starting with the least significant bit. For example, after the least significant bit is occupied, storing a new data symbol requires first placing the data symbol from the least significant bit into the next least significant bit, and then storing the new data symbol into the least significant bit. At the compression end, when an input data symbol fails to match the lookup table, the data symbol needs to be stored in the least significant bit of the lookup table. The process of updating the lookup table is as follows: Figure 5 As shown. When the input data symbol successfully matches the lookup table, the matching data symbol needs to be moved to the lowest position of the lookup table. The process of updating the lookup table is as follows. Figure 6 As shown.

[0040] If all rows of the lookup table are occupied after multiple updates, when storing new data symbols, the highest-order data symbol must first be popped and discarded, the other bits are moved sequentially to logically higher bits, and the new data symbol is stored in the lowest-order bits after the lowest-order bits are cleared. Figure 7 As shown. However, after multiple verifications, when the table is full, the instantaneous entropy at this time...

[0041]

[0042] This situation results in both compressed and uncompressed data being N+1 bits, i.e., the highest flag bit + N bits of compressed data / N bits of original data. This not only fails to compress but actually increases the number of sign bits. To reduce the number of bits compressed, a matching counter C is defined. When the input data sign matches the data sign in the table C times, the highest bit in the lookup table is popped and discarded. The value of C can be customized. For example, when C = 2, after a sign matches successfully for the second time, the highest bit in the table becomes invalid. Figure 8 As shown.

[0043] There's a special case in the lookup table update process: the matching data symbol is located in a higher position of the occupied table row. When the number of occupied rows is large, the matching data symbol is moved to the lowest position. For example, moving from row (K-1) of the lookup table to the first row would take a very long time. Figure 9 As shown. In this case, a finite number of rows is shifted. Taking d=2 as an example, after a successful match of the input data symbol, the matched data is shifted two rows towards the least significant bit, while other rules remain unchanged. Figure 10 As shown. If the logical number of rows to move for the matching data symbol is less than the custom limit, the operation will be performed according to the logical number of rows to move.

[0044] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A lossless data compression algorithm based on adaptive instantaneous entropy, characterized in that, The system includes a compression end and a decompression end. The compression end consists of five parts: an input data stream, a lookup table, an instantaneous entropy encoder, a highest-order flag, and a serializer. The decompression end consists of five parts: a deserializer, a highest-order flag, an instantaneous entropy decoder, a lookup table, and an output data stream. The total number of rows in the lookup table is set to K, the number of occupied rows is k, and the data symbol in each row is N bits. Each row in the lookup table has a corresponding M-bit row index, where M is calculated as follows: ， The decompression end uses the same lookup table as the compression end to decode the compressed data and output the original data symbols. Instantaneous entropy coding is a process of adaptively encoding the data stream by calculating the instantaneous entropy of the original data symbols. The calculation of the instantaneous entropy E depends on the number of rows k occupied in the lookup table. ， Entropy encoding reduces the data stream to E bits by deleting the high-order bits of the matching row identifier strings in the lookup table. The serializer reconstructs the resulting data stream into a D-bit compressed data stream. The decompression end receives the compressed data stream and uses instantaneous entropy decoding to deserialize and extract the compressed data, outputting the original symbols. The algorithm specifically includes a compression process, a decompression process, and a lookup table update process. The compression process specifically involves the following steps: When the input data stream arrives at the compression end, the compression end receives the first N bits of the original data symbols from the input data stream and matches these symbols in the lookup table. If no matching data symbol is found, the instantaneous entropy E=M, indicating that the symbol has not yet been occupied by the lookup table. The symbol is output without compression, and the data symbol is stored in the lowest bit of the lookup table, updating the lookup table. If a match is successful, it indicates that the symbol has been occupied by the lookup table. The corresponding row identifier string is used as the data to be compressed, and the instantaneous entropy is calculated to reduce the entropy encoding of the compressed data, generating the compressed data.

2. The lossless data compression algorithm based on adaptive instantaneous entropy according to claim 1, characterized in that, In the compression process, a highest flag bit is set to distinguish whether the data is compressed. If the data is not compressed, the flag bit is set to "0" and the output is 0 + original data; if the data is compressed, the flag bit is set to "1" and the output is 1 + compressed data. After all the input data has been compressed, the serializer receives all the data generated by the data compression operation and reconstructs it into a D-bit compressed data stream for output.

3. The lossless data compression algorithm based on adaptive instantaneous entropy according to claim 2, characterized in that, The specific steps of the decompression process are as follows: When the decompression end receives the first bit of the D-bit compressed data stream, the decoding operation begins. If the highest flag bit is 1, it indicates that the data is compressed. The deserializer will extract E bits of data from the D-bit compressed data stream based on the instantaneous entropy E obtained from the compression end and extend it to M bits. After decompressing the line identifier string, the data in the corresponding entry will be output as the original symbol. If the highest flag bit is 0, it indicates that the original data symbol is present. N bits are extracted from the compressed data stream and output as the original data symbol.

4. The lossless data compression algorithm based on adaptive instantaneous entropy according to claim 3, characterized in that, The specific steps of the lookup table update process are as follows: On the compression end, when the input data symbol fails to match the lookup table, the data symbol needs to be stored in the lowest bit of the lookup table to update the lookup table; when the input data symbol successfully matches the lookup table, the successfully matched data symbol needs to be moved to the lowest bit of the lookup table to update the lookup table.

5. The lossless data compression algorithm based on adaptive instantaneous entropy according to claim 4, characterized in that, During the lookup table update process, when the table is full, the instantaneous entropy is... ， Define a match counter C. When the input data symbol matches the data symbol in the table C times, the highest bit in the lookup table is popped and discarded. The value of C can be customized.

Images