A method for optimal iterative data compression based on huffman coding

Abstract

The present application belongs to the technical field of data processing, and in particular to an optimal iterative data compression method based on Huffman coding. The method first counts the frequency of each symbol in the original data and sorts them, and performs the 0th layer Huffman coding as the initial optimal result. Then, the method enters the iterative replacement compression loop, sorts the data of the current layer according to the frequency, replaces the 1st symbol with the 2nd symbol, and so on, generates a new data string and records the replacement mask string, performs Huffman coding on the new data string, and calculates the total compression length containing the mask string. The result of each layer is compared with the current optimal length, and if it is better, the optimal result is updated. The iteration continues until the symbol type is reduced to 1 or the total length of the mask exceeds the optimal length. Finally, the compressed data composed of the prefix coding, the mask string of each layer and the optimal layer compression coding string is output. The present application significantly improves the compression efficiency of Huffman coding by reducing the symbol type through iterative pairing replacement and combining with hierarchical optimal selection.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to an optimal iterative data compression method based on Huffman coding. Background Technology

[0002] With the rapid development of digital information technology, various terminal devices, IoT systems, vehicle networking platforms, and big data centers generate massive amounts of data every day. The transmission and storage of this data place extremely high demands on bandwidth and storage space. Data compression technology, as one of the core technologies for alleviating bandwidth pressure and reducing storage costs, has always been a research hotspot in computer science and information technology.

[0003] Currently, mainstream data compression algorithms can be divided into two main categories: lossy compression and lossless compression. Lossy compression, such as JPEG and MP3, is widely used in scenarios where high precision is not required, such as images and audio. Lossless compression, such as Huffman coding, LZ77, LZ78, and arithmetic coding, is widely used in scenarios where data integrity is strictly required, such as text, executable files, database records, and various protocol messages.

[0004] Huffman coding, a classic lossless compression algorithm, works by assigning different length codes to each symbol based on its frequency of occurrence, with higher-frequency symbols receiving shorter codes, thus minimizing the overall average length of the encoded data. However, traditional Huffman coding has an inherent limitation: when the original data contains a large number of symbol types with relatively uniform frequency distribution, the Huffman tree becomes quite deep, resulting in some symbols still having relatively long codes, thus limiting compression efficiency. For example, in a vehicle message containing 42 different byte types, even after Huffman coding, the compressed data still occupies a significant number of bits.

[0005] To further improve the compression performance of Huffman coding, academia and industry have proposed a series of improved schemes, such as adaptive Huffman coding, partial Huffman tree coding, and paradigmatic Huffman coding. These schemes mainly focus on the construction of the Huffman tree itself or the dynamic adjustment of the coding process, but they fail to optimize compression efficiency from the perspective of reducing the number of symbol types.

[0006] In addition, there are some compression techniques based on symbol substitution, such as the Re-Pair algorithm, which generates grammar rules by recursively finding and replacing frequently occurring adjacent symbol pairs, thereby achieving compression. However, the substitution rules of the Re-Pair algorithm are complex, requiring maintenance of the new symbols after substitution and their occurrence positions, and the decoding process is relatively cumbersome. Furthermore, its substitution granularity is adjacent position pairs rather than global symbol values, so its direct combination with Huffman coding is not natural.

[0007] Therefore, there is an urgent need to provide a lossless data compression method that can effectively reduce the number of symbol types, improve the compression efficiency of Huffman coding, and maintain the simplicity and reversibility of the encoding and decoding process, so as to better adapt to the needs of modern data processing scenarios. Summary of the Invention

[0008] Based on the technical problems existing in the prior art, this invention proposes an optimal iterative data compression method based on Huffman coding.

[0009] This invention proposes an optimal iterative data compression method based on Huffman coding, comprising the following steps:

[0010] Step S1: Obtain the raw data, count the frequency of each symbol in the raw data, and sort them from high to low frequency;

[0011] Step S2: Based on the sorted frequency distribution, construct a Huffman tree, perform layer 0 Huffman coding on the original data to obtain layer 0 compressed data, record the length of the layer 0 compressed data as the current optimal compression length, and use the layer 0 compressed data as the current optimal compression result;

[0012] Step S3: Enter the iterative replacement compression loop. Perform a symbol replacement preprocessing step for the current layer data, including: counting and sorting the frequency of each symbol in the current layer data; replacing the symbol in the 1st position with the symbol in the 2nd position, the symbol in the 3rd position with the symbol in the 4th position, and so on, according to the pairing replacement rule, to generate a new data string after replacement; at the same time, generating a replacement mask string for the current layer. The replacement mask string is a binary bit string, where each bit is used to indicate whether the symbol at the corresponding position in the current layer data has been replaced; then, Huffman encoding is performed again on the new data string to obtain the next layer compressed encoding string, and the total data length after the next layer compression is calculated. The total data length is equal to the sum of the length of the current layer replacement mask string and the length of the next layer compressed encoding string.

[0013] Step S4: Compare the total data length after compression of the next layer with the current optimal compression length. If the total data length after compression of the next layer is less than the current optimal compression length, update the current optimal compression length to the total data length after compression of the next layer, and update the current optimal compression result to a combination of prefix encoding, mask strings of each layer and compression encoding string of the next layer.

[0014] Step S5: Determine whether the iteration termination condition has been met. If not, use the new data string as the new current layer data and return to step S3; if it has been met, terminate the iteration and proceed to step S6.

[0015] Step S6: Output the final compressed data, which includes a prefix code for recording the optimal compression layer number, all mask strings from layer 1 to the optimal compression layer number, and the Huffman-coded compressed string corresponding to the optimal compression layer number.

[0016] Preferably, the pairing replacement rule in step S3 further includes: if the total number of different symbols in the current layer data is odd, the symbol in the last position after sorting remains unchanged and is not replaced.

[0017] Preferably, the iteration termination condition in step S5 includes: the number of different symbol types in the current layer data is reduced to 1, or the total bit length of all currently accumulated mask strings is greater than or equal to the current optimal compression length.

[0018] Preferably, in step S1, when multiple symbols appear with the same frequency, they are sorted according to their numerical values, with the smaller values ​​appearing first.

[0019] Preferably, the bit length of the prefix encoding is determined by a preset maximum number of iterations, which is the floor value of the logarithm to base 2, with the number of different symbol types in the original data as the argument.

[0020] Preferably, the symbol is a byte, word, double word, or binary block of any fixed length.

[0021] Preferably, the length of the replacement mask string is equal to the number of symbols in the current layer data, wherein a mask bit of 1 indicates that the symbol at the corresponding position has been replaced, and a mask bit of 0 indicates that the symbol at the corresponding position has not been replaced.

[0022] Preferably, the compressed data of the 0th layer Huffman coding in step S2 is used as the initial optimal compression result, the corresponding optimal compression layer number is recorded as 0, and the optimal compression result does not contain any mask string.

[0023] Preferably, the prefix code of the updated optimal compression result in step S4 is a binary representation of the optimal compression layer number.

[0024] Preferably, the decompression process of the method is the reverse process of the compression process: first, the prefix code is read to obtain the optimal number of compression layers; if the optimal number of compression layers is 0, then Huffman decoding is directly performed on the subsequent compressed code string to obtain the original data; if the optimal number of compression layers is greater than 0, then the mask strings of each layer are read first, and then Huffman decoding is performed on the compressed code string to obtain the data after the last layer is replaced. Then, according to the pairing and replacement rules of each layer mask string and the corresponding layer, the data is replaced layer by layer from the last layer in reverse until the original data is restored.

[0025] Compared with existing technologies, this invention provides an optimal iterative data compression method based on Huffman coding, which has the following advantages:

[0026] 1. By iteratively replacing symbol pairs, the number of different symbol types in the data is gradually reduced, making the data distribution more concentrated. This shortens the average code length of subsequent Huffman coding, thus achieving a higher compression ratio than traditional single-pass Huffman coding.

[0027] 2. This invention does not simply use the compression result of the last layer, but compares the compression effect after each layer of compression and dynamically selects the optimal layer for output. Since the mask string introduces additional overhead, the optimal compression layer often appears in the middle layer rather than the last layer. The hierarchical comparison mechanism of this invention can automatically capture the global optimal solution.

[0028] 3. The upper limit of the number of iterations is determined by the base-2 logarithm of the number of different symbol types in the original data. The algorithm complexity is controllable and it is suitable for real-time or near-real-time compression scenarios.

[0029] 4. The compressed data of this invention contains a complete mask string and a Huffman encoded stream. During decoding, the original data can be accurately recovered simply by replacing the mask string in reverse order, without losing any information.

[0030] 5. This invention is applicable to various types of data, including but not limited to vehicle communication messages, IoT sensor data, text files, program binary code, database records, etc. Attached Figure Description

[0031] Figure 1 This is a schematic diagram of the process of the present invention. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the invention.

[0033] Reference Figure 1 An optimal iterative data compression method based on Huffman coding includes the following steps:

[0034] Step S1: Obtain the raw data, group and statistically analyze the raw data, and count the frequency of each different symbol.

[0035] Symbols can be bytes, characters, or binary blocks of any fixed length. When calculating frequencies, each symbol and its frequency are recorded and sorted from highest to lowest frequency. If symbols have the same frequency, they are sorted by their numerical value, with smaller values ​​appearing first.

[0036] Step S2: Perform Huffman coding on the original data at level 0, calculate the length of the compressed data at level 0, and initialize the current best compression result as the compressed result at level 0.

[0037] Specifically, based on the frequency distribution obtained in step S1, a Huffman tree is constructed, and a corresponding Huffman code is assigned to each symbol. Then, each symbol in the original data is traversed, replaced with its corresponding Huffman code, and concatenated to form the 0th layer of compressed data. The bit length of the 0th layer of compressed data is recorded as the current optimal compression length, and the corresponding compressed data and the compression layer number of 0 are also recorded.

[0038] Step S3: Enter the iterative replacement compression loop and perform a symbol replacement preprocessing for the current layer data.

[0039] Let the current layer data be D_k (D_0 is the original data when k=0), and the number of different symbol types in the current layer be N_k. This step includes the following sub-steps:

[0040] Sub-step S3-1: Count the frequency of each symbol in the current layer data D_k, and sort them from highest to lowest frequency. If the frequencies are the same, sort them by the numerical value of the symbol, with the smaller value appearing first.

[0041] Sub-step S3-2: Generate a new data string D_k' according to the pairing replacement rule. The pairing replacement rule is as follows: replace the symbol in the 1st position (highest frequency) after sorting with the symbol in the 2nd position, the symbol in the 3rd position with the symbol in the 4th position, the symbol in the 5th position with the symbol in the 6th position, and so on. If the total number of different symbols N_k in the current layer of data is odd, the last symbol remains unchanged and is not replaced. This replacement is global, meaning that all positions in the current layer of data where the symbol to be replaced are replaced with the target symbol.

[0042] Sub-step S3-3: Generate the replacement mask string M_k for the current layer. The mask string M_k is a binary bit string with a length equal to the number of symbols in the current layer data D_k. For each symbol position in D_k, if the symbol at that position was replaced in sub-step S3-2 (i.e., the symbol belongs to the set of replaced symbols), the mask bit is set to 1; if the symbol at that position was not replaced (including the symbol used as the target symbol and the last symbol in the odd-numbered case), the mask bit is set to 0. The mask string M_k is used in the subsequent decompression process to recover the original data.

[0043] Sub-step S3-4: Recount the symbol frequencies of the replaced data string D_k', construct a new Huffman coding dictionary, and then perform Huffman coding on D_k' to obtain the (k+1)th layer compressed coding string C_{k+1}.

[0044] Sub-step S3-5: Calculate the total data length \(L_{k + 1}\) after compression at the \((k + 1)\)-th layer. \(L_{k+1}=\) the bit length of the mask string \(M_k+\) the bit length of the compressed encoding string \(C_{k + 1}\) at the \((k + 1)\)-th layer. Among them, the bit length of the mask string \(M_k\) is the number of symbols of the current layer data \(D_k\) (each symbol corresponds to a mask bit).

[0045] Step S4: Compare \(L_{k + 1}\) with the current optimal compression length \(L_{opt}\). If \(L_{k+1}<L_{opt}\), then update \(L_{opt}=L_{k + 1}\), and update the data corresponding to the current optimal compression result to: prefix encoding + mask strings of each layer + compressed encoding string at the \((k + 1)\)-th layer. At the same time, record the optimal compression layer number as \(k + 1\).

[0046] Step S5: Determine whether the iteration termination condition is reached. The iteration termination condition includes the following two situations. Terminate the iteration as long as any one of them is satisfied:

[0047] Termination condition A: The number of different symbol types \(N_k\) in the current layer data has been reduced to 1. At this time, continuing the replacement is meaningless, and the compression reaches the theoretical limit.

[0048] Termination condition B: The total bit length of all the accumulated mask strings has been greater than or equal to the current optimal compression length \(L_{opt}\). Because the mask string is part of the compressed data, if the mask string itself has exceeded the total length of the current optimal compressed data, then any subsequent compression cannot be better than the current optimal result.

[0049] If the termination condition is not reached, then take \(D_k'\) as the new current layer data, set \(k = k + 1\), and return to step S3 to continue the next iteration; if the termination condition is reached, then terminate the iteration and enter step S6.

[0050] Step S6: Output the final compressed data. The final compressed data consists of three parts: The first part is a fixed-length prefix encoding, which is used to record the optimal compression layer number; the second part is all the mask strings corresponding to the first layer to the optimal compression layer, concatenated in the layer order; the third part is the Huffman coding compressed string corresponding to the optimal compression layer number. Among them, the length of the prefix encoding is determined according to the preset maximum number of iteration layers. For example, when the maximum number of iteration layers is 8, the prefix encoding occupies 3 bit positions.

[0051] This embodiment takes the vehicle-mounted 808 protocol message data to be compressed as an example to illustrate the implementation process of the present invention in detail. The vehicle-mounted 808 protocol is a vehicle positioning communication protocol defined in the transportation industry standard JT / T808. Its messages usually contain a large number of flag bits, check bits, and repeatedly occurring fields, and have high compression potential.

[0052] Step 1: Obtain the original data and count the frequencies.

[0053] The original data is a hexadecimal string, the specific content of which is as follows:

[0054] 7E02000062332404239921A8A10008000000000003020A0A65067C9CFF0199000000002405071153580104000000000202000003020000140 4000000001504000000001604000000001702000018020000300116F119020101A8021301CC0801078ECF4890F316FFFFFFFFFFFFFFFFFE37E

[0055] This string is a standard automotive 808 message, containing several bytes. We iterate through this raw data and count the occurrences of each byte (i.e., each hexadecimal number). The results are as follows (the numbers in parentheses are the hexadecimal bytes and their corresponding decimal occurrences):

[0056] 00 (36 times), 02 (9 times), FF (9 times), 01 (7 times), 04 (5 times), 16 (3 times), 03 (2 times), 07 (2 times), 08 (2 times), 0A (2 times), 24 (2 times), 7E (2 times), 99 (2 times), A8 (2 times), 05 (1 time), 06 (1 time), 11 (1 time), 13 (1 time), 14 (1 time), 15 (1 time), 17 (1 time) 18 (1 time), 19 (1 time), 21 (1 time), 23 (1 time), 30 (1 time), 33 (1 time), 48 (1 time), 53 (1 time), 58 (1 time), 62 (1 time), 65 (1 time), 7C (1 time), 8E (1 time), 90 (1 time), 9C (1 time), A1 (1 time), CC (1 time), CF (1 time), E3 (1 time), F1 (1 time), F3 (1 time).

[0057] As can be seen, there are 42 different byte types in the original data. Sort them by frequency from highest to lowest, and those with the same frequency by byte value from smallest to largest. The resulting sequence of symbols is: 00, 02, FF, 01, 04, 16, 03, 07, 08, 0A, 24, 7E, 99, A8, 05, 06, 11, 13, 14, 15, 17, 18, 19, 21, 23, 30, 33, 48, 53, 58, 62, 65, 7C, 8E, 90, 9C, A1, CC, CF, E3, F1, F3.

[0058] Step 2: Level 0 Huffman coding (traditional Huffman compression).

[0059] Based on the frequency distribution described above, a Huffman tree is constructed, and a binary code is assigned to each symbol. The resulting Huffman codes are as follows (symbol → code):

[0060] 00→11

[0061] 01→0110

[0062] 02→000

[0063] 03→100110

[0064] 04→0100

[0065] 05→1010010

[0066] 06→1010011

[0067] 07→100111

[0068] 08→101010

[0069] 0A→101011

[0070] 11→0111100

[0071] 13→0111101

[0072] 14→1011000

[0073] 15→1011001

[0074] 16→01010

[0075] 17→0111110

[0076] 18→0111111

[0077] 19→1001000

[0078] 21→1001001

[0079] 23→1000110

[0080] 24→011100

[0081] 30→1000111

[0082] 33→1010000

[0083] 48→1010001

[0084] 53→0101100

[0085] 58→0101101

[0086] 62→1011010

[0087] 65→1011011

[0088] 7C→1001010

[0089] 7E→011101

[0090] 8E→1001011

[0091] 90→0101110

[0092] 99→101110

[0093] 9C→0101111

[0094] A1→1000000

[0095] A8→101111

[0096] CC→1000001

[0097] CF→1000010

[0098] E3→1000011

[0099] F1→1000100

[0100] F3→1000101

[0101] FF→001

[0102] Then, each byte in the original data is replaced with its corresponding Huffman code, and the resulting concatenation yields the 0th level compressed data (binary string). The total length of this compressed data is 481 bits. The current optimal compression length is recorded as 481 bits, the optimal compression level is 0, and the optimal compressed data is this 481-bit binary string.

[0103] Step 3: Enter the first layer of iterative replacement compression.

[0104] Step 3-1: Calculate and sort the symbol frequencies of the data in the current layer (layer 0). The data in layer 0 is the original data, and its frequency sorting was completed in step 1, as shown above.

[0105] Step 3-2: Generate a new data string D_1 according to the pairing and replacement rules. The pairing rules are as follows: the 1st symbol (00) is replaced with the 2nd symbol (02), the 3rd symbol (FF) is replaced with the 4th symbol (01), the 5th symbol (04) is replaced with the 6th symbol (16), the 7th symbol (03) is replaced with the 8th symbol (07), and so on. Since there are a total of 42 symbols (an even number), all symbols participate in the pairing, and there are no isolated symbols.

[0106] The specific replacement operation is as follows: Iterate through each byte in the original data. If the byte equals 00, replace it with 02; if it equals FF, replace it with 01; if it equals 04, replace it with 16; if it equals 03, replace it with 07; if it equals 08, replace it with 0A; if it equals 24, replace it with 7E; if it equals 99, replace it with A8; if it equals 05, replace it with 06; if it equals 11, replace it with 13; if it equals 14, replace it with 15; if it equals 17, replace it with 18; if it equals 19, replace it with 21; if it equals 23, replace it with... 30; if equal to 33, replace with 48; if equal to 53, replace with 58; if equal to 62, replace with 65; if equal to 7C, replace with 8E; if equal to 90, replace with 9C; if equal to A1, replace with CC; if equal to CF, replace with E3; if equal to F1, replace with F3; the remaining bytes (such as 02, 01, 16, 07, 0A, 7E, A8, 06, 13, 15, 18, 21, 30, 48, 58, 65, 8E, 9C, CC, E3, F3) remain unchanged because they are themselves the target symbols.

[0107] After the global replacement, a new hexadecimal string D_1 is obtained. Since many different source symbols are mapped to the same target symbol after the replacement, the number of different symbol types in D_1 is reduced to about half of the original (reduced to 21 in this embodiment). The specific content of D_1 is as follows (to save space, only a portion is listed here; the complete content can be derived based on the rules):

[0108] 7E02020265487E1630A821A8CC020A020202020207020A0A65068E9C0101A8020202027E06071358580116020202020202020207020202151 6020202021516020202021616020202021802020218020202300116F321020101A8021301CC0A01078EE3489CF3160101010101010101E37E

[0109] Step 3-3: Generate the first-level replacement mask string M_1. M_1 is a binary bit string with a length equal to the number of bytes in the original data (level 0 data). Iterate through each byte position of the original data. If the byte at that position belongs to the set of symbols to be replaced (i.e., all the source symbols that have undergone replacement), the corresponding mask bit is 1; if the byte at that position belongs to the target symbol (i.e., it has not undergone replacement or is the target of replacement), the mask bit is 0. The resulting mask string M_1 is 113 bits long, and its specific content is as follows: 001111111100111111111000001011111110110011111001111111101111110111110110011000110000000010001010011111111100

[0111] Steps 3-4: Re-encode D_1 using Huffman coding. Count the frequencies of the 21 symbols in D_1, construct a Huffman tree, and obtain the new Huffman-coded dictionary as follows (symbol → code):

[0112] 01→110

[0113] 02→0

[0114] 06→100110

[0115] 07→10001

[0116] 0A→10110

[0117] 13→100111

[0118] 15→100000

[0119] 16→1110

[0120] 18→100001

[0121] 21→101000

[0122] 30→101001

[0123] 48→1111110

[0124] 58→1111111

[0125] 65→1111100

[0126] 7E→10111

[0127] 8E→1111101

[0128] 9C→101010

[0129] A8→11110

[0130] CC→101011

[0131] E3→100100

[0132] F3→100101

[0133] Then, each symbol in D_1 is Huffman encoded to obtain the first layer of compressed encoded string C_1. The total length of C_1 is 381 bits.

[0134] Steps 3-5: Calculate the total data length L_1 after the first layer of compression. L_1 = length of mask string M_1 (113 bits) + length of C_1 (381 bits) = 494 bits.

[0135] Step 4: Compare and update the optimal result. The current optimal compression length is 481 bits (layer 0). L_1 = 494 bits is greater than 481 bits, so the optimal result is not updated, and the optimal compression layer number remains 0.

[0136] Step 5: Determine if the iteration termination condition has been met. The number of symbol types in the current layer (layer 0) is 42, which is much greater than 1, so termination condition A has not been met. The current total length of the cumulative mask is 113 bits, which is less than the current optimal compression length of 481 bits, so termination condition B has not been met. Therefore, continue to the next iteration.

[0137] Step 6: Proceed to the second layer of iterative replacement compression.

[0138] Using D_1 as the current layer data (its symbol type count is 21), repeat step 3.

[0139] Step 6-1: Count and sort the frequencies of each symbol in D_1 (the specific frequencies are obtained from the D_1 data and are omitted here).

[0140] Step 6-2: Generate a new data string D_2 according to the pairing replacement rules. Replace the first symbol after sorting with the second, the third with the fourth, and so on. Since 21 is an odd number, the last symbol remains unchanged. After the replacement, the number of symbol types in D_2 is further reduced to 11.

[0141] Step 6-3: Generate the second-layer mask string M_2. The length of M_2 is equal to the number of symbols in D_1 (the same as the number of bytes in the original data), totaling 113 bits. The specific content of M_2 is as follows: 0111000111011111111101110010001111100011101111111110111111111111111111111101001001110110010000100000000000

[0143] Step 6-4: Perform Huffman coding on D_2. Count the frequencies of the 11 symbols in D_2, construct a Huffman tree, and obtain the following coding dictionary (symbol → code):

[0144] 01→1

[0145] 06→0100

[0146] 07→001

[0147] 15→00011

[0148] 21→01010

[0149] 48→01011

[0150] 65→01100

[0151] 7E→0111

[0152] 9C→01101

[0153] E3→0000

[0154] F3→00010

[0155] Perform Huffman coding on D_2 to obtain the second-level compressed encoded string C_2, which has a length of 279 bits.

[0156] Step 6-5: Calculate the total data length L_2 after the second layer of compression. L_2 = length of M_1 (113) + length of M_2 (113) + length of C_2 (279) = 505 bits.

[0157] Step 4 (repeat): Compare and update the best result. The 505-bit result is greater than the current best result of 481 bits, so no update is performed.

[0158] Step 5 (Repeat): Determine the termination condition. The number of symbol types in the current layer is 11, which is greater than 1; the total length of the cumulative mask is 113 + 113 = 226 bits, which is less than 481 bits. Continue iterating.

[0159] Subsequent iterations: Theoretically, this embodiment can iterate up to log2(42)≈5.39, that is, a maximum of 6 iterations. As the number of iterations increases, the number of symbol types will gradually decrease to 1, but the total length of the mask string will continue to accumulate. In this embodiment, the total length after compression in subsequent layers such as the third and fourth layers will all exceed 481 bits, so the optimal result always remains at layer 0. This is precisely the significance of the hierarchical comparison mechanism of this invention—avoiding the blind pursuit of deep compression that leads to an increase in the total length.

[0160] Step 7: Construct the final compressed data.

[0161] Since the optimal number of compression layers is 0, the final compressed data consists of the following three parts:

[0162] Prefix encoding: Used to record the optimal number of compression layers. In this embodiment, the maximum number of iteration layers is set to 8, so 3 bits are used to represent the layer number, and the 0th layer is encoded as 000.

[0163] Mask string part: Since the optimal layer is 0, there is no corresponding mask string, so the length of the mask string part is 0.

[0164] Compressed encoded string: That is, Huffman encoded compressed data at level 0, totaling 481 bits.

[0165] Therefore, the final compressed data is: 000 + (empty) + Level 0 compressed data. During actual transmission or storage, the receiving end first reads the first 3 bits to obtain the level number 0, and then directly decodes the data according to the Level 0 Huffman coding dictionary to recover the original data.

[0166] Brief Description of the Decompression Method: The decompression method of this invention is the reverse process of the compression method described above. Taking the case where the optimal number of layers is 0 in this embodiment as an example, during decompression, the prefix code is first read to obtain the compression layer number 0, and then Huffman decoding is directly performed on the subsequent compressed code string to obtain the original data. If the optimal number of layers is greater than 0, the decompression process includes: first reading the mask strings of each layer, then performing Huffman decoding on the compressed code string to obtain the data after the last layer replacement, and then replacing layer by layer in reverse from the last layer according to the mask string until the original data is recovered. Specifically, for the mask string M_k of the k-th layer, if a certain bit is 1, it means that the position was replaced by the replaced symbol before compression, and it needs to be replaced back to the original symbol during decompression (the original symbol is determined by the pairing rule of this layer); if it is 0, it remains unchanged. By performing reverse operation layer by layer, the original data can be completely recovered, ensuring lossless operation.

[0167] The above embodiments use the vehicle-mounted 808 message as an example for illustration, but the present invention is not limited thereto. Those skilled in the art should understand that the present invention is also applicable to various scenarios such as text file compression, IoT device data reporting compression, database backup compression, and embedded system firmware compression. For example, for a piece of English text, each character can be treated as a symbol, and the frequency of letters, spaces, punctuation marks, etc., can be counted. Then, compression is performed according to the iterative replacement and Huffman coding method of the present invention. For different types of original data, the optimal number of iteration layers may differ, and the hierarchical comparison mechanism of the present invention can automatically adapt.

[0168] Furthermore, the symbols in this invention are not limited to 8-bit bytes; they can also be 16-bit words, 32-bit double words, or binary blocks of any fixed length, as long as they remain consistent during compression and decompression. The length of the mask string is equal to the number of symbols in the current layer of data, regardless of the bit width of each symbol.

[0169] The scope of protection of this invention is not limited to the specific embodiments described above. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. An optimal iterative data compression method based on Huffman coding, characterized in that, Includes the following steps: Step S1: Obtain the raw data, count the frequency of each symbol in the raw data, and sort them from high to low frequency; Step S2: Based on the sorted frequency distribution, construct a Huffman tree, perform layer 0 Huffman coding on the original data to obtain layer 0 compressed data, record the length of the layer 0 compressed data as the current optimal compression length, and use the layer 0 compressed data as the current optimal compression result; Step S3: Enter the iterative replacement compression loop. Perform a symbol replacement preprocessing step for the current layer data, including: counting and sorting the frequency of each symbol in the current layer data; replacing the symbol in the 1st position with the symbol in the 2nd position, the symbol in the 3rd position with the symbol in the 4th position, and so on, according to the pairing replacement rule, to generate a new data string after replacement; at the same time, generating a replacement mask string for the current layer. The replacement mask string is a binary bit string, where each bit is used to indicate whether the symbol at the corresponding position in the current layer data has been replaced; then, Huffman encoding is performed again on the new data string to obtain the next layer compressed encoding string, and the total data length after the next layer compression is calculated. The total data length is equal to the sum of the length of the current layer replacement mask string and the length of the next layer compressed encoding string. Step S4: Compare the total data length after compression of the next layer with the current optimal compression length. If the total data length after compression of the next layer is less than the current optimal compression length, update the current optimal compression length to the total data length after compression of the next layer, and update the current optimal compression result to a combination of prefix encoding, mask strings of each layer and compression encoding string of the next layer. Step S5: Determine whether the iteration termination condition has been met. If not, use the new data string as the new current layer data and return to step S3; if it has been met, terminate the iteration and proceed to step S6. Step S6: Output the final compressed data, which includes a prefix code for recording the optimal compression layer number, all mask strings from layer 1 to the optimal compression layer number, and the Huffman-coded compressed string corresponding to the optimal compression layer number.

2. The optimal iterative data compression method based on Huffman coding according to claim 1, characterized in that, The pairing and replacement rule in step S3 also includes: if the total number of different symbols in the current layer data is odd, the symbol in the last position after sorting remains unchanged and is not replaced.

3. The optimal iterative data compression method based on Huffman coding according to claim 1, characterized in that, The iteration termination conditions in step S5 include: the number of different symbol types in the current layer data is reduced to 1, or the total bit length of all currently accumulated mask strings is greater than or equal to the current optimal compression length.

4. The optimal iterative data compression method based on Huffman coding according to claim 1, characterized in that, In step S1, when multiple symbols appear with the same frequency, they are sorted according to their numerical values, with the smaller values ​​appearing first.

5. The optimal iterative data compression method based on Huffman coding according to claim 1, characterized in that, The bit length of the prefix encoding is determined by a preset maximum number of iterations, which is the floor value of the logarithm to base 2, with the number of different symbol types in the original data as the argument.

6. The optimal iterative data compression method based on Huffman coding according to claim 1, characterized in that, The symbol is a byte, word, double word, or binary block of any fixed length.

7. The optimal iterative data compression method based on Huffman coding according to claim 1, characterized in that, The length of the replacement mask string is equal to the number of symbols in the current layer data, where a mask bit of 1 indicates that the symbol at the corresponding position has been replaced, and a mask bit of 0 indicates that the symbol at the corresponding position has not been replaced.

8. The optimal iterative data compression method based on Huffman coding according to claim 1, characterized in that, In step S2, the compressed data of the 0th layer Huffman coding is used as the initial optimal compression result, and its corresponding optimal compression layer number is recorded as 0. The optimal compression result does not contain any mask string.

9. The optimal iterative data compression method based on Huffman coding according to claim 1, characterized in that, The updated optimal compression result in step S4 has a prefix code value that is a binary representation of the optimal compression layer number.

10. An optimal iterative data compression method based on Huffman coding according to any one of claims 1 to 9, characterized in that, The decompression process of the method is the reverse of the compression process: First, the prefix code is read to obtain the optimal number of compression layers. If the optimal number of compression layers is 0, the subsequent compressed code string is directly Huffman decoded to obtain the original data. If the optimal number of compression layers is greater than 0, the mask strings of each layer are read first, and then the compressed code string is Huffman decoded to obtain the data after the last layer is replaced. Then, according to the pairing and replacement rules of each layer mask string and the corresponding layer, the data is replaced layer by layer from the last layer in reverse until the original data is restored.

Images