A repeatable prefix compression encoding method based on data features

Abstract

The application discloses a repeatable prefix compression coding method based on data characteristics, and the method is characterized in that the data characteristics of different types of files are adaptively adjusted through adaptive symbol bit width, and the unique decodable non-prefix relaxation is combined to further mine the compression space on the basis of ensuring coding stability and realizability, and a double-flow structured container coding scheme is designed for high-repetition continuous data, wherein ENCR is an optimal implementation form, the compression rate and engineering realizability are considered, and the overall compression effect can be preferably obtained in the applicable data distribution scene. Through adaptive adjustment of the symbol bit width for different types of files, the application can adapt to various scenes such as text, executable files and high-repetition binary, and fully mines the compression space under different data characteristics. Taking the Huffman initial tree as the starting point, the stability and engineering realizability of the Huffman initial codebook construction are retained.

Description

Technical Field

[0001] This invention relates to the field of data compression coding technology, and specifically to a repeatable prefix compression coding method based on data features. Background Technology

[0002] Data compression has always been a fundamental technology in the field of computer science. In recent years, more efficient compression technologies have been proposed for specific problems and application scenarios, such as matrix reconstruction optimization of the process—Huffman coding, cooperative particle swarm optimization of Huffman coding, and lossless coding conversion of big data based on time-series data compression. Each of these technologies has its own characteristics and limitations in its respective field.

[0003] While existing technologies have made some progress in data compression, several significant shortcomings remain. For fixed-structure encoding compression schemes, code length allocation rules cannot be dynamically adjusted to adapt to changes in the characteristics of the data to be compressed, making it difficult to adapt to data with different distribution characteristics and maintain a high compression ratio across various scenarios. Furthermore, compression schemes for multi-type data mostly rely on fixed training datasets for model building. When faced with new data whose distribution differs significantly from the training samples, their generalization ability is poor, and they are prone to catastrophic forgetting, failing to continuously adapt to the compression needs of new data. Additionally, for compression requirements in specific scenarios with real-time demands, such as large model key-value caching, existing methods struggle to achieve ideal compression results while maintaining inference performance and task accuracy. Dictionary-based sensor data compression schemes, on the other hand, see the compression dictionary continuously expand during the compression process, ultimately increasing the number of bits in the encoding and reducing overall compression efficiency.

[0004] Therefore, this invention provides a repeatable prefix compression coding method based on data features, which can dynamically adjust the coding strategy according to the features of the data to be compressed, thereby further improving compression efficiency and adaptability while ensuring data integrity. Summary of the Invention

[0005] The purpose of this invention is to provide a repeatable prefix compression coding method based on data features, the method specifically including the following steps: S1. Input Profile Layer: Read the original byte sequence of the file to be compressed, and statistically form the profile information of the file to be compressed. The profile information includes at least the total number of bytes in the file, the number of different byte values, the proportion of printable characters, the proportion of word-like characters, information entropy, the proportion of adjacent identical bytes, the cumulative proportion of the first 8 high-frequency adjacent byte pairs, and whether the file header contains the MZ identifier or the ELF identifier. Determine the main symbol width based on the profile information, and when the super-repeating binary judgment condition is met, further expand the current 8-bit symbol width to the second symbol width. S2, Basic Codeword Optimization Layer: Symbol width determined by the input profile layer. The bit stream is cut into fixed-length symbols, the frequency of each symbol is counted, a Huffman tree is built using the frequency, an initial prefix code is assigned to all leaves, and then a uniquely decodable non-prefix relaxation is performed to move high-frequency symbols from deeper leaf positions to shallower internal node prefixes, thereby shortening the code length and obtaining the final codeword. The specific steps of the uniquely decodable non-prefix relaxation in S2 are as follows: S21. Collect internal nodes and leaf nodes: From the current... The tree structure collects two types of nodes, including leaf nodes and internal nodes whose corresponding actual symbol paths are not empty and have at least one child node. The root node path is empty and is not used as a source of new codewords that can be directly lifted. S22. Construct "Internal Path - Target Leaf Symbol" Uplift Candidates: For each available internal node path, select multiple target leaf symbols that can be shortened and pair them to form uplift candidates. The candidate construction rules include: the original codeword length of the target leaf symbol is greater than the internal node path length; the new codeword corresponding to the internal node path does not repeat the current codeword set; after applying the candidate, The theoretical benefit of a candidate that meets the following rules is calculated: (The numbers must not exceed a preset threshold, each internal node must retain only a preset number of preferred candidates, and the global candidate count must be controlled within a preset upper limit.) in, Indicates the frequency of the target leaf symbol. Indicates the original codeword length of the target leaf symbol. The candidate new codeword length is represented by the target leaf symbol frequency, the current codeword length, and the payoff value. The candidates are sorted to form a candidate frontier set. S23, Execute multi-level Search: Construct a candidate frontier set based on the theoretical returns of candidate uplifts, and perform multi-level... To avoid an excessively large search space, a theoretically optimistic upper bound on the candidate path is introduced. S24. Duplicate codeword check: For each candidate to be proposed, a duplicate codeword check is first performed. If the candidate new codeword already exists in the current codeword set, it means that applying the candidate will directly result in a duplicate codeword. There is no need to continue the subsequent legality judgment, and the candidate is directly rejected. S25, Constraint Incremental Decision: Perform the push-up of each candidate path. Constrained incremental determination, maintaining the current codebook and:

[0006] in, Indicates the first Since candidate lifting only changes the codeword length of the target leaf symbol, it is not necessary to recalculate the entire codeword when judging a single candidate. And, using an incremental update method:

[0007] in, The original codeword length of the target leaf symbol, For the candidate new codeword length, if the updated and If the value is greater than the preset threshold, it means that the candidate does not even meet the necessary length constraint of a uniquely decodable codebook, and the candidate is directly rejected. S26. Short string ambiguity witness pre-pruning: Before performing a complete, unique, and decodable determination, first prune based on the code. Constructing a fast pre-pruning link, including code Construction: Insert the current codeword set bit by bit into the binary branch. In the structure, the corresponding symbol set is recorded at the terminal node. Candidate new codeword separability check: If the candidate new codeword itself can be obtained by segmenting other codewords, then applying the candidate will inevitably introduce ambiguity, therefore the candidate is directly rejected. Short string ambiguity witness check: At least the following short strings are constructed for multi-segmentation detection: " + " + "as well as" + If any of the above short strings can be segmented in two or more ways, it means that the candidate will introduce ambiguity at the short string level, and it will be rejected directly. S27. Internal path followed by symbol check: For each internal node, pre-record all existing codewords with the internal node path as a prefix and their corresponding symbols. If a codeword can still be divided by other codewords after removing the internal node path prefix, the corresponding internal path lifting will cause ambiguity and will be directly rejected. S28 Type-unique decodable determination: For those that pass the duplicate codeword check, Constrained incremental decision-making and short string ambiguous witness pre-pruning candidates are then executed. Type unique decodable determination; S29. Candidate Path Submission and Codebook Update: If the total number of bytes in the compressed file corresponding to a candidate path is less than the current codebook, then the candidate path is submitted. Candidate path submission includes at least replacing the original codeword of the target leaf symbol with the new codeword corresponding to the internal node path; updating the codeword count table; updating the symbol attribution of internal nodes and leaf nodes in the tree structure; and updating... And; save the current best codebook, and stop further upward search if no legal and better candidate path is found in this round of search; S3, Container Output Layer: The original symbol stream is encoded using a dual-stream structured container. First, the original symbol stream is converted into a run sequence. Then, the run sequence is split into a symbol stream and a length stream. The symbol stream and the length stream are entropy encoded separately. Finally, they are encapsulated into a structured container for encoding.

[0008] Furthermore, S1 employs a bit-width selection algorithm to determine the actual symbol width. The specific steps are as follows: S11. Read the original byte sequence of the file to be compressed and obtain the profile information. The profile information includes at least: the total number of bytes in the file, the number of different byte values, the proportion of printable characters, the proportion of word-like characters, information entropy, the proportion of adjacent identical bytes, the cumulative proportion of the first 8 high-frequency adjacent byte pairs, and whether the file header contains the MZ identifier or the ELF identifier. S12. When the user requests a width of less than 8 bits, determine whether the file to be compressed meets the conditions for a text file or an executable file based on the profile information; when the total number of bytes in the file to be compressed is greater than or equal to 1024 bytes, the proportion of printable characters is greater than or equal to 98%, the proportion of word-like characters is greater than or equal to 90%, and the number of different byte values ​​is less than or equal to 96, increase the main symbol width to 8 bits; when the total number of bytes in the file to be compressed is greater than or equal to 4096 bytes, the proportion of printable characters is greater than or equal to 95%, and the number of different byte values ​​is less than or equal to 128, increase the main symbol width to 8 bits; when the total number of bytes in the file to be compressed is greater than or equal to 65536 bytes and the file header contains an MZ identifier or an ELF identifier, increase the main symbol width to 8 bits. S13. When the user-requested width is greater than or equal to 8 bits, the user-requested width is directly used as the current symbol width; when the user-requested width is less than 8 bits and the file to be compressed does not meet the conditions of text file and executable file, the user-requested width is kept as the current symbol width. S14. When the current symbol width is 8 bits, and the total number of bytes in the file to be compressed is greater than or equal to 65536 bytes, the proportion of printable characters is less than or equal to 10%, the proportion of word-like characters is less than or equal to 5%, the number of different byte values ​​is less than or equal to 192, the information entropy is less than or equal to 2.25, the proportion of adjacent identical bytes is greater than or equal to 65%, and the cumulative proportion of the first 8 high-frequency adjacent byte pairs is greater than or equal to 60%, the file to be compressed is determined to meet the super-repeating binary judgment condition, and the current symbol width is further expanded to the second symbol width; among them, when the number of different byte values ​​is greater than or equal to 96, the information entropy is greater than or equal to 1.0, the proportion of printable characters is less than or equal to 3%, the proportion of word-like characters is less than or equal to 2%, and the proportion of adjacent identical bytes is less than or equal to 87%, 12 bits are selected as the second symbol width; otherwise, 16 bits are selected as the second symbol width.

[0009] Furthermore, the multi-layer beam search in S23 includes selecting candidates with higher theoretical returns from the current round of candidate frontier set to enter the first-level beam, re-collecting internal nodes and leaf nodes after applying the first-level candidates to construct the second-level candidate frontier set, continuing to expand the second-level candidates to form multiple candidate paths, continuing to expand the third-level and fourth-level candidate paths within the preset budget, and performing legality judgment and total byte evaluation of compressed files on different candidate paths respectively.

[0010] Furthermore, the upper bound of the theoretical optimistic return in S23 includes estimating the maximum theoretical return that can be obtained by the remaining single-step upward movement, converting the theoretical return into an optimistic estimate of the total bytes of the compressed file, and terminating the subsequent expansion of the candidate path if the optimistic estimate is not better than the current best result.

[0011] Furthermore, in S28 The process for determining a uniquely decodable codeword includes: collecting all non-empty codewords; checking for duplicate codewords; constructing a suffix state set; iteratively expanding the suffix state set; if an empty suffix conflict occurs, the candidate is determined not to be uniquely decodable; if the state expansion terminates and no conflict occurs, the candidate is determined to be uniquely decodable, and the determination result is cached to reuse the determination result under duplicate candidate states.

[0012] Furthermore, the specific steps of S3 are as follows: S31. Constructing a run sequence: Merge consecutive identical symbols into a single run and represent it as follows: binary; S32. Calculate the two frequency tables: calculate the frequency table of the run symbol and the frequency table of the run length respectively; S33. Establish two sets of Huffman codes respectively: Construct two sets of Huffman codes according to the run symbol frequency table and the run length frequency table respectively; S34. Encode two bit streams: Traverse each run, write the run symbol into the symbol stream, and write the run length into the length stream to obtain independent symbol bit streams and length bit streams; S35. Construct two sets of sparse code tables: the first symbol value is recorded directly, and subsequent symbol values ​​are represented using delta and passed through... Encoding, code length adopts Sequence representation; S36. Encapsulate as ENCR encoding: First encapsulate the container header, then write the symbol table index part, symbol code length part, length table index part, length code length part, symbol bit stream, and length bit stream in sequence.

[0013] The present invention has the following advantages: By adaptively adjusting the symbol bit width for different types of files, the present invention can adapt to various scenarios such as text, executable files, and highly repetitive binary files, fully explore the compression space under different data characteristics, and retain the stability and engineering feasibility of the Huffman initial codebook construction by starting with the Huffman initial tree.

[0014] This invention provides a new search space for the redistribution of codewords that are the only decodable codewords outside of traditional prefix codes by using internal node paths as candidate sources of new codewords.

[0015] This invention checks for duplicate codewords, Constrained incremental decision-making, short string ambiguous witness pre-pruning and The combination of uniquely decodable types reduces the risk of ambiguity caused by internal path lifting.

[0016] This invention improves the engineering feasibility of uplift search in a large candidate space by combining multi-layer beam search with optimistic reward upper bound pruning.

[0017] This invention avoids the problem of optimizing only local encoded segments and thus resulting in an overall suboptimal file by submitting candidate paths based on the total number of bytes in the compressed file.

[0018] This invention proposes a dual-stream structured container encoding scheme, in which ENCR is a preferred implementation. The run sequence is split into a symbol stream and a length stream, which are encoded and compressed separately. Combined with a sparse code table, the storage volume of the code table itself is further compressed, thereby improving the compression efficiency of highly repetitive continuous data. Attached Figure Description

[0019] Figure 1 This is a flowchart of the bit width selection algorithm of the present invention; Figure 2 This is the flowchart of the unique decodable non-prefix relaxation algorithm of the present invention; Figure 3 This is a flowchart of the ENCR encoding process of the present invention; Figure 4 This is a flowchart of the present invention. Detailed Implementation

[0020] This invention provides a repeatable prefix compression coding method based on data features.

[0021] In Examples 1 and 2, the profile information in S1 is obtained in the following way: the encoding end reads all the original byte sequences of the file to be compressed, and the total number of bytes in the file is denoted as N; a 256-dimensional byte frequency statistics table is established to count the number of occurrences of each byte value; at the same time, an adjacent byte pair frequency statistics table is established to count the number of occurrences of each adjacent byte pair.

[0022] The percentage of printable characters was determined by statistics. The number of printable characters, newline characters, carriage return characters, and tab characters, divided by the total number of bytes in the file; the percentage of word-like characters is obtained by counting the number of letters, numbers, spaces, newline characters, carriage return characters, and tab characters, divided by the total number of bytes in the file; information entropy is based on the frequency distribution of each byte value according to... Entropy formula calculation; the proportion of adjacent identical bytes is obtained by counting the number of times that two adjacent bytes have the same value and dividing by the total number of adjacent byte pairs N-1; the cumulative proportion of the top 8 high-frequency adjacent byte pairs is obtained by counting the occurrences of all adjacent byte pairs, selecting the 8 most frequent adjacent byte pairs, summing their occurrences and dividing by the total number of adjacent byte pairs N-1.

[0023] Furthermore, the file header identifier is obtained by detecting the sequence of the first byte of the file: if the first two bytes of the file are the characters "M" and "Z" respectively, the file header is determined to contain the MZ identifier; if the first four bytes of the file are... " " " When the value is "", it is determined that the file header contains the ELF identifier.

[0024] When the user requests a width of less than 8 bits, if the file to be compressed meets the conditions for a text file or an executable file, the main symbol width is increased to 8 bits. When the current symbol width is already 8 bits and the file to be compressed meets the condition for super-repeating binary, the second symbol width is selected according to the profile information, and the current symbol width is further expanded from 8 bits to the second symbol width, where the second symbol width is 12 bits or 16 bits. In the following embodiments, the dual-stream structured container is implemented in the form of ENCR.

[0025] Example 1 uses an English text file as the object to be compressed. The encoding end first reads the original byte sequence of the file to be compressed and performs portrait information statistics according to S1 to obtain the total number of bytes in the file, the number of different byte values, the proportion of printable characters, the proportion of word-like characters, information entropy, the proportion of adjacent identical bytes, the cumulative proportion of the first 8 high-frequency adjacent byte pairs, and the file header identifier.

[0026] The text file to be compressed is 152089 bytes in size, with 74 different byte values, 99.9993% printable characters, 94.54% word-like characters, 4.5677 information entropy, 4.7098% adjacent identical bytes, and 18.4544% cumulatively from the top 8 high-frequency adjacent byte pairs. The file header does not contain the MZ or ELF identifiers.

[0027] S1. When the user requests a width of less than 8 bits, since the file meets the criteria of being a text file ("total file size greater than or equal to 1024 bytes, printable character percentage greater than or equal to 98%, word-like character percentage greater than or equal to 90%, number of different byte values ​​less than or equal to 96"), the system increases the main symbol width to 8 bits.

[0028] S2. Perform fixed-length symbolization on the input bitstream according to the determined 8-bit symbol width, count the frequency of each symbol, construct the Huffman initial tree, and generate the initial prefix code; then perform uniquely decodable non-prefix relaxation, including candidate construction, theoretical payoff sorting, multi-level beam search, and duplicate codeword checking. Constraint incremental decision-making, short string ambiguous witness pre-pruning, internal path follow-up symbol checking, and The type is uniquely decodable, thus obtaining an optimized codebook.

[0029] S3. After obtaining the optimized codebook, the original symbol stream is converted into a run sequence. The run sequence is split into a symbol stream and a length stream. Huffman codes are established and encoded separately. Finally, the output is encapsulated according to the ENCR dual-stream container format. The decoding end recovers the original file based on the corresponding code table information and container structure. The recovery result is consistent with the input file byte by byte.

[0030] Example 2 uses a highly repetitive binary file as the object to be compressed. The encoding end first reads the original byte sequence of the file to be compressed and performs portrait information statistics according to S1 to obtain the total number of bytes in the file, the number of different byte values, the proportion of printable characters, the proportion of word-like characters, information entropy, the proportion of adjacent identical bytes, the cumulative proportion of the first 8 high-frequency adjacent byte pairs, and the file header identifier.

[0031] The binary file to be compressed is 513216 bytes in size, with 159 different byte values. The percentage of printable characters is 1.7501%, the percentage of word-like characters is 0.3591%, the information entropy is 1.2102, the percentage of adjacent identical bytes is 85.2037%, and the cumulative percentage of the top 8 high-frequency adjacent byte pairs is 86.9593%. The file header does not contain the MZ identifier or the ELF identifier.

[0032] S1. When the current symbol width is 8 bits, since the file meets the super-repeating binary judgment conditions of "total number of bytes of file greater than or equal to 65536 bytes, printable character ratio less than or equal to 10%, word-like character ratio less than or equal to 5%, number of different byte values ​​less than or equal to 192, information entropy less than or equal to 2.25, adjacent identical byte ratio greater than or equal to 65%, and the cumulative ratio of the first 8 high-frequency adjacent byte pairs greater than or equal to 60%", the system further attempts to expand from 8 bits to the second symbol width.

[0033] S2. Since the file simultaneously meets the conditions of "the number of different byte values ​​is greater than or equal to 96, the information entropy is greater than or equal to 1.0, the proportion of printable characters is less than or equal to 3%, the proportion of word-like characters is less than or equal to 2%, and the proportion of adjacent identical bytes is less than or equal to 87%", therefore, in this embodiment, 12 bits are selected as the second symbol width instead of 16 bits.

[0034] S3. Perform fixed-length symbolization on the input bitstream according to the 12-bit symbol width, count the frequency of each symbol, construct the Huffman initial tree, and generate the initial prefix code; then perform uniquely decodable non-prefix relaxation to obtain the optimized codebook. After the codebook optimization is completed, construct the run sequence and encode the symbol stream and length stream respectively. Finally, encapsulate it into an ENCR dual-stream container format output compressed file. The decoding end recovers the original file according to the corresponding container structure and code table information. The recovery result is consistent with the input file byte by byte.

[0035] In this embodiment, not only can the symbol width be increased to 8 bits according to the characteristics of text files, but also the current 8-bit symbol width can be further expanded to a second symbol width when super-repeating binary features are detected, thereby improving the compression adaptation capability for highly repetitive continuous data.

[0036] Although specific embodiments of the present invention have been described in detail with reference to the accompanying drawings, this should not be construed as limiting the scope of protection of the present invention. Various modifications and variations that can be made by those skilled in the art without inventive effort within the scope described in the claims are still within the scope of protection of the present invention.

Claims

1. A repeatable prefix compression coding method based on data features, characterized in that: The method specifically includes the following steps: S1. Input Profile Layer: Read the original byte sequence of the file to be compressed, and statistically form the profile information of the file to be compressed. The profile information includes at least the total number of bytes in the file, the number of different byte values, the proportion of printable characters, the proportion of word-like characters, information entropy, the proportion of adjacent identical bytes, the cumulative proportion of the first 8 high-frequency adjacent byte pairs, and whether the file header contains the MZ identifier or the ELF identifier. Determine the main symbol width based on the profile information, and when the super-repeating binary judgment condition is met, further expand the current 8-bit symbol width to the second symbol width. S2, Basic Codeword Optimization Layer: Symbol width determined by the input profile layer. The bit stream is cut into fixed-length symbols, the frequency of each symbol is counted, a Huffman tree is built using the frequency, an initial prefix code is assigned to all leaves, and then a uniquely decodable non-prefix relaxation is performed to move high-frequency symbols from deeper leaf positions to shallower internal node prefixes, thereby shortening the code length and obtaining the final codeword. The specific steps of the uniquely decodable non-prefix relaxation in S2 are as follows: S21. Collect internal nodes and leaf nodes: From the current... The tree structure collects two types of nodes, including leaf nodes and internal nodes whose corresponding actual symbol paths are not empty and have at least one child node. The root node path is empty and is not used as a source of new codewords that can be directly lifted. S22. Construct "Internal Path - Target Leaf Symbol" Uplift Candidates: For each available internal node path, select multiple target leaf symbols that can be shortened and pair them to form uplift candidates. The candidate construction rules include: the original codeword length of the target leaf symbol is greater than the internal node path length; the new codeword corresponding to the internal node path does not repeat the current codeword set; after applying the candidate, The theoretical benefit of a candidate that meets the following rules is calculated: (The numbers must not exceed a preset threshold, each internal node must retain only a preset number of preferred candidates, and the global candidate count must be controlled within a preset upper limit.) in, Indicates the frequency of the target leaf symbol. Indicates the original codeword length of the target leaf symbol. The candidate new codeword length is represented by the target leaf symbol frequency, the current codeword length, and the payoff value. The candidates are sorted to form a candidate frontier set. S23, Execute multi-level Search: Construct a candidate frontier set based on the theoretical returns of candidate uplifts, and perform multi-level... To avoid an excessively large search space, a theoretically optimistic upper bound on the candidate path is introduced. S24. Duplicate codeword check: For each candidate to be proposed, a duplicate codeword check is first performed. If the candidate new codeword already exists in the current codeword set, it means that applying the candidate will directly result in a duplicate codeword. There is no need to continue the subsequent legality judgment, and the candidate is directly rejected. S25, Constraint Incremental Decision: Perform the push-up of each candidate path. Constrained incremental determination, maintaining the current codebook and: ； in, Indicates the first Since candidate lifting only changes the codeword length of the target leaf symbol, it is not necessary to recalculate the entire codeword when judging a single candidate. And, using an incremental update method: ； in, The original codeword length of the target leaf symbol, For the candidate new codeword length, if the updated and If the value is greater than the preset threshold, it means that the candidate does not even meet the necessary length constraint of a uniquely decodable codebook, and the candidate is directly rejected. S26. Short string ambiguity witness pre-pruning: Before performing a complete, unique, and decodable determination, first prune based on the code. Constructing a fast pre-pruning link, including code Construction: Insert the current codeword set bit by bit into the binary branch. In the structure, the corresponding symbol set is recorded at the terminal node. Candidate new codeword separability check: If the candidate new codeword itself can be obtained by segmenting other codewords, then applying the candidate will inevitably introduce ambiguity, therefore the candidate is directly rejected. Short string ambiguity witness check: At least the following short strings are constructed for multi-segmentation detection: "、" + "、" + "as well as" + If any of the above short strings can be segmented in two or more ways, it means that the candidate will introduce ambiguity at the short string level, and it will be rejected directly. S27. Internal path followed by symbol check: For each internal node, pre-record all existing codewords with the internal node path as a prefix and their corresponding symbols. If a codeword can still be divided by other codewords after removing the internal node path prefix, the corresponding internal path lifting will cause ambiguity and will be directly rejected. S28 Type-unique decodable determination: For those that pass the duplicate codeword check, Constrained incremental decision-making and short string ambiguous witness pre-pruning candidates are then executed. Type unique decodable determination; S29. Candidate Path Submission and Codebook Update: If the total number of bytes in the compressed file corresponding to a candidate path is less than the current codebook, then the candidate path is submitted. Candidate path submission includes at least replacing the original codeword of the target leaf symbol with the new codeword corresponding to the internal node path; updating the codeword count table; updating the symbol attribution of internal nodes and leaf nodes in the tree structure; and updating... And; save the current best codebook, and stop further upward search if no legal and better candidate path is found in this round of search; S3, Container Output Layer: The original symbol stream is encoded using a dual-stream structured container. First, the original symbol stream is converted into a run sequence. Then, the run sequence is split into a symbol stream and a length stream. The symbol stream and the length stream are entropy encoded separately. Finally, they are encapsulated into a structured container for encoding.

2. The repeatable prefix compression coding method based on data features as described in claim 1, characterized in that: In step S1, a bit-width selection algorithm is used to determine the actual symbol width. The specific steps are as follows: S11. Read the original byte sequence of the file to be compressed and obtain the profile information. The profile information includes at least: the total number of bytes in the file, the number of different byte values, the proportion of printable characters, the proportion of word-like characters, information entropy, the proportion of adjacent identical bytes, the cumulative proportion of the first 8 high-frequency adjacent byte pairs, and whether the file header contains the MZ identifier or the ELF identifier. S12. When the user requests a width of less than 8 bits, determine whether the file to be compressed meets the conditions for a text file or an executable file based on the profile information; when the total number of bytes in the file to be compressed is greater than or equal to 1024 bytes, the proportion of printable characters is greater than or equal to 98%, the proportion of word-like characters is greater than or equal to 90%, and the number of different byte values ​​is less than or equal to 96, increase the main symbol width to 8 bits; when the total number of bytes in the file to be compressed is greater than or equal to 4096 bytes, the proportion of printable characters is greater than or equal to 95%, and the number of different byte values ​​is less than or equal to 128, increase the main symbol width to 8 bits; when the total number of bytes in the file to be compressed is greater than or equal to 65536 bytes and the file header contains an MZ identifier or an ELF identifier, increase the main symbol width to 8 bits. S13. When the user-requested width is greater than or equal to 8 bits, the user-requested width is directly used as the current symbol width; when the user-requested width is less than 8 bits and the file to be compressed does not meet the conditions of text file and executable file, the user-requested width is kept as the current symbol width. S14. When the current symbol width is 8 bits, and the total number of bytes in the file to be compressed is greater than or equal to 65536 bytes, the proportion of printable characters is less than or equal to 10%, the proportion of word-like characters is less than or equal to 5%, the number of different byte values ​​is less than or equal to 192, the information entropy is less than or equal to 2.25, the proportion of adjacent identical bytes is greater than or equal to 65%, and the cumulative proportion of the first 8 high-frequency adjacent byte pairs is greater than or equal to 60%, the file to be compressed is determined to meet the super-repeating binary judgment condition, and the current symbol width is further expanded to the second symbol width; among them, when the number of different byte values ​​is greater than or equal to 96, the information entropy is greater than or equal to 1.0, the proportion of printable characters is less than or equal to 3%, the proportion of word-like characters is less than or equal to 2%, and the proportion of adjacent identical bytes is less than or equal to 87%, 12 bits are selected as the second symbol width; otherwise, 16 bits are selected as the second symbol width.

3. The repeatable prefix compression coding method based on data features as described in claim 1, characterized in that: The multi-layer beam search in S23 includes selecting candidates with higher theoretical returns from the current round of candidate frontier set to enter the first-level beam, re-collecting internal nodes and leaf nodes after applying the first-level candidates to construct the second-level candidate frontier set, continuing to expand the second-level candidates to form multiple candidate paths, continuing to expand the third-level and fourth-level candidate paths within the preset budget, and performing legality judgment and total byte evaluation of compressed files on different candidate paths respectively.

4. The repeatable prefix compression coding method based on data features as described in claim 1, characterized in that: The upper bound of the theoretical optimistic return in S23 includes estimating the maximum theoretical return that can be obtained by the remaining single-step upward movement, converting the theoretical return into an optimistic estimate of the total bytes of the compressed file, and terminating the subsequent expansion of the candidate path if the optimistic estimate is not better than the current best result.

5. The repeatable prefix compression coding method based on data features as described in claim 1, characterized in that: S28 The determination of a uniquely decodable codeword includes collecting all non-empty codewords; checking for duplicate codewords; and constructing a suffix state set. Iteratively expand the set of suffix states; If an empty suffix conflict occurs, the candidate is determined not to be uniquely decodable; If the state expansion terminates without any conflict, the candidate is determined to be uniquely decodable. The determination result is then cached to reuse the determination result under repeated candidate states.

6. The repeatable prefix compression coding method based on data features as described in claim 1, characterized in that: The specific steps of S3 are as follows: S31. Constructing a run sequence: Merge consecutive identical symbols into a single run and represent it as follows: binary; S32. Calculate the two frequency tables: calculate the frequency table of the run symbol and the frequency table of the run length respectively; S33. Establish two sets of Huffman codes respectively: Construct two sets of Huffman codes according to the run symbol frequency table and the run length frequency table respectively; S34. Encode two bit streams: Traverse each run, write the run symbol into the symbol stream, and write the run length into the length stream to obtain independent symbol bit streams and length bit streams; S35. Construct two sets of sparse code tables: The first sign value is recorded directly, and subsequent sign values ​​are represented using the delta method and passed through... Encoding, code length adopts Sequence representation; S36. Encapsulate as ENCR encoding: First encapsulate the container header, then write the symbol table index part, symbol code length part, length table index part, length code length part, symbol bit stream, and length bit stream in sequence.

Images