A data compression method and system

Abstract

The present application relates to the technical field of data compression, in particular to a data compression method and system, comprising: an LZ module, a text stream module, a sequence stream module, an interval entropy encoding framework, the LZ module is used for screening repeated data of original data, generating a text stream and a sequence stream; the LZ module screens repeated data through a hash verification method, specifically: calculating a hash value for the input data block, determining repeated data based on hash value comparison to avoid direct data content comparison; the text stream module is used for entropy encoding the text stream; the present application innovatively proposes an interval entropy encoding framework, constructs a more optimal probability model through artificial intelligence and mathematical methods, trains a more optimal finite state machine, and uses the optimal state machine in the compression and decompression process, efficiently solving the problem of slow decompression speed of neural network type methods.

Description

Technical Field

[0001] This invention relates to the field of lossless data compression technology, specifically a data compression method and system. Background Technology

[0002] Data compression is a technology that uses algorithms to reduce data redundancy and improve storage and transmission efficiency. It is mainly divided into lossless compression (completely restoring the original data) and lossy compression (allowing the loss of some information in exchange for a higher compression ratio). Data compression uses algorithms to reorganize data, reduce redundant information, thereby reducing storage space or accelerating transmission. Lossless compression ensures that the decompressed data is completely consistent with the original data, and is suitable for scenarios such as text and programs that require accurate restoration.

[0003] In the prior art, a data compression device and a data compression method are disclosed in publication number CN108804021B. The data compression method includes a processing unit generating compressed data based on the data to be compressed in a cache unit, the processing unit storing the compressed data in one of a plurality of physical blocks of a hard disk, each physical block of the hard disk having the same block size, the processing unit comparing the size of the compressed data with the block size, and when the size of the compressed data is smaller than the block size, the processing unit storing redundant data in the physical block, wherein the sum of the size of the redundant data and the size of the compressed data is equal to the block size.

[0004] However, existing lossless data compression technologies have the following problems:

[0005] (1) Low efficiency of duplicate data screening: Traditional LZ series algorithms identify duplicate segments by directly comparing data content, which requires frequent memory access, resulting in slow screening speed, especially when processing large volumes of raw data, the time consumption increases significantly;

[0006] (2) Insufficient adaptability of entropy coding: Although conventional Huffman coding is simple to implement, it has a fixed allocation of coding length for high-frequency and low-frequency bytes, which cannot dynamically adapt to changes in data distribution, resulting in limited compression ratio; although FSE coding is fast, it occupies a high amount of storage space.

[0007] (3) Lack of correlation utilization in sequence stream coding: Some compression techniques use independent coding for sequences that characterize repetitive data features, without mining the inherent correlation between data, resulting in coding redundancy. Summary of the Invention

[0008] The purpose of this invention is to provide a data compression method and system to solve the problems mentioned in the background art.

[0009] To achieve the above objectives, the present invention provides the following technical solution:

[0010] A data compression system, comprising:

[0011] The LZ module is used to screen for duplicate data in the original data and generate text streams and sequence streams. The LZ module screens for duplicate data using a hash verification method. The text stream is the data that appears for the first time or only once in the original data. The sequence stream contains three numbers that represent the repetition characteristics of the original data. The three numbers are the current position, the repetition position, and the repetition length.

[0012] The text stream module is used for entropy encoding of the text stream. The text stream module adopts a novel entropy encoding method, which includes constructing a hybrid code table structure of static code table and dynamic code table. By statistically analyzing the byte distribution of the initial data of the text stream, high-frequency bytes and low-frequency bytes are divided. High-frequency bytes are assigned fixed short codes and stored as a static code table. Low-frequency bytes are constructed with equal-length basic codes and extended codes and stored as a dynamically adjustable dynamic code table. During the encoding process, the byte frequency deviation and data accumulation are monitored in real time, and the code table is updated to adapt to changes in data distribution.

[0013] The sequence stream module is used to encode the sequence stream. The sequence stream module compresses the sequence stream by mining the correlation between three numbers and using a joint encoding method. The correlation between the three numbers includes the correlation between the difference range of the repeat position and the current position, and the correlation between the repeat length and the numerical range of the repeat position.

[0014] The interval entropy coding framework is used to optimize the encoding of the output of text stream and sequence stream modules by constructing probabilistic models and finite state machines based on artificial intelligence and mathematical methods. The finite state machine is a trained optimal state machine used for efficient state transitions in the compression and decompression process. The decompression process does not rely on neural network inference. The number of states of the finite state machine is finite, and the state transition logic is compatible with the CPU instruction system.

[0015] Preferably, the hash verification method specifically includes the following steps:

[0016] Step A1: Divide the original data into blocks according to the preset block size, and calculate the hash value of each data block. The hash value is calculated using the CRC32 hash function or the XXHash hash function.

[0017] Step A2: Create a hash table, which is used to store the hash value of each data block and the corresponding location information of the data block in the original data;

[0018] Step A3: When processing a new data block, calculate the hash value of the new data block and query the hash table; if there is a record in the hash table that matches the hash value of the new data block, then the new data block is determined to be duplicate data; if there is no matching record in the hash table, then the new data block is determined to be the first occurrence of data, and the hash value and position information of the new data block are stored in the hash table.

[0019] Preferably, in the novel entropy coding method, the division between high-frequency bytes and low-frequency bytes is as follows:

[0020] The information entropy of the text stream is calculated using the formula for calculating the information entropy of the text stream. A frequency threshold is set, and bytes that appear more frequently than the frequency threshold are classified into the high-frequency byte set, while the remaining bytes are classified into the low-frequency byte set.

[0021] The frequency threshold is 0.5% of the byte frequency in the text stream, and the information entropy calculation formula is:

[0022]

[0023] in, For byte types, For the first Frequency of occurrence of each byte;

[0024] The encoding length of high-frequency bytes in the static code table ranges from 3 to 8 bits. The basic code of low-frequency bytes in the dynamic code table is fixed at 10 bits, the initial length of the extended code is 2 bits, and the length of the extended code can be adjusted according to the real-time frequency of the bytes. After adjustment, the length of the extended code ranges from 1 to 4 bits.

[0025] Preferably, the joint encoding method specifically includes:

[0026] Let the current positions of the three numbers in the sequence stream be... The repeated positions are The repeat length is Calculate the difference between the repeated position and the current position. ;

[0027] Based on difference The preset value range is divided into intervals, and each interval is... Combine and construct conditional probability distribution models;

[0028] Based on the conditional probability distribution model The three numbers are co-coded, and the code length satisfies the formula:

[0029]

[0030] in, for The conditional probability of a combination within the corresponding interval.

[0031] Preferably, the construction of the interval entropy coding framework includes the following steps:

[0032] Step B1: Construct a probability model based on the training data using machine learning algorithms and mathematical statistical methods; the machine learning algorithm is gradient descent, the mathematical statistical method is maximum likelihood estimation, and the parameters of the probability model are solved by maximizing the likelihood function;

[0033] Step B2: Train a finite state machine based on a probabilistic model, determine the state transition rules of the finite state machine, so that the finite state machine can complete the encoding of the text stream module and sequence stream module output through state transitions during the compression process, and complete the decoding of compressed data through state transitions during the decompression process;

[0034] Step B3: Verify the coding efficiency of the finite state machine, ensuring that the coding efficiency satisfies the formula:

[0035]

[0036] in, For coding efficiency, The actual information entropy of the output data for text streams and sequence streams. This represents the maximum information entropy of the corresponding data.

[0037] A data compression method includes the following steps:

[0038] Step C1: Input the raw data into the LZ module, screen for duplicate data using the hash verification method, mark the data that appears for the first time or only once as a text stream, and combine the three numbers representing the characteristics of duplicate data—current position, duplicate position, and duplicate length—to form a sequence stream;

[0039] Step C2: Input the text stream into the text stream module, encode the text stream using a novel entropy coding method, and obtain the text stream encoded data;

[0040] Step C3: Input the sequence stream into the sequence stream module, mine the correlation of the difference range and the correlation of the numerical range of the three numbers in the sequence stream, and encode the sequence stream through joint encoding to obtain the sequence stream encoded data;

[0041] Step C4: Invoke the interval entropy coding framework. The interval entropy coding framework loads the trained optimal finite state machine to optimize the encoding of text stream and sequence stream data, generating a compressed file and the corresponding model file. The model file stores the parameters and state transition rules of the optimal finite state machine.

[0042] Preferably, the method further includes a decompression step, which includes:

[0043] Step C5: Obtain the compressed file and model file, load the model file into the interval entropy coding framework, decode the compressed file through the decompression state machine in the interval entropy coding framework, and restore the text stream encoded data and sequence stream encoded data; the decompression state machine matches the optimal finite state machine used in the compression process, and the state transition logic of the decompression state machine is compatible with the CPU instruction system;

[0044] Step C6: Input the restored text stream encoded data into the text stream module, and decode it through the inverse process of the novel entropy encoding method to obtain the original text stream; input the restored sequence stream encoded data into the sequence stream module, and decode it through the inverse process of the joint encoding method to obtain the original sequence stream;

[0045] Step C7: Input the original text stream and the original sequence stream into the LZ module. The LZ module copies the corresponding segments from the decompressed text stream data and the previously restored repeated data according to the current position, repetition position and repetition length in the sequence stream, and splices them together to obtain the original data.

[0046] Compared with the prior art, the beneficial effects of the present invention are:

[0047] During the compression process of the LZ module, duplicate data is screened out using a hash verification method to avoid direct comparison, thereby reducing memory access and improving compression speed.

[0048] In the text stream module, a new entropy coding method was designed, which has a compression and decompression speed close to that of FSE coding, but occupies less storage space, thereby improving the compression ratio of the text stream part, while having little impact on the compression and decompression speed.

[0049] In the sequence stream module, a correlation between a set of three numbers was found, which can be used to design a targeted compression method to improve the compression ratio.

[0050] This invention innovatively proposes an interval entropy coding framework, constructs a better probability model through artificial intelligence and mathematical methods, trains a better finite state machine, and uses the optimal state machine in the compression and decompression process, which efficiently solves the problem of slow decompression speed of neural network methods.

[0051] This invention improves data storage efficiency by increasing the data compression ratio (more data can be stored in the same storage space).

[0052] This invention improves data transmission efficiency by increasing the data compression ratio (allowing for the transmission of more data under the same bandwidth network conditions). Attached Figure Description

[0053] Figure 1This is a schematic diagram of a data compression system according to the present invention.

[0054] Figure 2 This is a schematic flowchart of a data compression method according to the present invention. Detailed Implementation

[0055] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0056] like Figure 1 As shown, a data compression system includes:

[0057] The LZ module is used to screen for duplicate data in the original data and generate text streams and sequence streams. The LZ module screens for duplicate data using a hash verification method. The text stream is the data that appears for the first time or only once in the original data. The sequence stream contains three numbers that represent the repetition characteristics of the original data. The three numbers are the current position, the repetition position, and the repetition length.

[0058] The hash verification method specifically includes the following steps:

[0059] Step A1: Divide the original data into blocks according to the preset block size, and calculate the hash value of each data block. The hash value is calculated using the CRC32 hash function or the XXHash hash function.

[0060] The formula for calculating hash value is:

[0061] ,

[0062] in, For data blocks, For hash functions (such as CRC32, XXHash). For data blocks The hash value;

[0063] Step A2: Create a hash table, which is used to store the hash value of each data block and the corresponding location information of the data block in the original data;

[0064] Step A3: When processing a new data block, calculate the hash value of the new data block and query the hash table; if there is a record in the hash table that matches the hash value of the new data block, the new data block is determined to be duplicate data; if there is no matching record in the hash table, the new data block is determined to be the first occurrence of data, and the hash value and position information of the new data block are stored in the hash table.

[0065] The text stream module is used for entropy encoding of the text stream. The text stream module adopts a novel entropy encoding method, which includes constructing a hybrid code table structure of static code table and dynamic code table. By statistically analyzing the byte distribution of the initial data of the text stream, high-frequency bytes and low-frequency bytes are divided. High-frequency bytes are assigned fixed short codes and stored as static code tables. Low-frequency bytes are constructed with equal-length basic codes and extended codes and stored as dynamically adjustable dynamic code tables. During the encoding process, the byte frequency deviation and data accumulation are monitored in real time, triggering code table updates to adapt to changes in data distribution.

[0066] The novel entropy coding method is implemented as follows:

[0067] Step E1, Initialization Phase: Construct the basic code table. By statistically analyzing the byte distribution of the initial text stream data, generate a static code table and an initial dynamic code table to provide a benchmark for subsequent encoding.

[0068] Step E2: Data sampling and frequency statistics;

[0069] Before reading the text stream Using bytes as a statistical window, iterate through all the data within the window and count the occurrences of 256 different bytes. ;

[0070] Calculate the frequency of occurrence of each byte. The formula for calculating frequency is:

[0071] ,

[0072] in, For the first The number of times each byte appears. This represents the total number of bytes in the text stream. For the first The frequency of the seed byte;

[0073] Step E3: Divide the bytes into high-frequency and low-frequency bytes:

[0074] By calculating the information entropy formula of the text stream, high-frequency and low-frequency bytes are divided.

[0075] in, ,

[0076] in, For byte types, The information entropy of the text stream is determined by allocating shorter codes to high-frequency bytes and longer codes to low-frequency bytes, so that the total actual encoding length satisfies the following:

[0077] ,

[0078] in, The total length of the encoding (bits). A constant (representing code table overhead, typically) );

[0079] Set frequency threshold (i.e., the frequency of occurrence exceeds 0.5%), which will satisfy the condition. The bytes are categorized into the high-frequency byte set. The remaining bytes are grouped into the low-frequency byte set. .

[0080] Step E4: Construct a static code table:

[0081] right For high-frequency bytes, a fixed short code is assigned using the Huffman algorithm:

[0082] Based on byte frequency sorting, the higher the frequency, the shorter the code length is allocated, and the code length range is controlled between 3-8 bits to ensure optimal encoding efficiency.

[0083] Store the "byte-encoding-code length" mapping relationship as a static code table. It is preloaded into a fixed area of ​​memory to improve call speed.

[0084] Step E5: Construct the initial dynamic code table:

[0085] right The low-frequency bytes are used to construct an initial dynamic code table with a "fixed-length basic code + extended code" structure. :

[0086] The base code is fixed at 10 bits, and the initial length of the extended code is 2 bits (total code length 12 bits). The mapping relationship is stored in a read-write buffer and supports subsequent dynamic adjustment.

[0087] The sequence stream module is used to encode the sequence stream. The sequence stream module compresses the sequence stream by mining the correlation between three numbers and using a joint encoding method. The correlation between the three numbers includes the correlation between the difference range between the repeat position and the current position, and the correlation between the repeat length and the numerical range of the repeat position.

[0088] Joint encoding method: Efficient byte-by-byte encoding, based on the hybrid code table generated during the initialization phase, encodes the remaining data of the text stream, with the following steps:

[0089] Step D1, Byte Classification and Encoding Matching:

[0090] Read the text stream data byte by byte and determine the current byte. Belonging to:

[0091] like Call the static code table The output corresponds to a fixed short code;

[0092] like Call the dynamic code table Output the current "basic code + extended code" combination encoding.

[0093] Step D2, Encoding Cache and Count Update:

[0094] The encoded result of each output is written to the encoding buffer (default size = 32KB) to avoid frequent I / O operations; at the same time, the real-time count of the current byte b is updated. This provides data support for subsequent code table updates.

[0095] Step D3: Update trigger condition check:

[0096] Two trigger conditions are monitored in real time; if either condition is met, the code table update phase begins.

[0097] Frequency deviation condition: Calculate the current byte real-time frequency ( (Total number of bytes encoded since the last update), if it exists and (Frequency deviation exceeds 0.3%), or there is and (Low-frequency bytes are upgraded to high-frequency bytes), triggering an update;

[0098] Data accumulation condition: If the number of bytes encoded since the last update reaches... Regardless of whether the frequency deviation meets the standard, an update will be forcibly triggered.

[0099] Step D4, Code Table Update Phase: Adaptive Adjustment of the Code Table. When the update condition is triggered, the dynamic and static code tables are adaptively adjusted to ensure that the code tables are synchronized with changes in data distribution. The steps are as follows:

[0100] Step D5: Recalculate real-time frequencies:

[0101] Using the encoded data from the last update to the present as the new statistical window, recalculate the real-time frequency of all bytes within the window. Update the high-frequency / low-frequency byte set: [The original text appears to be incomplete and requires further context.] China satisfies byte shift , will the original China satisfies byte shift .

[0102] Step D6: Fine-tune the static code table:

[0103] For the updated The Huffman algorithm was re-adjusted to adjust the code length allocation of the static code, ensuring that high-frequency bytes always correspond to the shortest code, and the static code table was updated. And synchronize it to a fixed area of ​​memory.

[0104] Step D7: Optimize the dynamic code table:

[0105] For the updated The length of the spreading code is adjusted according to the real-time frequency: bytes with increasing frequency shorten the spreading code (minimum 1 bit, total code length 11 bits), bytes with decreasing frequency length lengthen the spreading code (maximum 4 bits, total code length 14 bits), and the dynamic code table is updated. And synchronize it to the cache.

[0106] Step D8: Reset the counter:

[0107] Reset real-time count With cumulative encoded bytes Then, return to the encoding stage to continue processing subsequent data.

[0108] The interval entropy coding framework is used to optimize the encoding of the output of text stream and sequence stream modules by constructing probabilistic models and finite state machines based on artificial intelligence and mathematical methods. The finite state machine is a trained optimal state machine used for efficient state transitions in the compression and decompression process. The decompression process does not rely on neural network inference. The number of states of the finite state machine is finite, and the state transition logic is compatible with the CPU instruction system.

[0109] like Figure 2 As shown, a data compression method includes the following steps:

[0110] Step C1: Input the raw data into the LZ module, screen for duplicate data using the hash verification method, mark the data that appears for the first time or only once as a text stream, and combine the three numbers representing the characteristics of duplicate data—current position, duplicate position, and duplicate length—to form a sequence stream;

[0111] Step C2: Input the text stream into the text stream module, encode the text stream using a novel entropy coding method, and obtain the text stream encoded data;

[0112] Step C3: Input the sequence stream into the sequence stream module, mine the correlation of the difference range and the correlation of the numerical range of the three numbers in the sequence stream, and encode the sequence stream through joint encoding to obtain the sequence stream encoded data;

[0113] Step C4: Invoke the interval entropy coding framework. The interval entropy coding framework loads the trained optimal finite state machine to optimize the encoding of text stream and sequence stream data, generating a compressed file and the corresponding model file. The model file stores the parameters and state transition rules of the optimal finite state machine.

[0114] The construction of the interval entropy coding framework includes the following steps:

[0115] Step B1: Construct a probability model based on the training data using machine learning algorithms and mathematical statistical methods; the machine learning algorithm is gradient descent, the mathematical statistical method is maximum likelihood estimation, and the parameters of the probability model are solved by maximizing the likelihood function;

[0116] Step B2: Train a finite state machine based on a probabilistic model, determine the state transition rules of the finite state machine, so that the finite state machine can complete the encoding of the text stream module and sequence stream module output through state transitions during the compression process, and complete the decoding of compressed data through state transitions during the decompression process;

[0117] Step B3: Verify the coding efficiency of the finite state machine, ensuring that the coding efficiency satisfies the formula:

[0118]

[0119] in, For coding efficiency, The actual information entropy of the output data for text streams and sequence streams. This represents the maximum information entropy of the corresponding data.

[0120] The decompression steps include:

[0121] Step C5: Obtain the compressed file and model file, load the model file into the interval entropy coding framework, decode the compressed file through the decompression state machine in the interval entropy coding framework, and restore the text stream encoded data and sequence stream encoded data; the decompression state machine matches the optimal finite state machine used in the compression process, and the state transition logic of the decompression state machine is compatible with the CPU instruction system;

[0122] Step C6: Input the restored text stream encoded data into the text stream module, and decode it through the inverse process of the novel entropy encoding method to obtain the original text stream; input the restored sequence stream encoded data into the sequence stream module, and decode it through the inverse process of the joint encoding method to obtain the original sequence stream;

[0123] Step C7: Input the original text stream and the original sequence stream into the LZ module. The LZ module copies the corresponding segments from the decompressed text stream data and the previously restored repeated data according to the current position, repetition position and repetition length in the sequence stream, and splices them together to obtain the original data.

[0124] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A data compression system, characterized in that, include: The LZ module is used to screen for duplicate data in the original data and generate text streams and sequence streams. The LZ module screens for duplicate data using a hash verification method. The text stream is the data that appears for the first time or only once in the original data. The sequence stream contains three numbers that represent the repetition characteristics of the original data. The three numbers are the current position, the repetition position, and the repetition length. The text stream module is used for entropy encoding of the text stream. The text stream module adopts a novel entropy encoding method, which includes constructing a hybrid code table structure of static code table and dynamic code table. By statistically analyzing the byte distribution of the initial data of the text stream, high-frequency bytes and low-frequency bytes are divided. High-frequency bytes are assigned fixed short codes and stored as a static code table. Low-frequency bytes are constructed with equal-length basic codes and extended codes and stored as a dynamically adjustable dynamic code table. During the encoding process, the byte frequency deviation and data accumulation are monitored in real time, and the code table is updated to adapt to changes in data distribution. The sequence stream module is used to encode the sequence stream. The sequence stream module compresses the sequence stream by mining the correlation between three numbers and using a joint encoding method. The correlation between the three numbers includes the correlation between the difference range of the repeat position and the current position, and the correlation between the repeat length and the numerical range of the repeat position. The interval entropy coding framework is used to optimize the encoding of the output of text stream and sequence stream modules by constructing probabilistic models and finite state machines based on artificial intelligence and mathematical methods. The finite state machine is a trained optimal state machine used for efficient state transitions in the compression and decompression process. The decompression process does not rely on neural network inference. The number of states of the finite state machine is finite, and the state transition logic is compatible with the CPU instruction system.

2. The data compression system according to claim 1, characterized in that, The hash verification method specifically includes the following steps: Step A1: Divide the original data into blocks according to the preset block size, and calculate the hash value of each data block. The hash value is calculated using the CRC32 hash function or the XXHash hash function. Step A2: Create a hash table, which is used to store the hash value of each data block and the corresponding location information of the data block in the original data; Step A3: When processing a new data block, calculate the hash value of the new data block and query the hash table; if there is a record in the hash table that matches the hash value of the new data block, then the new data block is determined to be duplicate data; if there is no matching record in the hash table, then the new data block is determined to be the first occurrence of data, and the hash value and position information of the new data block are stored in the hash table.

3. The data compression system according to claim 1, characterized in that, In the novel entropy coding method, the division between high-frequency bytes and low-frequency bytes is as follows: The information entropy of the text stream is calculated using the formula for calculating the information entropy of the text stream. A frequency threshold is set, and bytes that appear more frequently than the frequency threshold are classified into the high-frequency byte set, while the remaining bytes are classified into the low-frequency byte set. The frequency threshold is 0.5% of the byte frequency in the text stream, and the information entropy calculation formula is: in, For byte types, For the first The frequency of occurrence of each byte.

4. The data compression system according to claim 1, characterized in that, The joint coding method specifically includes: Let the current positions of the three numbers in the sequence stream be... The repeated positions are The repeat length is Calculate the difference between the repeated position and the current position. Based on difference The preset value range is divided into intervals, and each interval is... Combining and constructing conditional probability distribution models; based on conditional probability distribution models, The three numbers are co-coded, and the code length satisfies the formula: in, for The conditional probability of a combination within the corresponding interval.

5. A data compression system according to claim 1, characterized in that, The construction of the interval entropy coding framework includes the following steps: Step B1: Construct a probability model based on the training data using machine learning algorithms and mathematical statistical methods; the machine learning algorithm is gradient descent, the mathematical statistical method is maximum likelihood estimation, and the parameters of the probability model are solved by maximizing the likelihood function; Step B2: Train a finite state machine based on a probabilistic model, determine the state transition rules of the finite state machine, so that the finite state machine can complete the encoding of the text stream module and sequence stream module output through state transitions during the compression process, and complete the decoding of compressed data through state transitions during the decompression process; Step B3: Verify the coding efficiency of the finite state machine, ensuring that the coding efficiency satisfies the formula: in, For coding efficiency, The actual information entropy of the output data for text streams and sequence streams. This represents the maximum information entropy of the corresponding data.

6. A data compression method, characterized in that, Includes the following steps: Step C1: Input the raw data into the LZ module, screen for duplicate data using the hash verification method, mark the data that appears for the first time or only once as a text stream, and combine the three numbers representing the characteristics of duplicate data—current position, duplicate position, and duplicate length—to form a sequence stream; Step C2: Input the text stream into the text stream module and encode the text stream using a novel entropy coding method to obtain text stream encoded data. The novel entropy coding method includes constructing a hybrid code table structure of static code table and dynamic code table. By statistically analyzing the byte distribution of the initial text stream data, high-frequency bytes and low-frequency bytes are divided. Fixed short codes are assigned to high-frequency bytes and stored as a static code table. An equal-length basic code and extended code structure is constructed for low-frequency bytes and stored as a dynamically adjustable dynamic code table. During the encoding process, the byte frequency deviation and data accumulation are monitored in real time, triggering code table updates to adapt to changes in data distribution. Step C3: Input the sequence stream into the sequence stream module, mine the correlation between the difference range and the numerical range of the three numbers in the sequence stream, and encode the sequence stream using a joint encoding method to obtain the sequence stream encoded data. The joint encoding method specifically includes: assuming the current position of the three numbers in the sequence stream is... The repeated positions are The repeat length is Calculate the difference between the repeated position and the current position. Based on difference The preset value range is divided into intervals, and each interval is... Combining and constructing conditional probability distribution models; based on conditional probability distribution models, The three numbers are co-coded, and the code length satisfies the formula: in, for The conditional probability of a combination within the corresponding interval; Step C4: Invoke the interval entropy coding framework. The interval entropy coding framework loads the trained optimal finite state machine to optimize the encoding of text stream and sequence stream data, generating a compressed file and the corresponding model file. The model file stores the parameters and state transition rules of the optimal finite state machine.

7. A data compression method according to claim 6, characterized in that, It also includes a decompression step, which includes: Step C5: Obtain the compressed file and model file, load the model file into the interval entropy coding framework, decode the compressed file through the decompression state machine in the interval entropy coding framework, and restore the text stream encoded data and sequence stream encoded data; the decompression state machine matches the optimal finite state machine used in the compression process, and the state transition logic of the decompression state machine is compatible with the CPU instruction system; Step C6: Input the restored text stream encoded data into the text stream module, and decode it through the inverse process of the novel entropy encoding method to obtain the original text stream; input the restored sequence stream encoded data into the sequence stream module, and decode it through the inverse process of the joint encoding method to obtain the original sequence stream; Step C7: Input the original text stream and the original sequence stream into the LZ module. The LZ module copies the corresponding segments from the decompressed text stream data and the previously restored repeated data according to the current position, repetition position and repetition length in the sequence stream, and splices them together to obtain the original data.

Images