A data compression method based on data difference features

Abstract

This invention discloses a data compression method based on data difference features. The method includes a compression code table generation stage and a data compression stage. During code table generation, the data is first segmented, retaining the first data element of each segment. Subsequent data elements are then subtracted from their preceding elements, and the difference features are statistically analyzed to generate the compression code table. During data compression, the same steps are used to obtain the data difference distribution. The data is then encoded using the compression code table, reducing the total data length, and finally, the compressed data segment is output. This invention generates a code table based on the difference distribution features between data elements, performs lossless data compression, and, considering the characteristics of hardware devices, performs segmented data compression, allowing for parallel operation, thereby improving compression and transmission efficiency and saving transmission bandwidth.

Description

Technical Field

[0001] This invention belongs to the field of computer technology, especially the field of data processing technology, and specifically relates to a data compression method based on data difference features. Background Technology

[0002] In the information age, massive amounts of data are generated daily. The transmission of this data places enormous demands on hardware storage capacity and network bandwidth, bottlenecking the speed at which people can access information. Simply expanding hardware storage resources and upgrading network bandwidth is insufficient to solve the problem of storing and transmitting massive amounts of information. Therefore, the development of compression algorithms is crucial. Data compression is the most economical and efficient method for solving data storage and transmission problems. For general data, there is varying degrees of redundancy. Taking image data as an example, this redundancy manifests in: 1. Spatial redundancy: In an image, the current pixel is correlated with surrounding pixels; 2. Temporal redundancy: In a continuous image sequence, pixels in the current frame are correlated with pixels in previous and subsequent frames; 3. Visual redundancy: The information received by the human eye is limited, and removing some details is irrelevant to the human eye. Therefore, excellent compression algorithms can remove redundant information and retain only the essential information.

[0003] Common compression algorithms can be divided into lossless compression and lossy compression. Lossless compression algorithms include: differential coding, RLE run-length encoding, Huffman coding, LZW coding, arithmetic coding, etc. Lossy compression algorithms include: predictive coding, perceptual coding, fractal compression, wavelet compression, JPEG / MPEG, etc. Huffman proposed a coding method in 1952 that constructs codewords with the shortest average length based entirely on the probability of character occurrence; this is generally called Huffman coding. This method is a variable-length coding, and its core idea is that "information with higher frequency should be represented using fewer bits." Its coding steps can be summarized as follows:

[0004] (1) Perform probability statistics on the information appearing in the information block to obtain the probability distribution of the information;

[0005] (2) Arrange the information in descending order of probability of occurrence;

[0006] (3) Assign values ​​of 0 and 1 to the two lowest probability information respectively;

[0007] (4) The sum of the two lowest probabilities is added back into the probability distribution as a new piece of information;

[0008] (5) Rearrange the probabilities in descending order;

[0009] (6) Repeat (1)-(5) until only two probability distributions remain;

[0010] (7) After assigning binary 0 and 1 respectively, take this as the root node and write out the binary code on the path in reverse order of the assignment to obtain the Huffman code.

[0011] However, for compression algorithms to be fully implemented, modifications to the hardware are needed to suit the specific characteristics of the hardware, and multiple compression algorithms need to be combined to minimize data redundancy. Summary of the Invention

[0012] The purpose of this invention is to provide a data compression method based on data difference features.

[0013] The method of the present invention includes generating a compression code table, selecting a compression code table, and data compression.

[0014] The process of generating the compressed code table begins by segmenting the input data. For the 0th segment, sec[0] = 0; for the 1st segment, sec[1] = 1; and for the nth segment, sec[n] = [2...]. n-1 ,2 n -1],2≤n≤8;Statistically analyze the distribution of input data in each segment, and then perform Huffman coding on the data distribution to generate a compressed code table.

[0015] The selection of compression code tables involves first creating M Huffman compression code tables to adapt to different distributions of input data. The input data data is then compressed sequentially using the m-th code table, where m = 1, 2, ..., M, resulting in M ​​compression results, namely output(1), output(2), ..., output(M). The binary lengths len(1), len(2), ..., len(M) of the output data output(1), output(2), ..., output(M) are then calculated. Finally, the input data is compressed using the compression code table corresponding to the minimum value among len(1), len(2), ..., len(M).

[0016] The data compression specifically refers to:

[0017] Step (1) For input data data of length L, keep the first data data(0), and subtract the previous data from the subsequent data to obtain the difference sequence diff, where the i-th data in the difference sequence diff(i) = data(i) - data(i-1), 1≤i≤L-1, diff(0) = data(0);

[0018] Step (2) writes the first data data(0) directly into the output data output, that is, the first bit of the data is not compressed;

[0019] Step (3) Write the sign bit sign(i) of the data diff(i) into the first bit of the corresponding i-th output data output(i); if diff(i) < 0, then sign(i) = 1, if diff(i) ≥ 0, then sign(i) = 0;

[0020] Step (4) Query the binary code (i) corresponding to the absolute value of the data diff(i) in the compression code table, and write the complete code(i) into the corresponding i-th output data, starting from the second bit when writing; code(i) represents the segment interval sec in which diff(i) is located;

[0021] Step (5) Calculate the offset (offset(i)) of diff(i) relative to the left boundary of sec, and write the binary code corresponding to offset(i) into the i-th output data, starting from the next bit of code(i); offset(i) = |diff(i)| - 2 n′-1 n′ is the segment number of code(i) in the compressed code table; if diff(i) = 0 or diff(i) = 1, then there is only 1 value in the corresponding interval, and offset(i) is not calculated or written.

[0022] In step (6), the i-th output data in the difference sequence is output(i) = {sign(i), code(i), offset(i)}. The complete output data of the input data is as follows:

[0023] output={data(0),[sign(1),code(1),offset(1)],…,[sign(L-1),code(L-1),offset(L-1)]}.

[0024] The beneficial effects of this invention are as follows: This invention proposes a data compression algorithm based on data difference characteristics. It generates a code table according to the difference distribution characteristics between data, performs lossless compression of data, and compresses data in segments according to the characteristics of hardware devices. It can be parallelized, thereby improving compression and transmission efficiency and saving transmission bandwidth. Attached Figure Description

[0025] Figure 1 This invention generates a compressed code representation of its intent.

[0026] Figure 2 A schematic diagram illustrating an implementation example of generating a compressed code table for the invention;

[0027] Figure 3 This is a schematic diagram of data compression according to the present invention;

[0028] Figure 4This is a schematic diagram illustrating an embodiment of the data compression method of the present invention;

[0029] Figure 5 This is a schematic diagram illustrating an implementation example of selecting a compression code table when compressing data according to the present invention.

[0030] Figure 6 This is a schematic diagram of an implementation example of the present invention, which segments data to generate a compressed header. Detailed Implementation

[0031] A data compression algorithm based on data difference characteristics is proposed. This method generates a code table based on the distribution characteristics of data differences, performs lossless compression, and segments the data for compression according to the characteristics of the hardware device. The operation can be parallelized, thereby improving compression and transmission efficiency and saving transmission bandwidth.

[0032] The present invention will now be further described in conjunction with the accompanying drawings and specific embodiments.

[0033] A data compression algorithm based on data difference features includes generating a compression code table, selecting a compression code table, and data compression.

[0034] like Figure 1 To generate a compression code table, the input data is first segmented. For the 0th and 1st segments, sec[0] = 0 and sec[1] = 1. For the nth segment, sec[n] = [2...]. n-1 ,2 n -1],2≤n≤8;Statistically analyze the distribution of input data in each segment, and then perform Huffman coding on the data distribution to generate a compressed code table.

[0035] Figure 2 This is a schematic diagram of an implementation example of the present invention based on Huffman coding to generate a code table. Input data is input to generate a Huffman code table. The input data is classified according to its numerical value to obtain the distribution of the data in different intervals. Huffman coding is used on this distribution to generate a code table for data compression.

[0036] Choosing a compression code table: In actual use, M Huffman compression code tables are created to adapt to input data with different distributions. The input data data is compressed sequentially using the m-th code table, m = 1, 2, ..., M, resulting in M ​​compression results, namely output(1), output(2), ..., output(M). The binary lengths len(1), len(2), ..., len(M) of the output data output(1), output(2), ..., output(M) are calculated. The input data is compressed using the compression code table corresponding to the minimum value among len(1), len(2), ..., len(M), thus achieving the highest compression ratio.

[0037] like Figure 3 As shown, data compression specifically involves:

[0038] Step (1) For input data data of length L, keep the first data data(0), and subtract the previous data from the subsequent data to obtain the difference sequence diff, where the i-th data in the difference sequence diff(i) = data(i) - data(i-1), 1≤i≤L-1, diff(0) = data(0);

[0039] Step (2) writes the first data data(0) directly into the output data output, that is, the first bit of the data is not compressed;

[0040] Step (3) Write the sign bit sign(i) of the data diff(i) into the first bit of the corresponding i-th output data output(i); if diff(i) < 0, then sign(i) = 1, if diff(i) ≥ 0, then sign(i) = 0;

[0041] Step (4) Query the binary code (i) corresponding to the absolute value of the data diff(i) in the compression code table, and write the complete code(i) into the corresponding i-th output data, starting from the second bit when writing; code(i) represents the segment interval sec in which diff(i) is located;

[0042] Step (5) Calculate the offset (offset(i)) of diff(i) relative to the left boundary of sec, and write the binary code corresponding to offset(i) into the i-th output data, starting from the next bit of code(i); offset(i) = |diff(i)| - 2 n′-1 n′ is the segment number of code(i) in the compressed code table; if diff(i) = 0 or diff(i) = 1, then there is only 1 value in the corresponding interval, and offset(i) is not calculated or written.

[0043] In step (6), the i-th output data in the difference sequence is output(i) = {sign(i), code(i), offset(i)}. The complete output data of the input data is as follows:

[0044] output={data(0),[sign(1),code(1),offset(1)],…,[sign(L-1),code(L-1),offset(L-1)]}.

[0045] Figure 4This is a schematic diagram of an implementation example of data compression according to the present invention. The diagram first inputs a data segment, `data`, retaining the first data segment. Starting from the second data segment, each data segment is subtracted from the previous data segment to obtain `diff`. Then, the encoding process of `diff` is shown: `diff(0) = 5`, where 5 is used as the data header and is not encoded; the binary code corresponding to 5 is 00000101. `diff(1) = -2`, using 1 to represent negative numbers and 0 to represent positive numbers, so the first bit of the encoding corresponding to -2 is 1, and 2 falls on... Figure 2 In interval 2, the corresponding code is 01, offset = 0, and the offset occupies 1 bit, so the code corresponding to 2 is 1010; diff(3) = 41, the code of the sign bit is 0, 41 falls in interval 6 of the Huffman code table, and the corresponding code is 111. The starting value of interval 6 is 32, so the offset of 41 relative to the starting value of interval 6 needs to be written, offset = 41 - 32 = 9, the offset occupies 5 bits, and the corresponding binary code is 01001, so the complete code of 41 is 011101001. The encoding of the subsequent data is completed in sequence, and the complete encoding is as follows. Figure 4 As shown.

[0046] Figure 5 This is a schematic diagram of an implementation example of selecting a compression code table when compressing data according to the present invention. Taking the selection of a compression code table from two compression code tables as an example, firstly, the data data(0) is retained, and the difference of the remaining data is calculated to obtain diff. The difference diff(i) is encoded, and the compression lengths of the two code tables are compared. The code table with the shorter compression length is selected as the compression code table.

[0047] Figure 6 This is a schematic diagram illustrating an implementation example of the present invention, which generates compressed headers by segmenting data. When compressing data, to facilitate processing, the data is usually compressed into segments, each segment of which can be of equal or unequal length. The data is segmented into d1, d2, ..., and then compressed segment by segment, generating compressed data headers h1, h2, ... and compressed data D1, D2, ... respectively. Depending on the data transmission and processing methods, the data headers can be placed together and the compressed data can be placed together (h1, h2, ..., D1, D2, ...) for transmission and processing, or the data headers and compressed data (h1, D1, h2, D2, ...) can be placed together for transmission and processing.

[0048] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A data compression method based on data difference value characteristics, comprising generating a compression code table, selecting a compression code table and data compression, characterized in that: The generating compression code table first segments input data data, the 0th segment data sec[0]=0, the 1st segment data sec[1]=1, for the nth segment data, sec[n]=[2 n-1 ,2 n -1], 2≤n≤8; statistics the distribution of input data in each segment, then the data distribution is hoffman coded to generate a compression code table; the compression code table is first made to adapt to different distribution of input data, and the input data data is compressed using the mth code table in turn, m = 1, 2, …, M, to obtain M compression results, output(1), output(2), …, output(M); the binary length len(1), len(2), …, len(M) of the output data output(1), output(2), …, output(M) is counted; and the input data is compressed using the compression code table corresponding to the minimum value of len(1), len(2), …, len(M); the data compression is specifically: step (1) for input data data with a length of L, the first data data(0) is retained, and the subsequent data is subtracted from the previous data to obtain a difference value sequence diff, wherein the ith data diff(i) = data(i) - data(i-1) in the difference value sequence, 1≤i≤L-1, and diff(0) = data(0); step (2) the first data data(0) is directly written into the output data output, and the first bit of the data is not compressed; step (3) the sign bit sign(i) of the data diff(i) is written into the first bit of the corresponding ith output data output(i); if diff(i) < 0, sign(i) = 1, and if diff(i) ≥ 0, sign(i) = 0; step (4) the absolute value of the data diff(i) is inquired in the compression code table to obtain the binary code code(i) corresponding to the absolute value, and the code(i) is completely written into the corresponding ith output data, and the writing starts from the second bit; code(i) represents the section interval sec in which diff(i) is located; Step (5) calculates the offset (offset(i)) of diff(i) relative to the left boundary of sec, and writes the binary code corresponding to offset(i) into the corresponding i-th output data, starting from the next bit of code(i); offset(i)=|diff(i)|-2 n′-1 n' is the segment number of code(i) in the compression code table; if diff(i)=0 or diff(i)=1, there is only one value in the corresponding interval, and offset(i) is not calculated and written. step (6) the ith output data output(i) in the difference value sequence is {sign(i), code(i), offset(i)}, and the complete output data of the input data data is as follows: output = {data(0), [sign(1), code(1), offset(1)], …, [sign(L-1), code(L-1), offset(L-1)]}.

Images