An encoding and decoding method for data compression transmission

Abstract

This invention discloses an encoding and decoding method for data compression transmission, comprising: at the transmitting end, inputting data u of length B*q into a sparsifier and grouping it to obtain the total number of string types m; constructing a sparsification code table of size m based on the frequency of occurrence of string types; performing sparsification mapping on data u according to the code table to obtain a sequence v of length m; interleaving it and dividing it into L groups according to each group of k bits to obtain a sequence of length Lk. v The sequence v is divided into single or multiple paths for encoding to obtain a codeword sequence c of length N. At the receiving end, the received sequence y of length N is decoded to obtain a sequence of length Lk. After deinterleaving the sequence and removing the zero-padding bits, a sequence of length Lk is obtained. After inputting the sequence into the desparser, the present invention can avoid error propagation compared with existing variable-length coding compression transmission schemes, and achieve efficient data transmission while allowing a certain degree of bit error.

Description

Technical Field

[0001] This invention belongs to the technical field of digital communication, and specifically relates to an encoding and decoding method for data compression and transmission. Background Technology

[0002] In some communication service scenarios, large amounts of data are often generated, typically consuming significant storage space and bandwidth, posing challenges to transmission and storage. Therefore, data compression technology has emerged to reduce data volume while preserving data quality as much as possible. By employing efficient compression algorithms, storage requirements and bandwidth consumption can be significantly reduced, thereby improving data transmission efficiency. This is particularly important in many applications, such as video streaming, audio transmission, and big data storage.

[0003] Existing data compression schemes such as Huffman coding and LZW coding achieve lossless data compression by constructing a dictionary and using variable-length encoding to represent more frequently occurring characters with shorter codewords. However, compression methods like Huffman coding are actually very sensitive to bit errors because they use variable-length encoding. Even a small number of bit errors can cause error propagation, thus affecting the overall data transmission quality.

[0004] Furthermore, data may be corrupted by noise during transmission. Therefore, channel coding techniques are introduced to correct these errors and ensure the reliability of data transmission. Block Markov superposition transmission (BMST) is a type of convolutional long code that can be constructed from short codes. It has a simple coding algorithm and can be decoded using a sliding window iterative decoding algorithm. The lower bound of its decoding performance can be determined by the performance of the basic code and the size of the coding memory (Sun Yat-sen University, A Block Markov Superposition Coding Method [P]: CN103152060A). By designing the construction of the connection matrix, BMST codewords with more flexible code rates can be obtained, enabling source coding, channel coding, or joint source-channel coding. Summary of the Invention

[0005] The main objective of this invention is to overcome the shortcomings and deficiencies of the prior art and to provide an encoding and decoding method for data compression transmission in communication service scenarios. While tolerating a certain amount of errors, this method can avoid error propagation and make the data more readable compared to existing variable-length compression transmission schemes.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] This invention provides an encoding and decoding method for data compression and transmission, comprising the following steps:

[0008] (1) At the transmitting end, let the data to be transmitted with a length of B*q be u, and encode the data u into a transmission codeword with a length of N. in To represent an N-dimensional binary space, the encoding method includes the following steps:

[0009] (1.1) Input the data u of length B*q into the sparsifier and map it to a length of... sequence The specific steps are as follows:

[0010] (1.1.1) Divide the data u to be transmitted, which has a length of B*q, into B groups of equal length, u = (u... (0) ,u (1) ,…,u (B-1) Each group consists of q characters. Frequency statistics are performed on each group to obtain the total number of types m and the frequency distribution results.

[0011] (1.1.2) Construct a sparse code table using the statistical frequency distribution results, for each group u (i) Use binary codewords This means that 0 ≤ i ≤ B-1;

[0012] (1.1.3) Convert the binary codeword p (i) Arranged in chronological order into a mapping sequence

[0013] (1.2) Interleave the sequence v and divide it into L groups according to each group of k bits, to obtain a sequence of length Lk. v And its sparsity is statistically obtained. Among them W H ( v )yes v The weight of Hamming;

[0014] (1.3) For the sequence v After single-channel or hierarchical multi-channel coding, a transmission codeword sequence c of length N is obtained;

[0015] (2) At the receiving end, for a received sequence y of length N, estimate the data of length B*q. The decoding method includes the following steps:

[0016] (2.1) A sequence of length Lk is obtained by single-channel or multi-channel decoding of a received sequence y of length N.

[0017] (2.2) Sequence After passing through the deinterleaver in step (1.3), a length of [length missing] is obtained. sequence

[0018] (2.3) Change the sequence Divided into groups B of equal length The length of each group is Will Input the desparser and demap it according to the sparse code table constructed in step (1.2) to obtain an estimate of the original data of length B*q.

[0019] As a preferred technical solution, the specific implementation method in step (1.1.2) is as follows:

[0020] The length of the binary codeword is determined based on the total number of types m counted in step (1.1.1). The binary codewords are reordered according to Hamming weight from smallest to largest, and finally correspond one-to-one with the character group type sorted by frequency of occurrence from largest to smallest.

[0021] As a preferred technical solution, step (1.1.2) specifically includes:

[0022] (1.1.2.1) Generate m elements of length... Non-repeating binary codeword p j Construct a sparse code table, where j = 0, 1, ..., m-1 is the type index;

[0023] (1.1.2.2) Map characters according to their frequency of occurrence, so that characters with higher frequency correspond to binary codewords with lower Hamming weight, and characters with lower frequency correspond to binary codewords with higher Hamming weight.

[0024] As a preferred technical solution, in step (1.3), the sequence is... v Perform single-channel or hierarchical multi-channel coding, specifically as follows:

[0025] Single-channel coding or multi-channel coding after data is classified can be used. Each channel can be channel coded or joint source channel coded using the grouped Markov superposition transmission coding method.

[0026] As a preferred technical solution, in step (2.1), the single-channel or multi-channel channel decoding or joint joint source-channel decoding algorithm corresponding to step (1.3) is adopted.

[0027] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0028] In the invented encoding and decoding method for data compression transmission, at the sending end, data u of length B*q is first input to a sparsifier, divided into B groups of q characters each, and the total number of string types m is obtained. Binary codewords are then assigned according to the frequency of occurrence of each string type from high to low, with a codeword length of... Furthermore, the Hamming weights are sorted from low to high, ultimately constructing a structure of size m× The code table; the data u is sparsely mapped according to the code table to obtain a length of... The sequence v is interleaved and divided into L groups of k bits each (padded with zeros if necessary) to obtain a sequence of length Lk. v . will sequence v The codeword sequence y is divided into single-channel or multi-channel encoding (channel coding / joint source-channel coding) to obtain a codeword sequence c of length N. At the receiving end, the received sequence y of length N is decoded to obtain a sequence of length Lk. will sequence After unintertwining and removing the zero padding, the length is obtained as follows: sequence After inputting the de-sparsifier, we get... Because this invention uses fixed-length codewords for mapping the original data, it avoids the error propagation phenomenon caused by bit errors that would occur with variable-length codewords, thus limiting errors to a single packet. This invention reduces the requirements for transmission reliability and, while allowing for a certain level of data error, improves data readability compared to variable-length compression transmission coding schemes. Furthermore, the coding scheme of this invention is simple to construct and can be combined with channel coding or source-channel joint coding techniques according to actual needs to achieve reliable and efficient data transmission. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is a transmission schematic diagram of the encoding and decoding method for data compression transmission according to the present invention;

[0031] Figure 2 This is the original Rubik's Cube image in Embodiment 1 of the present invention;

[0032] Figure 3This is the recovered image transmitted when the Sparsifier-BMST encoding of the Rubik's Cube image in Embodiment 1 of the present invention is performed at a signal-to-noise ratio (SNR) of 7dB.

[0033] Figure 4 This is the recovered image transmitted when the SNR = 7dB after Huffman concatenated BMST (Huffman-BMST) encoding of the Rubik's Cube image in Embodiment 1 of the present invention.

[0034] Figure 5 This is the original fireworks image from Embodiment 2 of the present invention;

[0035] Figure 6 This is the restored image of the fireworks image transmitted using Sparsifier-BMST at SNR=7dB in Embodiment 2 of the present invention;

[0036] Figure 7 This is the restored image transmitted when the fireworks image is Huffman-BMST encoded and the SNR is 7dB, as described in Embodiment 2 of the present invention. Detailed Implementation

[0037] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.

[0038] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.

[0039] Example 1

[0040] like Figure 1 As shown, this embodiment 1 provides an encoding and decoding method for data compression and transmission, including the following steps:

[0041] (1) At the transmitting end, let the data to be transmitted with a length of B*q be u, and encode the data u into a transmission codeword with a length of N. in The encoding method for representing an N-dimensional binary space includes the following steps:

[0042] (1.1) Input the data u of length B*q into the sparsifier and map it to a data of length B*q. sequence The specific steps are as follows:

[0043] (1.1.1) Divide the data u to be transmitted, which has a length of B*q, into B groups of equal length, u = (u... (0) ,u (1) ,…,u (B-1) Each group consists of q characters. Frequency statistics are performed on these groups to obtain the total number of types m and the frequency distribution results.

[0044] (1.1.2) Construct a code table using the statistical frequency distribution results, with each group u (i) Use binary codewords This indicates that 0 ≤ i ≤ B-1.

[0045] (1.1.2.1) Generate m elements of length... Non-repeating binary codeword p j Construct a codebook, where j = 0, 1, ..., m-1 are type indices.

[0046] (1.1.2.2) Map characters according to their frequency of occurrence, so that characters with higher frequency correspond to binary codewords with lower Hamming weight, and characters with lower frequency correspond to binary codewords with higher Hamming weight.

[0047] (1.1.3) The code word p (i) , 0≤i≤B-1, arranged in order to form a mapping sequence

[0048] (1.2) Interleave the sequence v and divide it into L groups according to each group of k bits (padding with zeros if necessary) to obtain a sequence of length Lk. v (Put zeros in if necessary) and calculate its sparsity. Among them W H ( v )yes v The weight of Hamming.

[0049] (1.3) Process the sequence in an appropriate manner v Single-channel or hierarchical multi-channel coding (using block Markov superposition transmission coding to achieve channel coding / joint source channel coding) yields a transmission codeword c of length N.

[0050] (2) At the receiving end, for a received sequence y of length N, estimate the data of length B*q. Its decoding method includes the following steps:

[0051] (2.1) A sequence of length Lk is obtained by single-channel or multi-channel decoding of a received sequence y of length N.

[0052] (2.2) Sequence After deinterleaving in step (1.3) (and removing padding zeros), the length is obtained as follows: sequence

[0053] (2.3) Change the sequence Divided into groups B of equal length The length of each group is Will Input the desparser and demap it according to the code table constructed in step (1.2) to obtain an estimate of the original data of length B*q.

[0054] The data in Example 1 is... Figure 2 The pixel sequence u of the displayed Rubik's Cube image has a length of 18,874,368 characters. Grouped into sets of 96 characters each, there are a total of 105,755 types. Each type has a codeword length of 17 bits, constructing a sparse code table of size 105,755 × 17. After sparse mapping, a sequence v of length 3,342,336 is obtained. Sequence v is padded with zeros and interleaved to obtain a sequence of length 3,344,000. v The sparsity was statistically determined to be θ = 0.284. For v BMST encoding is performed, generating subcodewords of length n = 2000 every k = 2000 bits, with a memory length of m = 2, ultimately resulting in a codeword sequence c of length 3348000. This codeword sequence c is then transmitted through an additive white Gaussian noise (AWGN) channel via binary phase-shift keying (BPSK) modulation. At the receiver, a sliding window iterative decoding algorithm is used with a decoding delay d = 9, yielding a sequence estimate of length 3348000. After deinterleaving and removing redundant zeros, the final sequence is 3342336. The estimate of the original data is obtained after desparse mapping. Image restoration. This embodiment 1 uses the image data restoration performance achieved by encoding and transmitting image data using Huffman coding concatenated with BMST coding as a comparison object. Huffman coding yields a sequence v of length 2513626. After padding and interleaving sequence v, a sequence of length 2514688 is obtained.v ,sequence v BMST encoding is performed, generating subcodewords of length n = 2000 for every k = 1504 bits, with a memory length of m = 2, ultimately resulting in a codeword sequence c of length 3348000. From Figure 3 and Figure 4 In comparison, the image recovered by the Sparsifier-BMST scheme is more readable when SNR = 7dB. The simulated peak signal-to-noise ratio (PSNR) results are shown in Table 1. It can be seen that in regions with low PSNR, the proposed scheme has higher image quality than the Huffman-BMST scheme. The coding method used in this embodiment 1 for data compression transmission allocates binary codewords of different Hamming weights based on the frequency of character group occurrences. BMST coding is used to achieve joint source-channel coding to improve transmission efficiency and combat transmission noise. The method is simple and easy to implement. Compared to the traditional Huffman coding compression-concatenated channel coding method, it can avoid error propagation and ensure data quality.

[0055] Table 1. Performance comparison of Sparsifier-BMST and Huffman-BMST schemes under different SNRs.

[0056] SNR 6dB 7dB 8dB 9dB Sparsifier-BMST 26.41dB 36.71dB 47.90dB 61.61dB Huffman-BMST 11.1dB 11.52dB 10.98dB 13.68dB

[0057] Example 2

[0058] This embodiment 2 provides an encoding method for data compression and transmission, where the data is... Figure 5 The displayed fireworks image pixel sequence u has a length of 88,510,445 characters. Grouped into sets of 96 characters each, there are a total of 626,390 types. Each type has a codeword length of 20 bits, constructing a sparse code table of size 626,390 × 20. After sparse mapping, a sequence v of length 18,439,680 is obtained. Sequence v is padded with zeros and interleaved to obtain a sequence of length 18,439,985. v The sparsity was statistically determined to be θ = 0.31827. For v BMST encoding is performed, generating sub-codewords of length n = 2005 every k = 2005, with a memory length of m = 2, ultimately resulting in a codeword sequence c of length 18443995. Codeword sequence c is transmitted through an AWGN channel via BPSK modulation. At the receiver, a sliding window iterative decoding algorithm is used with a decoding delay d = 10, yielding a sequence estimate of length 18439985. After deinterleaving and removing redundant zeros, a sequence of length 18,439,680 is obtained. The estimate of the original data is obtained after desparse mapping. Image restoration. In this embodiment 3, the image data restoration performance using Huffman coding concatenated with BMST coding is used as a comparison object. Huffman coding yields a sequence v of length 15165098. After zero-padding and interleaving of sequence v, a sequence of length 15165853 is obtained. v For the sequence v Perform BMST encoding, generating subcodewords of length n = 2005 for every k = 1649, and memorizing them with a length m = 2, resulting in a codeword sequence c of length 18443995. From Figure 6 and Figure 7 In comparison, the image recovered by the Sparsifier-BMST scheme is more readable when SNR = 7dB. The simulated PSNR results are shown in Table 3. It can be seen that in areas with low signal-to-noise ratio, the proposed scheme has higher image quality than the Huffman-BMST scheme. The coding scheme used in this embodiment 2 for data compression transmission allocates binary codewords of different Hamming weights based on the frequency of character group occurrences. BMST coding is used to implement joint source-channel coding to improve transmission efficiency and combat transmission noise. The method is simple and easy to implement. Compared to the traditional Huffman coding compression-concatenated channel coding method, it can avoid error propagation and ensure data quality.

[0059] Table 3 Performance comparison of Sparsifier-BMST and Huffman-BMST schemes under different SNRs

[0060] SNR 6dB 7dB 8dB 9dB Sparsifier-BMST 17.51dB 40.86dB 52.95dB 63.61dB Huffman-BMST 15.70dB 15.66dB 16.24dB 17.16dB

[0061] Example 3

[0062] This embodiment 3 presents an encoding / decoding method for data compression and transmission. Consider a data sequence u of length 1,000,000, composed of four symbols: A, B, C, and D, with corresponding probabilities {0.1081, 0.3244, 0.5405, 0.0270}, and a source entropy of 1.5067. Statistics are performed grouped by single characters, resulting in four types. Each type has a codeword length of 2 bits, constructing a sparse code table of size 4×2, as follows:

[0063] symbol probability Typing A 0.1081 10 B 0.3244 01 C 0.5405 00 D 0.0270 11

[0064] After the code table is sparsely mapped, a sequence v of length 2,000,000 is obtained. It can be encoded using single-path coding or multi-path coding after hierarchical processing. (1) When using single-path coding, the bit sparsity of the sequence v is θ = 0.2432. After interleaving, a sequence of length 2,000,000 bits is obtained.v .right v BMST encoding is performed, generating subcodewords of length n = 1700 every k = 1000 bits, with a memory length of m = 8, ultimately resulting in a codeword sequence c of length 1713600, with a code rate of 1.7136. At the decoding end, a sliding window iterative decoding algorithm is used, with a decoding delay d = 16, yielding a sequence estimate of length 2000000. After deinterleaving, a sequence of length 2,000,000 is finally obtained. The estimate of the original data is obtained after desparse mapping. No errors occurred. (2) When using multi-way coding after grading, the sequence v is divided into two paths according to the number of codeword bits to obtain sequences v1 and v2 with a length of 1,000,000 bits each. The sparsity of v1 is θ1 = 0.8649 and the sparsity of v2 is θ2 = 0.6486. After interleaving, two sequences with a length of 1,000,000 bits each are obtained. v 1 and v 2. Regarding v 1 and v 2. BMST encoding is performed separately. The first path generates sub-codewords of length n = 646 every k = 1000 bits, with a memory length of m = 8, resulting in a codeword sequence c1 of length 651168. The second path generates sub-codewords of length n = 998 every k = 1000 bits, with a memory length of m = 8, resulting in a codeword sequence c2 of length 1005984. Finally, the two paths are merged to obtain a codeword sequence c of length 1657152, with a code rate of 1.6572. Assume that codeword sequence c is transmitted through a noise-free channel. At the decoding end, the received sequence y of length 1657152 is divided into two sequences y1 and y2 of lengths 651168 and 1005984 respectively. Using a sliding window iterative decoding algorithm with conditional probability, and a decoding delay d = 16, sequence estimates of length 1000000 are obtained for each path. and After deinterleaving and merging, a sequence of length 2,000,000 is obtained. The estimate of the original data is obtained after desparse mapping. No errors were found.

[0065] The encoding method used in this embodiment 3 for data compression transmission allocates binary codewords with different Hamming weights based on the frequency of character group occurrence. It can be divided into single-channel or hierarchical multi-channel BMST encoding to achieve compression. The method is simple, easy to implement, and approximates the source entropy.

[0066] It should be noted that, for the sake of simplicity, the aforementioned method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously.

[0067] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0068] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. An encoding and decoding method for data compression and transmission, characterized in that, Includes the following steps: (1) At the transmitting end, let the data to be transmitted with a length of B*q be u, and encode the data u into a transmission codeword with a length of N. in To represent an N-dimensional binary space, the encoding method includes the following steps: (1.1) Input the data u of length B*q into the sparsifier and map it to a length of... sequence The specific steps are as follows: (1.1.1) Divide the data u to be transmitted, which has a length of B*q, into B groups of equal length, u = (u... (0) ,u (1) ,…,u (B-1) Each group consists of q characters. Frequency statistics are performed on each group to obtain the total number of types m and the frequency distribution results. (1.1.2) Construct a sparse code table using the statistical frequency distribution results, for each group u (i) Use binary codewords This means that 0 ≤ i ≤ B-1; (1.1.3) Convert the binary codeword p (i) Arranged in chronological order into a mapping sequence (1.2) Interleave the sequence v and divide it into L groups according to each group of k bits, to obtain a sequence of length Lk. v And its sparsity is statistically obtained. Among them W H ( v )yes v The weight of Hamming; (1.3) For the sequence v After single-channel or hierarchical multi-channel coding, a transmission codeword sequence c of length N is obtained; (2) At the receiving end, for a received sequence y of length N, estimate the data of length B*q. The decoding method includes the following steps: (2.1) A sequence of length Lk is obtained by single-channel or multi-channel decoding of a received sequence y of length N. (2.2) Sequence After passing through the deinterleaver in step (1.3), a length of [length missing] is obtained. sequence (2.3) Change the sequence Divided into groups B of equal length The length of each group is Will Input the desparser and demap it according to the sparse code table constructed in step (1.2) to obtain an estimate of the original data of length B*q.

2. The encoding and decoding method for data compression and transmission according to claim 1, characterized in that, In step (1.1.2), the specific implementation method is as follows: The length of the binary codeword is determined based on the total number of types m counted in step (1.1.1). The binary codewords are reordered according to Hamming weight from smallest to largest, and finally correspond one-to-one with the character group type sorted by frequency of occurrence from largest to smallest.

3. The encoding and decoding method for data compression and transmission according to claim 2, characterized in that, Step (1.1.2) is as follows: (1.1.2.1) Generate m elements of length... Non-repeating binary codeword p j Construct a sparse code table, where j = 0, 1, ..., m-1 is the type index; (1.1.2.2) Map characters according to their frequency of occurrence, so that characters with higher frequency correspond to binary codewords with lower Hamming weight, and characters with lower frequency correspond to binary codewords with higher Hamming weight.

4. The encoding and decoding method for data compression and transmission according to claim 1, characterized in that, In step (1.3), sequence v is encoded using single-path or hierarchical multi-path encoding, specifically as follows: Single-channel coding or multi-channel coding after data is classified can be used. Each channel can be channel coded or joint source channel coded using the grouped Markov superposition transmission coding method.

5. The encoding and decoding method for data compression and transmission according to claim 1, characterized in that, In step (2.1), the single-channel or multi-channel channel decoding or joint source-channel decoding algorithm corresponding to step (1.3) is adopted.

Images