Two-stage pulse data lossless coding method and device based on lossy coding and residual coding

Abstract

The invention discloses a two-stage pulse data lossless coding method and device based on lossy coding and residual coding. Through a deep neural network, the preprocessed pulse data are used as input, probability distribution of latent variables and residual errors is obtained through two stages of lossy coding and residual error coding, and therefore corresponding code stream files are output, and the purpose of lossless compression of the pulse data is achieved. The existing coding model is not suitable for processing pulse data or is difficult to realize lossless coding; according to the method, a first pulse data lossless coding model is adopted, a bit splicing data preprocessing mode is adopted, so that the neural network is easier to mine data correlation, and a two-stage framework of lossy coding and residual coding is applied to ensure lossless coding. On the basis of the mode, the neural network coding model can be integrally optimized by utilizing an end-to-end training method, so that better pulse data lossless coding performance is obtained.

Description

Technical Field

[0001] The present invention belongs to the field of image coding, and mainly relates to the processing of pulse data and an end-to-end neural network image coding method. It can be used for lossless coding of pulse data. Background Art

[0002] With the increasing demand for high-quality image data, the importance of image compression has gradually emerged. Although lossy compression can effectively reduce the overhead of digital images during storage and transmission, for some applications with strict requirements for image fidelity, such as medical imaging or scientific images, the most reliable option is still lossless image compression. Therefore, lossless compression has good development potential and has broad application prospects especially in fields with high requirements for image quality.

[0003] In the context of the rapid development of deep learning, neural networks have demonstrated powerful capabilities in many image processing fields. Among them, the end-to-end neural network image coding method has also been widely and deeply studied. End-to-end image compression uses a deep neural network to transform the image in the pixel domain, encodes it through quantization and an entropy model, and decodes it through the entropy model and the deep neural network. During training, the rate-distortion function is used for optimization, achieving results equivalent to or even better than traditional encoders.

[0004] Traditional cameras have difficulties in processing ultra-high-speed dynamic scene imaging based on the exposure method. On the one hand, within the exposure time of the shutter, fast-moving targets may cause motion blur; on the other hand, due to the synchronous imaging principle, the imaging facility cannot record the information between two consecutive exposures. Compared with traditional exposure-based cameras, pulse cameras mimic the working principle of the fovea of the retina and generate pulses when the accumulated light intensity reaches the assigned threshold by continuously accumulating light intensity. Such pulse cameras show significant advantages in capturing fast-moving scenes and can record the originally lost continuous time information. However, their extremely high time resolution brings a huge amount of data, making the transmission or storage of the pulse camera output data (i.e., pulse data) extremely difficult. At the same time, a lossless coding method is needed to ensure the fidelity of the continuous time information recorded by the pulse camera. Summary of the Invention

[0005] The present invention proposes a two-stage lossless coding method and device for pulse data based on lossy coding and residual coding, effectively solving the above problems. Different from traditional end-to-end image compression models, the present invention preprocesses pulse data for input into the model through bit splicing, and through a lossy coding module and a residual coding module, obtains relatively good coding performance while ensuring the losslessness of coding.

[0006] The technical solution adopted by the present invention is as follows:

[0007] A two-stage lossless coding method for pulse data based on lossy coding and residual coding, comprising the following steps:

[0008] Perform bit splicing on the original pulse data;

[0009] The data after bit splicing is successively passed through a lossy coding module and a residual coding module for lossy coding and residual coding;

[0010] Perform bit splitting on the data after lossy coding and residual coding to obtain lossless pulse data.

[0011] Further, the bit splicing includes:

[0012] Obtain the tensor corresponding to the original pulse data;

[0013] Divide the tensor corresponding to the original pulse data along the time axis, each sub-tensor block contains 8 pulse frames on the time axis, and the sub-tensor blocks with less than 8 frames are padded;

[0014] Splice the 8 pulse frames in each sub-tensor block in the order of the time axis, and the 8-bit positions at each pixel form an unsigned integer ranging from 0 to 255, making the entire pulse block into an 8-bit grayscale image.

[0015] Further, the lossy coding module adopts a hyperprior variational autoencoder structure, including a latent variable encoder, a latent variable decoder, a hyperprior encoder and a hyperprior decoder; the entropy model of the latent variable encoder adopts a conditional Gaussian entropy model, the entropy model of the hyperprior encoder adopts a fully factorized entropy model, and the coding algorithm adopted is arithmetic coding.

[0016] Further, the lossy coding module performs lossy coding using the following steps:

[0017] Send the input tensor x into the latent variable encoder for transformation, and send the result of the transformation into the hyperprior encoder to obtain a hyperprior variable;

[0018] Quantize the obtained hyperprior variable, and perform arithmetic coding and arithmetic decoding on the hyperprior variable using the fully factorized entropy model;

[0019] The hyperprior variable after arithmetic decoding passes through the hyperprior decoder to obtain the Gaussian parameters required by the entropy model of the latent variable encoder, and uses the Gaussian parameters to guide the arithmetic coding and arithmetic decoding of the latent variable using the conditional Gaussian entropy model;

[0020] The latent variable after arithmetic decoding passes through the latent variable decoder to obtain the lossy reconstruction tensor of the input tensor

[0021] Further, the residual coding module includes masked convolution and a residual entropy model. The residual entropy model adopts a hybrid logic entropy model, and the coding algorithm used is arithmetic coding.

[0022] Further, the residual coding module performs residual coding using the following steps:

[0023] Using the original input tensor x and the lossy reconstruction tensor to obtain the input residual r;

[0024] Performing lossy reconstruction to obtain the auxiliary information u of the lossy reconstruction through convolution operation;

[0025] Using masked convolution to obtain the context information C of the residual r r ;

[0026] Concatenating the auxiliary information u of the lossy reconstruction and the context information C of the residual r and passing through the entropy model parameter estimation layer to obtain the corresponding entropy model parameters;

[0027] Using the entropy model parameters to guide the hybrid logic entropy model to perform arithmetic coding and arithmetic decoding on the residual.

[0028] Further, the bit splitting includes:

[0029] Obtaining the tensor y corresponding to the decoding results output by the lossy coding module and the residual coding module, with the dimension represented as T×H×W;

[0030] Separately taking out each slice on the time axis in the tensor y, denoted as

[0031] For the value at each pixel (i, j), calculating where % represents the modulo operation and >> represents arithmetic right shift;

[0032] Concatenating y′0, y′1, …, y′7 along the time axis into a tensor y′, thereby splitting into a tensor of 8×H×W;

[0033] Concatenating all the split tensors along the time axis and removing the redundant all-zero frames according to the number of frames filled in the tail of the compressed file, thus obtaining the lossless pulse data.

[0034] A two-stage pulse data lossless coding device based on lossy coding and residual coding, which includes:

[0035] A bit concatenation module, used for bit concatenation of the original pulse data;

[0036] A lossy coding module, used for sequentially passing the bit-spliced data through lossy coding;

[0037] A residual coding module, used for performing residual coding on the lossy coded data;

[0038] The bit splitting module is used to perform bit splitting on the data after lossy coding and residual coding to obtain lossless pulse data.

[0039] The beneficial effects of the present invention are as follows:

[0040] The present invention implements a two-stage pulse data lossless coding model based on lossy coding and residual coding, which efficiently compresses and encodes the pulse data while ensuring the fidelity of the pulse data information. The present invention uses a data processing method of bit splicing and bit splitting to represent the pulse data in a form similar to a grayscale image, making it easier for neural networks to mine its correlations. The reversibility of this data processing provides a guarantee for the losslessness of the coding. In addition, the present invention adopts a two-stage coding architecture that includes lossy coding and residual coding, and utilizes an end-to-end overall optimization method to achieve superior coding performance while ensuring the losslessness of the coding.

[0041] Most existing coding models focus on processing traditional image and video data and are unsuitable for processing pulse data. Furthermore, some models that can process pulse data struggle to achieve lossless coding. This invention, however, is the first lossless coding model for pulse data. It employs a bit-splitting data preprocessing method to make it easier for neural networks to exploit data correlations, and employs a two-stage architecture of lossy and residual coding to ensure lossless coding. Based on this approach, an end-to-end training method can be used to comprehensively optimize the neural network coding model, resulting in superior pulse data lossless coding performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 Schematic diagram of the lossless coding model framework.

[0043] Figure 2 Schematic diagram of bit splicing and splitting.

[0044] Figure 3 Schematic diagram of lossy coding and residual coding process. DETAILED DESCRIPTION

[0045] In order to further illustrate the technical method of the present invention, the two-stage pulse data lossless coding model based on lossy coding and residual coding of the present invention is further described in detail below with reference to the accompanying drawings and specific examples.

[0046] The present invention designs a two-stage lossless coding method for pulsed data based on lossy coding and residual coding. The overall framework of this method is mainly implemented based on a deep convolutional neural network, as Figure 1 shown. The input of the model proposed by the present invention is the pulsed data after bit splicing preprocessing. The output is the compressed code stream file and the decoded pulsed data.

[0047] Next, the main steps of the method of the present invention will be described.

[0048] I. Preprocessing process:

[0049] Step 1: Collect a natural video dataset, perform frame interpolation on the video sequence, and generate simulated pulsed data through an integrate-and-fire method. These data will constitute the training set for subsequent model training, while the test set consists of real pulsed data. At this time, the pulsed data can be regarded as a three-dimensional tensor containing only 0 and 1 elements.

[0050] Step 2: Divide the pulsed data generated in Step 1 along the time axis. Each sub-tensor block should contain 8 pulsed frames on the time axis, and the sub-tensor blocks with fewer than 8 frames are padded.

[0051] Step 3: Bit-splice the 8 pulsed frames in each tensor block in the order of the time axis. In this way, 8 bit positions at each pixel form an unsigned integer ranging from 0 to 255, making the entire pulsed block an 8-bit grayscale image.

[0052] Step 4: For the training set, randomly shuffle the pulsed blocks processed in Step 3 and then input them into the coding model. For the test set, retain the temporal order of the pulsed data, input them into the coding model in sequence, and output the results one by one.

[0053] Step 5: The pulsed blocks pass through the lossless coding model to obtain the compressed code stream file and the lossless decoding result.

[0054] Step 6: For the lossless decoding result output by the model, represent the value of each pixel in the pulsed block in binary, and decompose each bit position into a separate element to obtain 8 pulsed frames containing only 0 and 1 elements.

[0055] Step 7: Combine all the pulsed frames in order to form the original three-dimensional pulsed tensor, and remove the padded part in Step 2 to obtain the lossless pulsed input data in Step 1.

[0056] II. Training process

[0057] Step 1: Feed the preprocessed training set pulsed blocks into the lossless coding model. At this time, the input can be regarded as a single-channel grayscale image.

[0058] Step 2: The input single-channel grayscale image first passes through a lossy coding module. The lossy coding module can adopt a hyperprior variational autoencoder structure to output the lossy reconstruction of the given image and the coding rate of the corresponding latent variable.

[0059] Step 3: Take the coding rate in Step 2 and the mean square error of the lossy reconstruction as part of the final loss function.

[0060] Step 4: Calculate the residual between the lossy reconstruction and the true input value, and use the context information of the lossy reconstruction and the residual to obtain the parameters of the entropy model of the residual coding module.

[0061] Step 5: Use the entropy model of the residual coding module to obtain the estimated residual coding rate, and take it as part of the loss function.

[0062] Step 6: Weight the losses in Step 2 and Step 5 to obtain the final rate-distortion loss function. Through optimization iteration, backpropagate this loss to each layer of the neural network to update the weights of each layer.

[0063] Step 7: Repeat Steps 1 - Step 6 until the rate-distortion loss function of the neural network converges.

[0064] III. Encoding and Decoding Process

[0065] Step 1: During encoding, send the preprocessed true pulse data block of the test set into the lossless coding model. At this time, the input can be regarded as a single-channel grayscale image.

[0066] Step 2: The input single-channel grayscale image first passes through a lossy coding module. The lossy coding module can adopt a hyperprior variational autoencoder structure to output the lossy reconstruction of the given image and the probability distribution of the corresponding latent variable.

[0067] Step 3: Calculate the residual between the lossy reconstruction and the true input value, and use the context information of the lossy reconstruction and the residual to obtain the parameters of the entropy model of the residual coding module.

[0068] Step 4: Use the entropy model of the residual coding module to obtain the probability distribution parameters of the residual.

[0069] Step 5: According to the obtained probability distribution, use arithmetic coding to perform entropy coding on the latent variable and the residual, and write their corresponding bitstreams into the bitstream file.

[0070] Step 6: Repeat Steps 1 - Step 5 until all pulse blocks of the corresponding pulse data sequence are completely processed, and write the possible number of padding frames into the bitstream file to facilitate removing the corresponding part during decoding.

[0071] Step 7: During decoding, read the bitstream file and load the decoding part of the lossy coding module.

[0072] Step 8: The residual can be directly obtained through the arithmetic decoder, and the lossy reconstruction can be obtained by outputting the latent variable after arithmetic decoding through the decoding part of the lossy coding module.

[0073] Step 9: Add the residual to the lossy reconstruction to obtain the lossless grayscale image input. After corresponding bit splitting processing, the lossless pulse reconstruction data can be obtained.

[0074] The following provides an example. This example will focus on elaborating the specific implementation methods of bit concatenation and splitting and the specific process of two-stage lossless coding in this technical method. The processing method of bit transformation is as Figure 2 shown, and the implementation of the lossy coding module and the residual coding module is as Figure 3 shown.

[0075] I. Bit Concatenation and Splitting

[0076] 1. Bit Concatenation:

[0077] Suppose the tensor x corresponding to the original pulse data has been obtained, with the total number of frames being T, the height being H, and the width being W (abbreviated as T×H×W, and similar notations will be used later).

[0078] Step 1: Divide the original pulse tensor x along the time axis into sub-blocks of 8×H×W, denoted as x'. In this way, such sub-blocks can be obtained ( represents rounding up). If the last sub-block has less than 8 pulse frames, it is padded with all-zero pulse frames.

[0079] Step 2: Take out the 8 pulse frames of x' separately, denoted as x'0, x'1,..., x'7. For the pixel value at each pixel (i, j), calculate as the result of bit concatenation output, and the value range of its pixel value is from 0 to 255.

[0080] Step 3: Expand the dimension of to 1×H×W and send it into the coding model for subsequent coding. For the original pulse tensor of T×H×W, a total of such

[0081] 2. Bit Splitting:

[0082] Suppose the tensor y corresponding to the decoding result output by the coding model has been obtained, and its dimension is expressed as T×H×W.

[0083] Step 1: Take out each slice on the time axis in the tensor y separately, denoted as

[0084] Step 2: For the value at each pixel (i, j), calculate where % represents the modulo operation and >> represents arithmetic right shift.

[0085] Step 3: Concatenate y′0, y′1, …, y′7 along the time axis into a tensor y′, thus splitting into an 8×H×W tensor.

[0086] Step 4: Repeat Steps 1 - Step 3 until all slices are calculated.

[0087] Step 5: Concatenate all the split tensors along the time axis, and remove the redundant all - zero frames according to the number of frames padded at the end of the compressed file, then a pulse lossless compared with the original input can be obtained.

[0088] II. Two - stage lossless coding

[0089] 1. Lossy coding:

[0090] Assume that the input tensor x after bit concatenation processing has been obtained, with the number of channels being 1, height being H, and width being W (abbreviated as 1×H×W, and similar notations will be used later).

[0091] The overall structure of the lossy coding module is a variational auto - encoder with a hyper - prior, consisting of an encoder and a decoder for the latent variable, as well as an encoder and a decoder for the hyper - prior. The entropy model of the latent variable encoder uses a conditional Gaussian entropy model, the entropy model of the hyper - prior encoder uses a fully factorized entropy model, and the coding algorithm is arithmetic coding. Here, the latent variable refers to the feature variable y obtained after the input tensor x is transformed by the latent variable encoder, and the hyper - prior refers to the high - level feature variable z obtained after the feature variable y corresponding to the latent variable is transformed by the hyper - prior encoder.

[0092] Step 1: Send the input tensor x into the latent variable encoder for transformation, and send the result of the transformation into the hyper - prior encoder to obtain the hyper - prior variable.

[0093] Step 2: Quantize the hyper - prior variable obtained in Step 1, and perform arithmetic coding and arithmetic decoding on the hyper - prior variable using the fully factorized entropy model.

[0094] Step 3: The hyper - prior variable after arithmetic decoding passes through the hyper - prior decoder to obtain the Gaussian parameters required by the entropy model of the latent variable encoder, and use the Gaussian parameters to guide the arithmetic coding and arithmetic decoding of the latent variable using the conditional Gaussian entropy model.

[0095] Step 4: After arithmetic decoding the latent variable, pass it through the latent variable decoder to obtain the lossy reconstruction tensor of the input tensor

[0096] 2. Residual coding:

[0097] Assume that the lossy reconstruction of the input tensor has been obtained The number of channels is 1, the height is H, and the width is W (abbreviated as 1×H×W, and the subsequent expressions are similar)

[0098] The overall structure of the residual coding module consists of masked convolution and a residual entropy model. To more effectively represent the probability distribution of the residuals, the residual entropy model uses a mixture of logistic entropy models, and the coding algorithm is arithmetic coding.

[0099] Step 1: Use the original input tensor x and the lossy reconstruction tensor to obtain the input residual r.

[0100] Step 2: Perform a convolution operation on the lossy reconstruction to obtain the auxiliary information u of the lossy reconstruction.

[0101] Step 3: Use masked convolution to obtain the context information C of the residual r r .

[0102] Step 4: Concatenate the auxiliary information u of the lossy reconstruction and the context information C of the residual r and pass through the entropy model parameter estimation layer to obtain the corresponding entropy model parameters.

[0103] Step 5: Use the entropy model parameters to guide the mixture of logistic entropy model to perform arithmetic coding and arithmetic decoding on the residuals.

[0104] Experimental results and comparison:

[0105] The present invention has been tested for losslessness and compression performance on a standard test set consisting of 20 real pulse data sequences. The experimental results show that the present invention indeed meets the requirements of lossless coding. The present invention is compared with the traditional lossless encoding and decoding method, the context-based adaptive lossless coding method CALIC: the compression ratio of CALIC is 1.662 times (after compression, it is 60.17% of the original size), and the compression ratio of the present invention is 1.725 times (after compression, it is 57.98% of the original size). The method proposed in the present invention exceeds the traditional lossless encoding and decoding methods in terms of compression performance, supporting the effectiveness of the present invention.

[0106] In the present invention, the conditional Gaussian entropy model, the fully factorized entropy model, the mixture of logistic entropy model, the latent variable encoder, the latent variable decoder, the superprior encoder, and the superprior decoder all belong to the prior art and are well-known in the field of compression coding.

[0107] Another embodiment of the present invention provides a two-stage pulse data lossless coding device based on lossy coding and residual coding, which includes:

[0108] A bit splicing module for bit splicing of original pulse data;

[0109] A lossy coding module for successively performing lossy coding on the data after bit splicing;

[0110] A residual coding module for performing residual coding on the data after lossy coding;

[0111] A bit splitting module for bit splitting the data after lossy coding and residual coding to obtain lossless pulse data.

[0112] For the specific implementation process of each module, refer to the description of the method of the present invention above.

[0113] Another embodiment of the present invention provides a computer device (such as a computer, a server, a smart phone, etc.), which includes a memory and a processor. The memory stores a computer program, and the computer program is configured to be executed by the processor. The computer program includes instructions for executing each step in the method of the present invention.

[0114] Another embodiment of the present invention provides a computer-readable storage medium (such as ROM / RAM, a disk, an optical disc). When the computer program stored in the computer-readable storage medium is executed by a computer, each step of the method of the present invention is implemented.

[0115] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalent technologies, the present invention also intends to include these changes and modifications.

Claims

1. A two-stage pulse data lossless coding method based on lossy coding and residual coding, characterized in that: The following steps are involved: Perform bit splicing on the original pulse data; The bit-spliced data is sequentially passed through a lossy coding module and a residual coding module for lossy coding and residual coding; The data after lossy coding and residual coding is bit-splitting to obtain lossless pulse data.

2. The method according to claim 1, characterized in that The bit splicing includes: Get the tensor corresponding to the original pulse data; The tensor corresponding to the original pulse data is divided along the time axis. Each sub-tensor block contains 8 pulse frames on the time axis. The sub-tensor blocks with less than 8 frames are padded. The 8 pulse frames in each sub-tensor block are spliced in time sequence, and the 8 bits at each pixel form an unsigned integer ranging from 0 to 255, making the entire pulse block an 8-bit grayscale image.

3. The method according to claim 1, characterized in that The lossy coding module adopts a super-prior variational autoencoder structure, including a latent variable encoder, a latent variable decoder, a super-prior encoder and a super-prior decoder; the entropy model of the latent variable encoder adopts a conditional Gaussian entropy model, the entropy model of the super-prior encoder adopts a fully factored entropy model, and the coding algorithm adopted is arithmetic coding.

4. The method according to claim 3, characterized in that The lossy encoding module performs lossy encoding by following steps: The input tensor x is sent to the latent variable encoder for transformation, and the transformed result is sent to the super prior encoder to obtain the super prior variable; The obtained super-prior variables are quantified and arithmetic coding and decoding are performed on the super-prior variables using a fully factorized entropy model; After arithmetic decoding, the super prior variables are passed through the super prior decoder to obtain the Gaussian parameters required by the entropy model of the latent variable encoder. The Gaussian parameters are used to guide the conditional Gaussian entropy model to perform arithmetic coding and arithmetic decoding on the latent variables. After arithmetic decoding of the latent variable, the latent variable decoder is used to obtain the lossy reconstruction tensor of the input tensor.

5. The method according to claim 1, wherein The residual coding module includes mask convolution and a residual entropy model. The residual entropy model adopts a hybrid logical entropy model, and the coding algorithm adopted is arithmetic coding.

6. The method according to claim 5, characterized in that The residual coding module performs residual coding by the following steps: Takes the original input tensor x and the lossy reconstruction tensor Get the input residual r; Will be damaged reconstruction The lossy reconstructed auxiliary information u is obtained through convolution operation; Use mask convolution to obtain the context information C of the residual r r ; The lossy reconstructed auxiliary information u and the residual context information C r Splice them together and pass through the entropy model parameter estimation layer to obtain the corresponding entropy model parameters; The entropy model parameters are used to guide the hybrid logistic entropy model to perform arithmetic coding and arithmetic decoding on the residual.

7. The method according to claim 1, characterized in that The bit splitting includes: Get the tensor y corresponding to the decoding results output by the lossy coding module and the residual coding module, with dimensions expressed as T×H×W; Take out each slice on the time axis of the tensor y separately, and record it as for The value of each pixel (i, j) is calculated Among them, % represents the remainder operation, and >> represents the arithmetic right shift; Concatenate y′0, y′1, …, y′7 along the time axis into a tensor y′, thereby Split into 8×H×W tensors; All split tensors are spliced together along the time axis, and the redundant all-0 frames are removed according to the number of frames padded at the end of the compressed file to obtain lossless pulse data.

8. A two-stage pulse data lossless coding device based on lossy coding and residual coding, characterized in that: include: A bit splicing module, used for performing bit splicing on the original pulse data; A lossy coding module, used for sequentially passing the bit-spliced data through lossy coding; A residual coding module, used for performing residual coding on the lossy coded data; The bit splitting module is used to perform bit splitting on the data after lossy coding and residual coding to obtain lossless pulse data.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program comprises instructions for executing the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a computer, the method according to any one of claims 1 to 7 is implemented.

Images