Image coding method and device based on deep learning

Abstract

The invention provides an image coding method and device based on deep learning, and belongs to the field of image coding, and the image coding method comprises the steps: a coding end uses nonlinear transformation to project an original image to a hidden space; the coding end performs probability distribution estimation by using a causal context; the encoding end encodes the hidden variables by using an entropy encoding technology; the decoding end performs probability distribution estimation by using a causal context; the decoding end decodes the hidden variables by using an entropy coding technology; the coding end reconstructs the hidden variables into an original image by using nonlinear transformation; and performing end-to-end reverse gradient propagation optimization on an image coding and decoding process by using a distortion loss function and a causal context adjustment loss function. By using a better causal context to adjust a loss function and a more efficient network structure, better image compression capability and higher rate distortion performance are obtained.

Description

Technical Field

[0001] The present invention relates to the field of image coding, and in particular to an image coding method and device based on deep learning. Background Art

[0002] The growing demand for high-quality, high-resolution contemporary photos has driven a growing demand for advanced image storage and transmission technologies. As a result, lossy image compression technology has developed rapidly in recent years. In parallel with traditional coding techniques (such as JPEG, BPG, WebP, VVC), a series of deep learning-based image compression methods (LIC) have emerged, which achieve high peak signal-to-noise ratio (PSNR) and multi-scale structural similarity (MS-SSIM) while maintaining a fairly fast running speed. Their superior compression effect compared to VVC shows that LIC technology is expected to keep pace with traditional technologies in the near future.

[0003] The deep learning-based lossy image compression method is built on the variational autoencoder (VAE) framework proposed by Ballé et al. The VAE-based LIC framework mainly includes an autoencoder and an entropy model. The autoencoder performs nonlinear transformations between the image space and the latent space; while the entropy model minimizes the encoding length by estimating the probability distribution of the latent variables. Compared with the autoencoder, the entropy model is a unique and important component of LIC, which has a significant impact on the final compression result.

[0004] In the LIC literature, the entropy model usually refers to a parameterized distribution model. In some pioneering work, Ballé et al. established an end-to-end rate-distortion optimization framework and proposed that the average encoding length of the optimized latent variable is given by the Shannon cross-entropy between the actual marginal distribution and the learned entropy model. Since then, many types of entropy models have been studied. One type of research explores advanced network architectures for accurately predicting the distribution of latent representations. At the same time, another type of research studies the entropy model from a more fundamental perspective, namely conditional distribution modeling, in pursuit of a better rate-distortion balance. Conditioning on auxiliary information (also called super-prior) and decoded latent representations (also called causal context) has become a mainstream strategy in modern LIC models.

[0005] Existing works usually train LIC networks through a combination of rate loss and distortion loss. The conditional predictability of the representation is indirectly optimized, and the performance of the entropy model is highly dependent on manually designed causal context models, such as channel grouping or chessboard context models. Fixed context models are difficult to generalize to all types of natural images, and methods for obtaining a more widely applicable context model structure need to be improved. Summary of the invention

[0006] To address this problem, the present invention provides a causal context adjustment loss function (CCA-loss) to advance conditional distribution modeling in entropy models. The CCA loss function is the first attempt to explicitly adjust the causal context so that subsequent representations are more accurately predicted by previously decoded representations. Specifically, the autoregressive entropy model has a multi-stage autoregressive processing capability, which processes the previous stage information to obtain the current stage information by dividing the latent variables into two stages; in particular, considering a two-stage autoregressive context model with a hyper-prior z, the potential representations to be decoded in the first stage and the second stage are represented as y2 and y2, respectively; in addition to minimizing the cross-entropy loss by reducing the bitstream, the present invention introduces an auxiliary entropy model and a customized causal context adjustment loss, which allows y2 to be accurately estimated by y1 and z, while making y2 not accurately estimated by z alone. Therefore, the CCA loss function explicitly guides the encoder to adjust important information to the early stage of the autoregressive entropy model, providing a more reasonable causal context sequence for entropy coding. Since the encoding of the early stage is enhanced by the CCA loss function, and the present invention provides uneven channel dimension scheduling to obtain a better rate-distortion balance. Uneven channel scheduling also helps reduce the computational burden during encoding and decoding, allowing the model to achieve state-of-the-art compression performance in less running time.

[0007] The specific steps of an image encoding method based on deep learning provided by the present invention are as follows:

[0008] Step S1: Input the image to be compressed into the encoder to obtain the image latent variables in the high-dimensional latent space;

[0009] Step S2: input the image latent variables into the super prior encoder and the auxiliary entropy model respectively, and obtain the super prior latent variables and the auxiliary estimated probability respectively;

[0010] Step S3: quantize the super-a priori latent variable to become a super-a priori quantized latent variable, then go through an entropy coding process to become a bit stream, transmit it to the decoder, and then recover it to become a super-a priori quantized latent variable after entropy decoding, and then input it into a super-a priori decoder to obtain a priori information;

[0011] Step S4: the prior information is input into the auxiliary entropy model to help estimate the auxiliary probability, and the input entropy model estimates the probability distribution of the prior information. The entropy model is specifically a causal context entropy model based on an autoregressive framework;

[0012] Step S5: entropy encoding the image latent variables using the probability distribution estimated by the entropy model to obtain a bit stream, which is then restored to the image reconstruction latent variables through entropy decoding after compression transmission;

[0013] Step S6: Finally, the image reconstruction latent variables are restored through the decoder to obtain the reconstructed image;

[0014] During the training process, the model is optimized using the following loss functions: the rate-distortion loss function optimized by end-to-end backward gradient propagation adjusts the compression quality and compression size ratio; the causal context adjustment loss function adjusts the distribution of information in the latent variables estimated by the entropy model;

[0015] The following loss function is also introduced to enhance the predictability of the causal context entropy model based on the autoregressive framework:

[0016]

[0017] Among them, CCA-loss represents the CCA loss function, Represents the i-th part of the quantized latent variable. After the image latent variable is quantized, it is divided into n latent variable parts in the channel dimension. represents the decoded part of the i-th part of the image quantization latent variable, Indicates excluding the decoded part of the previous part; represents the super-prior quantized latent variable after the super-prior latent variable is quantized; and are the estimated distributions of the auxiliary entropy model and the entropy model, respectively.

[0018] The present invention also provides an electronic device, comprising:

[0019] memory; processor; and computer program;

[0020] The computer program is stored in the memory and is configured to be executed by the processor to implement the method as described above.

[0021] The present invention also provides a computer-readable storage device, characterized in that a computer program is stored thereon; the computer program is executed by a processor to implement the method as described above.

[0022] The advantages of the technical solution provided by this application are:

[0023] The present invention introduces causal context adjustment loss to explicitly adjust the causal context information, forcing the network to encode important information as early as possible, thereby improving the autoregressive prediction accuracy of the entropy model. The present invention adopts uneven autoregressive causal context scheduling and convolutional autoencoder architecture to provide an efficient compression network that is easy to implement on modern deep learning platforms. It has strong practicality and high performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present invention or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0025] Figure 1 It is a flowchart of an image encoding method based on deep learning provided by an embodiment of the present invention;

[0026] Figure 2 It is a schematic diagram of an entropy model network structure of an image coding method based on deep learning provided by an embodiment of the present invention;

[0027] Figure 3 It is a schematic diagram of the autoencoder network structure of an image encoding method based on deep learning provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments; based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0029] Embodiment 1

[0030] This embodiment provides an image encoding method based on deep learning, including:

[0031] The encoder module is a nonlinear transformation structure used for mutual transformation between images and latent variables;

[0032] The entropy model module is used for the causal context and nonlinear transformation structure of entropy coding;

[0033] Rate-distortion loss and causal context adjustment loss for end-to-end backward gradient propagation optimization.

[0034] The specific steps of the deep learning-based image coding method provided in this embodiment are as follows:

[0035] Step S1: Input the image to be compressed into the encoder to obtain the image latent variables in the high-dimensional latent space;

[0036] Step S2: input the image latent variables into the super prior encoder and the auxiliary entropy model respectively, and obtain the super prior latent variables and the auxiliary estimated probability respectively;

[0037] Step S3: quantizing the super a priori latent variable to become a super a priori quantized latent variable, and then undergoing an entropy coding process to become a bit stream, which is transmitted to the decoder and restored to the super a priori quantized latent variable after entropy decoding, and then input into the super a priori decoder to obtain the a priori information;

[0038] Step S4: the prior information is input into the auxiliary entropy model to help estimate the auxiliary probability, and the input entropy model estimates the probability distribution of the prior information. The entropy model is specifically a causal context entropy model based on an autoregressive framework;

[0039] Step S5: entropy encoding the image latent variables using the probability distribution estimated by the entropy model to obtain a bit stream, which is then restored to the image reconstruction latent variables through entropy decoding after compression transmission;

[0040] Step S6: Finally, the image reconstruction latent variables are restored through the decoder to obtain the reconstructed image.

[0041] Specifically, this embodiment relates to an encoder, a decoder, a super a priori encoder, a super a priori decoder, an entropy model, and an auxiliary entropy model; Figure 3 As shown, the encoder specifically includes a downsampling module, a residual module, and a module without non-linear activation function; further, after the input image passes through the downsampling module, it is processed three times in sequence by the residual module, four stacked modules without non-linear activation function, and the downsampling module to obtain the image latent variable; further, the residual module is composed of three stacked residual blocks, and the residual block includes: a convolution layer and its non-linear activation function included by an identity mapping.

[0042] The decoder specifically includes an upsampling module, a residual module, and a non-linear activation function module; further, after the image reconstruction latent variable passes through the upsampling module, it passes through the non-linear activation function module, the residual module, and the upsampling module for three times to obtain a reconstructed image. In the up / down sampling module, a convolution kernel is used for up / down sampling operations.

[0043] The super prior encoder specifically includes a downsampling module and a Gaussian error linear unit; further, the latent variable is sequentially passed through the downsampling module, the Gaussian error linear unit, the downsampling module, the Gaussian error linear unit, and the downsampling module to obtain a priori variables, and the Gaussian error linear unit is an activation function with Gaussian error in the negative domain;

[0044] The super-prior decoder specifically includes an upsampling module and a Gaussian error linear unit; further, after the super-prior quantized latent variables pass through the super-prior entropy model (i.e., entropy coding), they pass through the upsampling module, the Gaussian error linear unit, the upsampling module, the Gaussian error linear unit, and the upsampling module in sequence to obtain prior information.

[0045] Specifically, Figure 2As shown, the entropy model of this embodiment is specifically a causal context entropy model based on an autoregressive framework, and the structure specifically includes a non-linear activation function module, a Gaussian error linear unit, and a convolution layer; further, the prior information is input into the entropy model, and the entropy estimation probability distribution is obtained in sequence through 4 stacked non-linear activation function modules, convolution layers, Gaussian error linear units, convolution layers, Gaussian error linear units, and convolution layers, and finally the supplementary residual obtained by the residual prediction module is added to restore the error caused by quantization. The latent variable residual prediction module includes: 3 convolution layers and their nonlinear activation functions.

[0046] Furthermore, the structure of the module without nonlinear activation function is that the input data sequentially passes through layer normalization, convolution layer, deep convolution layer, simple gating, simple channel attention, convolution layer and layer normalization, convolution layer, simple gating, and convolution layer included in the identity mapping. Furthermore, layer normalization normalizes the parameters of the entire layer, deep convolution performs convolution operations on each channel of the variable, and simple gating divides the variable into two parts by the channel dimension and then multiplies them.

[0047] The training method provided in this example before using it for reasoning includes: rate-distortion loss function and causal context adjustment loss function for end-to-end reverse gradient propagation optimization; specifically includes: adjusting compression quality and compression size ratio through rate-distortion loss function; adjusting the distribution of information in latent variables estimated by entropy model through causal context adjustment loss function acting on auxiliary entropy model and probability distribution estimated by entropy model.

[0048] The expression of the rate-distortion loss function for end-to-end backward gradient propagation optimization is:

[0049]

[0050] The expression of the causal context adjustment loss function is:

[0051]

[0052] in, represents the rate-distortion loss function, represents the causal context adjusted loss function;

[0053] represents the reconstructed image, and x represents the original input image; represents the image quantization latent variable obtained through entropy model estimation and entropy coding, represents the auxiliary latent variable used to calculate the loss function for auxiliary entropy model estimation, represents the hyper-prior quantized latent variable, represents the probability distribution of the corresponding variable; represents the cross entropy of its estimated true probability distribution.

[0054] The main path of this embodiment includes: firstly, the original image is nonlinearly transformed into an image latent variable y through an encoder, and the image latent variable is quantized into an image quantized latent variable After that, it is divided into multiple hidden variable parts in the channel dimension Then, the transmitted image quantization latent variables are autoregressively decoded through the estimated probability distribution of the entropy model. Autoregressive decoding is a part-by-part decoding process using causal context, and the causal context includes the decoded parts. Specifically, the autoregressive decoding is as follows: consider a multi-part latent variable decoding process, first decode the first part, then decode the second part by using the decoded first part... and so on to complete the decoding of all parts. Finally, the decoded quantization latent variables are decoded through a decoding nonlinear transformation. Reconstruct the original image.

[0055] In addition, this embodiment uses CCA loss to explicitly encourage encoding important information of the image into the earlier causal context to enhance the predictability of the autoregressive entropy model. Specifically, in addition to the original autoregressive entropy model, the latent variables are quantized by the decoded part and the hyper-prior Estimate the hidden variables at the current stage: In addition, the present invention introduces an auxiliary entropy model, which converts the hyper-prior latent variable and the decoded part excluding the previous part as input: Get auxiliary latent variables in Indicates the decoded part, is the latent variable part after quantization, The representation entropy model F uses the decoded part and the hyper-prior quantized latent variables to obtain the latent variable part of the current stage; Indicates excluding the decoded part of the previous part. It means that the auxiliary latent variables are obtained by excluding the decoded part of the previous part and the super prior latent variables.

[0056] By introducing the auxiliary entropy model, a two-stage CCA loss can be defined as follows:

[0057]

[0058] Measures the information gain between latent variables, that is, the entropy gap between the auxiliary latent variables and the quantitative latent variables, where and are the estimated distributions of the auxiliary entropy model and the entropy model respectively. The auxiliary entropy model and the entropy model are parameterized by two networks, namely the auxiliary entropy model and entropy model It is worth noting that the auxiliary entropy model is only introduced in the training phase to better optimize the causal context; in the testing phase, the model still uses the entropy model To compress latent variables, the CCA loss function does not introduce additional computational burden for image compression.

[0059] The schematic diagram of the CCA loss function provided by the present invention can be found in Figure 1 , the information entropy of the latent variable obtained by inputting the entropy model into the super prior latent variable prediction and the auxiliary information entropy of the auxiliary latent variable obtained by inputting the auxiliary entropy model into the super prior latent variable prediction are substituted into the CCA loss function for gradient descent optimization. For autoregressive models with more than two stages, the equation can be easily expanded. The multi-stage CCA loss can be defined as follows:

[0060]

[0061] Among them, CCA-loss represents the CCA loss function, Represents the i-th part of the image quantization latent variable. The quantization latent variable is divided into n latent variable parts in the channel dimension. represents the decoded part of the i-th part of the quantized latent variable; and are the estimated distributions of the auxiliary entropy model and the entropy model, respectively. Through the CCA loss provided by the present invention, the image compression model of this embodiment can automatically adjust the causal context, thereby improving the rate-distortion performance. This embodiment uses a network based on a convolutional neural network (CNN). Due to the use of a convolutional structure, the operation efficiency of this embodiment is high and it is easy to be deployed on a lightweight device. The detailed structure of the autoencoder of this embodiment can be found in Figure 3 This embodiment adopts an autoregressive architecture with non-uniform channel grouping to design the entropy model. The detailed network architecture of the entropy model and auxiliary entropy model of this embodiment can be found in Figure 2 Found in.

[0062] In this embodiment, a causal context adjustment loss (CCA-loss) is applied on the CNN-based model to achieve advanced rate-distortion performance. Thanks to the advantages of convolutional neural networks, uneven grouping schemes, and the proposed CCA-loss training method, this embodiment maintains a good balance between compression delay and rate-distortion performance.

[0063] In addition, an embodiment of the present application further provides an electronic device, including:

[0064] memory; processor; and computer program;

[0065] The computer program is stored in the memory and is configured to be executed by the processor to implement the method as described above.

[0066] In addition, an embodiment of the present application further provides a computer-readable storage device on which a computer program is stored; the computer program is executed by a processor to implement the above-mentioned method.

[0067] In the present application, the method and the device are based on the same inventive concept. Since the principles of solving problems by the method and the device are similar, the implementation of the method and the device can refer to each other, and the repeated parts will not be repeated.

[0068] The memory may be a computer-readable storage medium, which may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0069] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. An image coding method based on deep learning, characterized in that: The following steps are involved: Step S1: Input the image to be compressed into the encoder to obtain the image latent variables in the high-dimensional latent space; Step S2: input the image latent variables into the super prior encoder and the auxiliary entropy model respectively, and obtain the super prior latent variables and the auxiliary estimated probability respectively; Step S3: quantize the super-a priori latent variable to become a super-a priori quantized latent variable, then go through an entropy coding process to become a bit stream, transmit it to the decoder, and then recover it to become a super-a priori quantized latent variable after entropy decoding, and then input it into a super-a priori decoder to obtain a priori information; Step S4: the prior information is input into the auxiliary entropy model to help estimate the auxiliary probability, and the input entropy model estimates the probability distribution of the prior information. The entropy model is specifically a causal context entropy model based on an autoregressive framework; Step S5: entropy encoding the image latent variables using the probability distribution estimated by the entropy model to obtain a bit stream, which is then restored to the image reconstruction latent variables through entropy decoding after compression transmission; Step S6: Finally, the image reconstruction latent variables are restored through the decoder to obtain the reconstructed image; During the training process, the model is optimized using the following loss functions: the rate-distortion loss function optimized by end-to-end backward gradient propagation adjusts the compression quality and compression size ratio; the causal context adjustment loss function adjusts the distribution of information in the latent variables estimated by the entropy model; The following loss function is also introduced to enhance the predictability of the causal context entropy model based on the autoregressive framework: Among them, CCA-loss represents the CCA loss function, Represents the i-th part of the quantized latent variable. After the image latent variable is quantized, it is divided into n latent variable parts in the channel dimension. represents the decoded part of the i-th part of the image quantization latent variable, Indicates excluding the decoded part of the previous part; represents the super-prior quantized latent variable after the super-prior latent variable is quantized; and are the estimated distributions of the auxiliary entropy model and the entropy model, respectively.

2. The image coding method based on deep learning according to claim 1, characterized in that: The image quantization latent variable is divided into n latent variable parts in the channel dimension Then, the transmitted latent variable part is autoregressively decoded by using the estimated probability distribution of the entropy model. The autoregressive decoding is a part-by-part decoding process using the causal context, and the causal context includes the decoded part. Specifically, the autoregressive decoding is as follows: consider a decoding process of n latent variable parts, first decode the first part, then decode the second part by using the decoded first part... and so on to complete the decoding of all parts; The auxiliary entropy model is only introduced during training. During testing and use, only the entropy model is used to compress latent variables.

3. The image coding method based on deep learning according to claim 1, characterized in that: The expression of the rate-distortion loss function of the end-to-end reverse gradient propagation optimization is: The expression of the causal context adjustment loss function is: in, represents the rate-distortion loss function, represents the causal context adjusted loss function; in, represents the reconstructed image, and x represents the original input image; represents the reconstructed image quantization latent variable, represents the auxiliary latent variable used to calculate the loss function for auxiliary entropy model estimation, represents the probability distribution of the corresponding variable; represents the cross entropy of its estimated true probability distribution.

4. The image coding method based on deep learning according to claim 3, characterized in that: The encoder specifically includes a downsampling module, a residual module, and a non-linear activation function module; further, after the input image passes through the downsampling module, it passes through the residual module, 4 stacked non-linear activation function modules, and the downsampling module to process the sequence three times to obtain the image latent variable; The decoder specifically includes an upsampling module, a residual module, and a non-linear activation function module; further, after the reconstructed image quantization latent variable passes through the upsampling module, it passes through the non-linear activation function module, the residual module, and the upsampling module for three times to obtain a reconstructed image.

5. The image coding method based on deep learning according to claim 4, characterized in that: The super-a priori encoder specifically includes a downsampling module and a Gaussian error linear unit; the image latent variable is sequentially passed through the downsampling module, the Gaussian error linear unit, the downsampling module, the Gaussian error linear unit, and the downsampling module to obtain a super-a priori latent variable; The super-prior decoder specifically includes an upsampling module and a Gaussian error linear unit; further, after the super-prior quantized latent variables are entropy encoded, they are sequentially passed through an upsampling module, a Gaussian error linear unit, an upsampling module, a Gaussian error linear unit, and an upsampling module to obtain prior information.

6. The image coding method based on deep learning according to claim 5, characterized in that: The structure of the entropy model specifically includes a non-linear activation function module, a Gaussian error linear unit, and a convolutional layer; further, prior information is input into the entropy model, and the entropy estimation probability distribution is obtained by sequentially passing through 4 stacked non-linear activation function modules, convolutional layers, Gaussian error linear units, convolutional layers, Gaussian error linear units, and convolutional layers, and finally the supplementary residual obtained by the residual prediction module is added to restore the error caused by quantization.

7. The image coding method based on deep learning according to claim 6, characterized in that: The residual module is composed of three stacked residual blocks, and the residual block includes a convolution layer and a nonlinear activation function including an identity mapping; the Gaussian error linear unit is an activation function with a Gaussian error in the negative domain.

8. The image coding method based on deep learning according to claim 7, characterized in that: The latent variable residual prediction module includes: 3 convolutional layers and their nonlinear activation functions.

9. An electronic device comprising a memory; a processor; and a computer program, characterized in that: The computer program is stored in the memory and is configured to be executed by the processor to implement the method according to any one of claims 1 to 8.

10. A computer readable storage device having a computer program stored thereon, characterized in that: The computer program is executed by a processor to implement the method according to any one of claims 1 to 8.

Images