A content-adaptive depth image compression method

Abstract

The present invention discloses a content-adaptive deep image compression method, including encoding and decoding. The encoding step is to extract the features of the image to be compressed and quantize them, and select a channel width adapted to the image content for each spatial position of the features according to the rate-distortion loss, so as to obtain the features after channel adaptation; the decoding step is to use a conditional decoder to reconstruct the features after channel adaptation into an image. Based on the selection strategy of the rate-distortion loss, the present invention selects the best quality level for each spatial position of the hidden layer features, and integrates the content characteristic information of the image into the decoding process through the conditional decoder, thereby realizing the content adaptability of deep image compression.

Description

Technical Field

[0001] The present invention relates to the technical fields of image compression and deep learning, and more specifically, to a content-adaptive deep image compression method. Background Art

[0002] Images, as important multimedia carriers, have been widely used in daily life. To reduce the cost of storing and transmitting images, traditional image coding has been studied for decades. In recent years, image compression algorithms based on deep learning have been continuously proposed. With the powerful non-linear expressiveness of neural networks and the end-to-end training method, their compression performance on some evaluation data sets has exceeded the intra-frame coding mode in the latest traditional video coding standard VVC. However, these deep learning-based image compression methods rely on the generalization of data and lack adaptability to image content. When the distribution of test data is quite different from that of training data, a domain shift problem will occur, resulting in a decrease in compression performance.

[0003] Currently, in deep image compression, most of the methods to solve this problem are based on online update methods. Since the encoder network does not participate in the operation during the decoding process, the network can be overfitted to the test samples to improve the adaptability to a single image. To ensure that the receiver reconstructs the same image, only the components at the encoding end are updated in such methods, and the adaptation ability of the decoding end to image content is not explored. At the same time, to further explore the adaptation ability of the decoding end, another type of method updates the entire autoencoder and compresses and transmits the changes in the network parameters of the decoding end to the receiver. This type of method involves model compression and is relatively complex, and both of these two types of methods require a large number of backpropagations of the network based on a single sample, which is very time-consuming.

[0004] In addition, deep image compression networks usually use the same network structure for models with all target bitrates, generating hidden layer features and hyperprior information with the same dimensions, resulting in redundant elements in the hidden layer features and hyperprior information generated by models with lower target bitrates; and the ability to convert bits into reconstruction quality is also different at different positions in the image, and the content of the image is not adapted when the same number of elements is assigned to all spatial positions.

[0005] Therefore, how to provide a new content-adaptive image compression method to overcome the above problems is an urgent problem for those skilled in the art. Summary of the Invention

[0006] In view of this, the purpose of the present invention is to provide a content - adaptive depth image compression method. By providing content adaptability from two parts: the hidden - layer features and the decoder, the method disclosed in the present invention can improve the performance of the depth image compression method without an additional online update step during encoding.

[0007] To achieve the above - mentioned purpose, the present invention adopts the following technical solutions:

[0008] On the one hand, the present invention discloses a content - adaptive depth image compression method, including encoding and decoding.

[0009] The encoding step includes:

[0010] Extracting the features of the image to be compressed, quantizing them, generating features of multiple levels, and each level of features corresponds to a channel width.

[0011] Calculating the rate - distortion loss generated when using different levels of features for the spatial position of the image to be compressed, and selecting the channel width corresponding to the feature with the minimum rate - distortion loss as the channel width of the spatial position to obtain the features after channel adaptation.

[0012] The decoding step includes:

[0013] Decoding the features after channel adaptation through a decoder network.

[0014] The decoder network includes an image feature information extraction network and a reconstruction convolutional layer, and a conditional spatial feature transformation layer is provided in the reconstruction convolutional layer.

[0015] The image feature information extraction network is used to generate image feature information according to the features.

[0016] The conditional spatial feature transformation layer generates transformation parameters according to the image feature information, and modulates the features after upsampling in the reconstruction convolutional layer using the transformation parameters to obtain the modulated features.

[0017] The reconstruction convolutional layer reconstructs the image to be compressed according to the modulated features.

[0018] Preferably, the extraction of the features of the image to be compressed includes the hidden - layer features obtained by mapping the image to be compressed through an encoder network, and the spatial dependence information extracted from the hidden - layer features using a hyper - prior encoder, and the spatial dependence information is a hyper - prior.

[0019] Preferably, the quantization is to quantize the features into a discrete value form using the method of rounding.

[0020] Preferably, the features of each level set to 0 the elements that exceed the corresponding channel width of the level;

[0021] Preferably, the encoding step further includes encoding the adapted features into a bitstream;

[0022] Before decoding through the decoder, the arithmetic coding algorithm is used to restore the quantized hyperprior from the bitstream. After calculating the probability distribution parameters of the hidden layer features through the entropy model parameter estimation network, the arithmetic coding algorithm is used to restore the quantized hidden layer features from the bitstream;

[0023] Preferably, the decoding process includes:

[0024] S1. Through the image feature information extraction network, extract the image feature information from the restored quantized hidden layer features and the quantized hyperprior, and perform upsampling on the image feature information to obtain image feature information with different resolutions. The number of the image feature information with different resolutions is the same as the number of the conditional spatial feature transformations;

[0025] S2. The conditional spatial feature transformation layer generates transformation parameters according to the image feature information corresponding to the resolution;

[0026] S3. The upsampling layer in the reconstruction convolutional layer performs upsampling on the restored hidden layer features, and the conditional spatial feature transformation layer modulates the upsampled features using the transformation parameters to obtain the modulated upsampled features;

[0027] S4. The image reconstruction layer in the reconstruction convolutional layer reconstructs the image to be compressed according to the modulated upsampled features;

[0028] Preferably, a conditional spatial feature transformation layer is arranged behind each upsampling layer in the reconstruction convolutional layer;

[0029] Preferably, it is combined with the content adaptive method at the encoding end based on online update to form a new content adaptive scheme.

[0030] It can be seen from the above technical solutions that the present invention discloses a content adaptive depth image compression method. Compared with the prior art, its beneficial effects at least include:

[0031] (1) Based on the rate-distortion loss selection strategy, select the best quality level for each spatial position of the hidden layer features, realize content adaptability from the hidden layer feature level; and discard redundant elements from the channel dimension to reduce redundancy in the hidden layer features, thereby achieving the effect of saving bits;

[0032] (2) Modulate the feature maps in the image reconstruction process through the transformation parameters generated by the conditional feature transformation layer, so that the content characteristic information of the image participates in the decoding process, thereby improving the adaptability of the decoder to the image content. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only the embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained according to the provided drawings.

[0034] Figure 1 Schematic flowchart of the image compression method provided by the present invention;

[0035] Figure 2 Schematic operation flowchart of the channel dropout module with content self-adaptation of the implicit feature layer provided by the present invention;

[0036] Figure 3 Schematic structural diagram of the content self-adaptive decoder network provided by the present invention;

[0037] Figure 4 For Figure 3 Schematic structural diagram of the conditional spatial feature transformation module and the Resblock module in

[0038] Figure 5 Results of applying the image compression method of the present invention on the Kodak dataset provided by the present invention;

[0039] Figure 6 Results of applying the image compression method of the present invention on the Tecnick dataset provided by the present invention;

[0040] Figure 7 Results of experiments based on the MS-SSIM metric on the Kodak dataset provided by the present invention;

[0041] Figure 8 Results evaluated using the BD savings rate provided by the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0042] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0043] An embodiment of the present invention discloses a content - adaptive depth image compression method, which provides adaptability to image content for depth image compression from two parts: hidden - layer features and the decoder.

[0044] For hidden - layer features, through a rate - distortion loss - based selection method, a suitable quality level is selected for each spatial block of the image, and redundant elements are removed from the channel dimension to save the bits required for compressing the image.

[0045] For the decoder, first, content characteristic information of the image is extracted based on hidden - layer features and hyper - prior information, and it is upsampled to obtain content characteristic information of the image at multiple scales. Then, through a conditional spatial feature transformation layer, transformation parameters are generated from the multi - scale content characteristic information of the image, and then a modulation operation is performed on the features after each upsampling in the decoder to fuse the characteristic information of the image into the image reconstruction process to improve the image reconstruction quality. The image compression method disclosed in this application solves the problem that the decoding end of the depth image compression method lacks adaptability to image content from the network architecture and does not require an additional online update step.

[0046] Specifically, the content - adaptive depth image compression method disclosed in the present invention, as Figure 1 shown, includes encoding and decoding. The encoding steps include:

[0047] Extract the features of the image to be compressed. After quantization, multiple levels of features are generated, and each level of features corresponds to a channel width, which is the minimum channel number that can guarantee the original RD performance when using this quality level (the RD trade - off hyperparameter λ in the rate - distortion loss). The number of levels is set according to empirical values. In this embodiment, the number of levels is 3.

[0048] Among them, extracting the features of the image to be compressed includes the hidden - layer features obtained by mapping the image to be compressed through an encoder network, and the spatial dependency information extracted from the hidden - layer features using a hyper - prior encoder. This spatial dependency information is the hyper - prior.

[0049] And quantization is to quantize the features of the compressed image into discrete values by the method of rounding.

[0050] At the same time, to achieve the purpose of saving the bits required for compressing the image, in this application, the elements in each level of features that exceed the channel width corresponding to this quality level are set to 0 to remove redundant elements from the channel dimension.

[0051] Further, an entropy model parameter estimation network and a decoder are used to calculate the rate-distortion loss generated when using features of different levels for the spatial positions of the image to be compressed, and the channel width corresponding to the feature with the minimum rate-distortion loss is selected as the channel width of the spatial position, obtaining the hidden layer features and the hyperprior after channel adaptation.

[0052] In addition, the encoding step further includes encoding the adapted features into a bitstream.

[0053] Before decoding through the decoder, the quantized hyperprior is restored from the bitstream using the arithmetic coding algorithm. After calculating the probability distribution parameters of the hidden layer features through the entropy model parameter estimation network, the quantized hidden layer features are restored from the bitstream using the arithmetic coding algorithm.

[0054] The decoding step mainly includes:

[0055] Decoding the adapted features through the decoder network.

[0056] Among them, the decoder network includes an image feature information extraction network and a reconstruction convolutional layer, and a conditional spatial feature transformation layer is provided in the reconstruction convolutional layer; specifically, the image feature information extraction network is used to generate image feature information according to the features.

[0057] The conditional spatial feature transformation layer generates transformation parameters according to the image feature information and modulates the features after upsampling in the reconstruction convolutional layer using the transformation parameters to obtain the modulated features.

[0058] The reconstruction convolutional layer reconstructs the image to be compressed according to the modulated features.

[0059] The decoding process specifically includes:

[0060] S1. Through the image feature information extraction network, image feature information is extracted from the restored quantized hidden layer features and the quantized hyperprior, and the image feature information is upsampled to obtain image feature information with the same resolution as the number of conditional spatial feature transformations.

[0061] S2. The conditional spatial feature transformation layer generates transformation parameters according to the image feature information of the corresponding resolution.

[0062] S3. The upsampling layer in the reconstruction convolutional layer upsamples the restored hidden layer features, and the features after upsampling are modulated by the conditional spatial feature transformation layer using the transformation parameters to obtain the modulated upsampled features.

[0063] S4. The image reconstruction layer in the reconstruction convolutional layer reconstructs the image to be compressed according to the modulated upsampled features.

[0064] Among them, the convolutional layer that generates a 3-channel reconstructed image in the reconstruction convolutional layer is the image reconstruction layer, and the convolutional layer that upsamples the input feature map is the upsampling layer. There can be multiple upsampling layers in the reconstruction convolutional layer. In one embodiment, a conditional spatial feature transformation layer is set behind each upsampling layer, and the conditional spatial feature transformation layer modulates the features of each upsampling. The specific method of modulation is f l a = f l ⊙γ l +β l , where f l is the feature after upsampling, γ l and β l are transformation parameters generated by the conditional spatial feature transformation layer, and ⊙ and + are element-wise multiplication and element-wise addition respectively;

[0065] In one embodiment, the image compression method disclosed in the present invention can be combined with a content adaptive method based on online update at the encoding end to form a complete content adaptive scheme, further improving the performance of the depth image compression method.

[0066] To make the above objects, features, and advantages of the present invention more obvious and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0067] The technical solution of the present invention is based on a convolutional neural network, and uses an autoencoder architecture to implement a depth image compression framework, and enhances the adaptability of the hidden layer features and the decoder to the image content.

[0068] In one embodiment, the image compression method disclosed in the present invention specifically has the following steps. Refer to Figure 1 ,

[0069] 1. For an input image x to be compressed, first use the encoder network to map it to the hidden layer feature y, and then use the hyperprior encoder to extract the spatial dependence information in the hidden layer feature y and store it in the hyperprior z;

[0070] 2. Use the method of rounding to quantize the hidden layer feature and the hyperprior into discrete value forms of and

[0071] 3. Generate three levels of features according to and . Among them, each level λ corresponds to a channel width w, and the features of each level will set the elements of and that exceed the corresponding channel width w in the channel dimension to 0;

[0072] In one embodiment, such asFigure 2 Calculate the rate-distortion loss generated by using features of three levels at each spatial position respectively, select the channel width corresponding to the quality level of the feature with the minimum rate-distortion loss as the channel width at this spatial position, and store the result in the binary tensor m a m a The elements in m whose number of channels does not exceed the calculated optimal channel width are 1, and the elements exceeding the calculated optimal channel width are 0. For example Figure 1 and Figure 2 , and By performing an element-wise multiplication operation with m a (and its downsampled result), the feature after channel adaptation is obtained and and and are encoded into a bitstream through the arithmetic coding algorithm;

[0073] 4. Use the arithmetic coding algorithm to restore the quantized hyperprior from the bitstream Calculate the probability distribution parameters of the hidden layer features through the entropy model parameter estimation network, and then use the arithmetic coding algorithm to restore the quantized hidden layer features from the bitstream

[0074] 5. Through the decoder network, is restored to the reconstructed image The decoder network includes a feature information extraction network and a reconstruction convolutional layer, and a conditional spatial feature transformation layer is embedded in the reconstruction convolutional layer. The feature information extraction network extracts the feature information of the image from the hyperprior and the hidden layer features , and upsamples these feature information to generate the same number of image feature information as the conditional spatial feature transformation layer. The conditional spatial feature transformation layer generates transformation parameters through the image feature information corresponding to the resolution

[0075] 6. At this time, the upsampling layer in the reconstruction convolutional layer will continuously upsample the hidden layer features , and for the upsampled features, the conditional spatial feature transformation layer uses the transformation parameters for modulation to obtain the modulated upsampled features, thereby fusing the image feature information into the reconstruction process. Finally, the image reconstruction layer in the reconstruction convolutional layer generates the reconstructed image according to the modulated upsampled features

[0076] Furthermore, the schematic diagram of the decoder network structure is as shown in Figure 3As shown, where the two dashed box parts are the image feature information extraction network. The lower dashed box part is used to extract the feature information of the image from the hyperprior and the latent feature layer, and output the content feature information c of the original image. 0 The upper right network part is used to upsample the aforementioned content feature information to output image feature information c of multiple different scales or resolutions. 1 、c 2 、c 3 ,

[0077] In one embodiment, as Figure 3 shown, the upper left part is the reconstruction convolutional layer. The reconstruction convolutional layer contains a total of 4 transposed convolutional layers. From bottom to top, the first three layers are upsampling layers, and the last layer is the image reconstruction layer. In this application, a conditional feature transformation layer is embedded behind each upsampling layer, which is used to generate transformation parameters according to the image feature information of the corresponding resolution to modulate the upsampling features of each upsampling layer, so as to fuse the image feature information into the reconstruction process. Then, the final image reconstruction layer reconstructs the image to be compressed according to the finally modulated image features.

[0078] And Figure 3 in, the three parameters in Conv and Deconv represent the number of output channels, the convolutional kernel size, and the convolutional stride in turn. IGDN and LReLU represent two commonly used activation functions in depth image compression.

[0079] Further, Figure 4 (a) is the structural schematic diagram of the conditional spatial feature transformation module in Figure 3 which. The conditional spatial feature transformation layer first generates transformation parameters γ l and β l according to the input image content feature information c l , and then uses these parameters to perform a modulation operation on the input image feature f l . The modulation method is:

[0080]

[0081] Performance evaluation:

[0082] The performance evaluation of image compression needs to measure both the size of the compressed bitstream and the distortion between the reconstructed image and the original image. Usually, the rate-distortion curve is used for evaluation. Better quality at the same bitrate or smaller bitrate at the same quality represents better performance.

[0083] The present invention discloses an image compression method (labeled as Ours and Ours w / o Context respectively) based on two versions of the baseline algorithm (Minnen D, J Ballé, Toderici G. Joint Autoregressive and Hierarchical Priors for Learned Image Compression [J]. 2018.) (with and without autoregressive context models), and the rate-distortion performance is evaluated on the Kodak dataset and the Tecnick dataset.

[0084] First, for each image, the entire network is used to generate channel-adapted hidden layer features, channel-adapted hyper priors, and reconstructed images.

[0085] The distortion evaluation is based on the distortion metric (PSNR or MS-SSIM) calculated from the obtained reconstructed image and the original image.

[0086] The bitrate evaluation is to calculate the sizes of the entropy-encoded hidden layer features and hyper priors, and divide by the number of pixels in the image to obtain the Bpp (Bits per pixel).

[0087] For all images in a dataset, after execution, an average is taken to obtain a (Bpp, PSNR) point. This application will use the rate-distortion loss to train multiple models. Each model uses a λ value as the rate-distortion trade-off during training, and each model obtains a point, and finally a curve is drawn.

[0088] Figure 5 and Figure 6 are the results of implementing this algorithm on the Kodak dataset and the results of comparison with other state-of-the-art algorithms respectively. Other state-of-the-art algorithms are labeled as "author (published conference or journal)". As shown in the above figure, the methods disclosed in the present invention enhance the corresponding baseline algorithms Minnen (NIPS18) and Minnen (NIPS18) w / o Context. In addition, the method of the present invention can also be combined with the encoder-side adaptive method based on online update (the curve with the +SGA suffix) to form a complete content-adaptive scheme. The experimental results prove that the content-adaptive scheme proposed in the present invention can achieve advanced coding performance and exceed other deep image compression methods in the high bitrate range.

[0089] Furthermore, as Figure 7 shown, the image compression method disclosed in the present invention can also enhance the corresponding baseline algorithm and improve the compression performance based on the MS-SSIM metric on the Kodak dataset.

[0090] In the ablation experiment, the BD savings rate on the Tecnick dataset was further used to evaluate the method disclosed in the present invention. The BD savings rate is based on the linear BDBR result and represents the bitrate savings compared to the baseline algorithm under the same quality.

[0091] As Figure 8 shown, the content-adaptive channel dropping method (labeled Baseline+CACD) related to the present invention can save approximately 1%-2% of the bitrate based on the baseline algorithm at all code points; the content-adaptive feature transformation method (labeled Baseline+CAFT) can save approximately 3%-8% of the bitrate based on the baseline algorithm at all code points; while the deep image compression method with content-adaptive hidden layer features and decoder proposed in the present invention (labeled Baseline+CACD+CAFT (Ours)) can save approximately 4%-12% of the bitrate at all code points.

[0092] The experimental results show that the methods proposed in the present invention can all improve the performance of the baseline algorithm, proving the effectiveness of the algorithm.

[0093] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The same or similar parts among the various embodiments can be referred to each other.

[0094] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be obvious to those skilled in the art. The general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to these embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A content-adaptive depth image compression method, characterized in that, Including encoding and decoding, The encoding step includes: Extracting the features of the image to be compressed, quantizing them, generating features of multiple levels, and each level of features corresponds to a channel width. Calculating the rate-distortion loss generated when using the features of different levels to represent the spatial position of the image to be compressed, and selecting the channel width corresponding to the feature with the minimum rate-distortion loss as the channel width of the spatial position, so as to obtain the features after channel adaptation. The decoding step includes: Decoding the features after channel adaptation through a decoder network. The decoder network includes an image feature information extraction network and a reconstruction convolutional layer, and a conditional spatial feature transformation layer is provided in the reconstruction convolutional layer. The image feature information extraction network is used to generate image feature information according to the features. The conditional spatial feature transformation layer generates transformation parameters according to the image feature information, and uses the transformation parameters to modulate the features after upsampling in the reconstruction convolutional layer to obtain the modulated features. The reconstruction convolutional layer reconstructs the image to be compressed according to the modulated features.

2. The content-adaptive depth image compression method according to claim 1, characterized in that, The extraction of the features of the image to be compressed includes the hidden layer features obtained by mapping the image to be compressed through an encoder network, and the spatial dependence information extracted from the hidden layer features by using a hyperprior encoder, and the spatial dependence information is a hyperprior.

3. The content-adaptive depth image compression method according to claim 1, characterized in that, The quantization is to quantize the features into a discrete value form by using the method of rounding.

4. The content-adaptive depth image compression method according to claim 1, characterized in that, For each level of features, the elements exceeding the channel width corresponding to the level are set to 0.

5. The content-adaptive depth image compression method according to claim 2, characterized in that, The encoding step further includes encoding the adapted features into a bitstream. Before decoding through the decoder network, the quantized hyperprior is restored from the bitstream by using an arithmetic coding algorithm. After calculating the probability distribution parameters of the hidden layer features through an entropy model parameter estimation network, the quantized hidden layer features are restored from the bitstream by using an arithmetic coding algorithm.

6. The content-adaptive depth image compression method according to claim 5, characterized in that, The decoding process includes: S1. Through the image feature information extraction network, extract image feature information from the restored quantized hidden layer features and quantized hyperprior, and upsample the image feature information to obtain image feature information of different resolutions. The number of the image feature information of different resolutions is the same as the number of the conditional spatial feature transformation layers. S2. The conditional spatial feature transformation layer generates transformation parameters according to the image feature information corresponding to the resolution. S3. The upsampling layer in the reconstruction convolutional layer upsamples the restored hidden layer features, and the conditional spatial feature transformation layer uses the transformation parameters to modulate the features after upsampling to obtain the modulated upsampled features. S4. The image reconstruction layer in the reconstruction convolutional layer reconstructs the image to be compressed according to the modulated upsampled features.

7. The content-adaptive depth image compression method according to claim 1, characterized in that, A conditional spatial feature transformation layer is arranged behind each upsampling layer in the reconstruction convolutional layer.

8. The content-adaptive depth image compression method according to any one of claims 1-7, characterized in that, Combined with the content adaptive method at the encoding end based on online update to form a new content adaptive scheme.

Images