Traditional image compression enhancement method based on reversible tone mapping network

Abstract

The present invention belongs to the field of image processing technology, and specifically is a method for enhancing a traditional image compression algorithm based on a reversible tone mapping network. The present invention takes a reversible deep neural network structure as the main body, maps the image to a linear space in the encoding stage, reduces the bits occupied by pixels, and establishes a robust tone mapping restoration in the decoding stage; a filter module based on window self-attention is used to suppress or enhance the mapped image in the encoding and decoding stage, thereby reducing the impact of quantization loss information. The present invention is an automatic traditional algorithm enhancement method that can effectively migrate the model of a deep learning method to a traditional algorithm. Experimental results show that the present invention can improve its performance on other indicators that are not the original design targets without training for a specific traditional algorithm.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing, and in particular relates to a traditional image compression algorithm enhancement method. Background Art

[0002] Lossy image compression aims to optimize the design target indicators and reduce the bit rate as much as possible. Traditional image compression algorithms mainly include JPEG [1] , WebP [2] , BPG based on the H.265 video coding standard [3] ,AV1 [4] and VVC based on the H.266 standard [5] . The pipeline of traditional image compression algorithms usually consists of block conversion, quantization and arithmetic coding. The bit rate of pixels is mainly determined by the selected quantization quality parameter QP. QP has different meanings in different traditional encodings. The higher the QP, the higher the bit rate of JPEG and WebP algorithms, and the lower the bit rate of BPG, AV1, VVC and other algorithms. Algorithms based on video coding standards introduce intra-frame prediction technology. However, the division of coding units for intra-frame prediction will inevitably produce blocking effects and ringing effects, thereby reducing the quality of image restoration.

[0003] Most of the image compression methods based on deep learning adopt the form of entropy coding autoencoder. Balle et al. [6] proposed a scale hyper-prior structure. Minnen et al. [7] The autoregressive model was introduced into the entropy model. Chen et al. [8] The entropy model was improved by the attention module and discrete Gaussian mixture likelihood. [9] Reversible neural network architecture

[10] Image compression is introduced to improve the conversion between image space and feature space.

[11] It can predict image details that are difficult to compress at low bit rates, thereby improving visual perception. Choi et al.

[12] The network predicts the JPEG quantization table to improve the JPEG algorithm. Klopp et al.

[13] A pre-filter network is learned to filter out unimportant image information, thereby reducing the bit rate of traditional algorithms.

[0004] In general, traditional algorithms are limited by the design target indicators (usually PSNR) and perform poorly in other evaluation criteria. Deep learning methods cannot completely replace traditional algorithms at present due to their popularity and model size limitations, and most of them use autoregressive models that cannot be calculated in parallel by GPU. The present invention provides a traditional image compression algorithm enhancement method based on a reversible tone mapping network, which can effectively migrate the model of deep learning methods to most traditional algorithms without touching traditional algorithms in the training stage, thereby improving their performance on other indicators that are not the original design targets. Summary of the invention

[0005] In order to overcome the shortcomings of the prior art, the purpose of the present invention is to provide a traditional image compression algorithm enhancement method based on a reversible tone mapping network, so as to improve the performance of the traditional algorithm on other indicators that are not the original design objectives.

[0006] The traditional image compression algorithm enhancement method based on the reversible tone mapping network provided by the present invention is based on the reversible deep neural network structure. In the encoding stage, the image is mapped to a linear space to reduce the bit occupied by pixels; in the decoding stage, a robust tone mapping restoration is established; and a filter module based on window self-attention is used to suppress or enhance the mapped image in the encoding and decoding stage, thereby reducing the impact of quantization loss information. The specific steps include:

[0007] (1) Construct a reversible tone mapping network to establish a robust tone mapping between the original image and the linear space image. The forward operation obtains the mapped image I m , the inverse operation is used to obtain the output image I d ;

[0008] (2) Construct a gradient proxy network to replace the non-differentiable traditional algorithm that cannot calculate gradients in deep learning, and obtain the gradient of the optimized neural network;

[0009] (3) A filtering module based on window self-attention is constructed to suppress or enhance the mapped image in the encoding and decoding stage, and obtain the filtered image before the traditional algorithm encoding and before the inverse operation of the tone mapping network.

[0010] In step (1), the reversible tone mapping network consists of 8 sets of affine coupling layers connected in series.

[14] Composition: Each group of affine coupling layers includes 3 basic modules and 1 reversible 1x1 convolution layer

[15] The first set of affine coupling layers inputs the original image I, the output of each set is passed to the next set as input, and the last set outputs the mapped image I mThe three basic modules are s, t1, and t2, and each basic module consists of three convolutional layers. The current input m of the affine coupling layer is divided into m1 and m2 in the channel dimension, and then the operation results n1 and n2 are combined to obtain the current output n. The positive operation process is:

[0011] n1=m1+t1(m2); (1)

[0012] n2=m2⊙exp(s(n1))+t2(n1); (2)

[0013] Where ⊙ represents the Hadamard product and exp represents the exponential operation; correspondingly, the inverse operation can be performed on each set of affine coupling layers, and the process is:

[0014] m2=(n2–t2(n1))⊙exp(-s(n1)); (3)

[0015] m1=n1–t1(m2); (4)

[0016] Among them, s(n1) represents the operation of basic module s on n1, t1(m2) represents the operation of basic module t1 on m2, and t2(n1) represents the operation of basic module t2 on n1.

[0017] In step (2), the gradient proxy network is the image compression network proposed by Minnen et al. [7] , which is used to replace the non-differentiable traditional algorithm that cannot calculate gradients in deep learning, and obtain the gradient of the optimized neural network. Therefore, there is no need to touch the traditional algorithm during the training phase; in the testing phase, the proxy network is replaced with the traditional algorithm, and the most suitable proxy network parameters for the target traditional algorithm quantization parameter QP (determines the bit rate occupied by pixels) are found by trying it on a small amount of data.

[0018] In step (3), the filter module includes two basic modules and one window self-attention module; each basic module consists of four convolutional layers, with step size or bilinear interpolation to up-sample and down-sample image features. The window self-attention module is controlled by SwinTransformer

[16] Inspired by the windowing idea of ​​​​the , the image features are divided into non-overlapping 16x16 windows through channel reorganization; the neural network features of each window are further obtained through the fully connected layer to obtain three intermediate features Q, K, and V. The calculation process of the window self-attention module is expressed as:

[0019] Attention(Q,K,V)=Softmax(QK T +B)V, (5)

[0020] Among them, B is the relative position offset of all elements in the window.

[0021] The window self-attention is finally reorganized back to its original shape and added to the input neural network features.

[0022] The filter module in the encoding stage inputs the mapping image I m , output the filtered image before traditional algorithm encoding.

[0023] The filtering module in the decoding stage inputs the decoded image of the traditional algorithm and outputs the filtered image before the inverse operation of the tone mapping network.

[0024] Furthermore, the first three convolutional layers of the proxy network are selected as the prior extraction module of the global redundancy removal mode of the deep learning method to extract prior features. The extracted prior features are adjusted for the number of channels and resolution through the convolutional layer and nearest neighbor interpolation, and the self-attention is calculated together with the neural network feature cascade in the window self-attention module to realize the migration from the deep learning method to the traditional image compression algorithm.

[0025] In the present invention, during network training, the loss function mainly consists of 4 parts:

[0026] First, in order to bring the input and output images closer, the cycle loss corresponding to the target selection is improved according to the present invention; if the MS-SSIM index is improved, the DSSIM loss, i.e., 1-MS-SSIM, is used; if the visual perception is improved, the mean square error loss and the perceptual loss are used.

[17] and Generative Adversarial Loss

[18] A weighted combination of

[0027] Second, in order to map the original image to a linear space that can reduce the number of bits occupied by pixels, the tone mapping network proposed by Xing et al.

[19] Generate reference image I raw , it is expected that after pooling, it will be close to the mapping result as a whole. For the mapping image I m , mapping loss L map The calculation method is as follows:

[0028] L map =|Pool(I m )–Pool(I raw )|, (6)

[0029] Where Pool(·) represents the pooling operation, and |·| represents taking the absolute value.

[0030] Third, in order to distinguish the functions of the tone mapping network and the two filtering modules as much as possible, the filtered image I f and the image before filtering I m The mean distance on each channel is expected to make the filter module more inclined to suppress or enhance local details. flt The calculation method is as follows:

[0031] L flt =|E HW (I m –I f )|, (7)

[0032] Among them, E HW (·) means retaining the channel dimension and taking the mean in the width and height dimensions.

[0033] Fourth, the sum of the above three loss functions is taken as the distortion loss D, and the bit rate loss R is provided by the proxy network. The balance is controlled by the weight parameter λ, and the calculation method is as follows:

[0034] L=R+λD, (8)

[0035] According to experience, λ can be less than 0.1, for example, 0.0035, 0.0067, 0.025 or 0.0932.

[0036] In the present invention, the training process of the network model is as follows:

[0037] Prepare 20295 natural images, and generate corresponding linear spatial reference images for calculating mapping loss through the tone mapping network proposed by Xing et al. The image compression network proposed by Minnen et al. is used as a proxy network to simulate the quantization behavior of the traditional image enhancement algorithm and provide gradient optimization of the neural network during the network training phase. Select the rate distortion parameter λ and the loss function corresponding to the target index of the enhanced network, minimize the loss function L by the gradient descent method of adaptive learning rate plus momentum, and obtain the network parameters of the reversible tone mapping network and filtering module in the present invention.

[0038] In the present invention, the operating process of the test is as follows:

[0039] Replace the proxy network with the traditional image compression algorithm. For a traditional algorithm that has not been used in the training phase, try it on a small amount of data to find the most suitable proxy network parameters for the target traditional algorithm quantization parameter QP;

[0040] In the encoding stage, after the original image is input, the encoding part of the present invention is first used to map it into a linear space, and then the encoding part of the traditional algorithm is used to output a compressed file;

[0041] In the decoding stage, the decoding part of the traditional algorithm is first used to obtain the mapping image, and then the decoding part of the present invention is used to process to obtain a restoration result with higher quality than that directly using the traditional algorithm.

[0042] In the present invention, the traditional image compression algorithm or traditional algorithm, etc., refers to an image compression algorithm or other algorithms that do not adopt deep learning.

[0043] The beneficial effects of the present invention are as follows: the present invention designs a deep neural network with a reversible structure as the main body, which is used to enhance the traditional image compression algorithm. In the encoding stage, the present invention can map the image to a linear space rocket to reduce the bits occupied by pixels. In the decoding stage, the present invention can establish a robust tone mapping restoration. The present invention can effectively transfer the model of the deep learning method to most traditional algorithms without touching the traditional image compression algorithm in the training stage, and improve its performance on other indicators that are not the original design targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 This is a network framework diagram of the present invention.

[0045] Figure 2 Detailed structure of the filtering module based on window self-attention.

[0046] Figure 3 This is the MS-SSIM rate-distortion curve of the present invention on the Kodak data set.

[0047] Figure 4 It is the MS-SSIM rate-distortion curve of the present invention on the Tecnick data set.

[0048] Figure 5 This is a diagram showing the enhancement effect of the present invention on the traditional algorithm BPG. DETAILED DESCRIPTION

[0049] The embodiments of the present invention are described in detail below, but the protection scope of the present invention is not limited to the embodiments.

[0050] use Figure 1 The network structure in , uses 20295 natural images to train a deep neural network and obtain an automatic enhancement model of the traditional image compression algorithm.

[0051] The specific implementation method is:

[0052] (1) Before training, the linear domain reference image corresponding to the natural image is generated through the tone mapping network proposed by Xing et al. The different rate-distortion parameter λ models of the image compression network proposed by Minnen et al. are found as the proxy network.

[0053] (2) During training, randomly crop the image to 224×224. Set the initial learning rate to 0.0001, and decay it to one-tenth after 40 rounds. Use the Adam optimizer to minimize the loss function. Set the batch size to 4. Select the rate-distortion parameter λ and the loss function corresponding to the enhanced network to improve the target indicator (MS-SSIM, visual perception, etc.), and minimize the loss function L to obtain multiple sets of models.

[0054] (3) During the test, the proxy network is replaced with the traditional image compression algorithm. For a traditional algorithm that the enhanced network has not been exposed to, it is tried on a small batch of test data to find the rate-distortion parameter λ model corresponding to the quantization parameter QP. In the encoding stage, after inputting the test image, the encoding part of the present invention is first used to map it to the linear domain, and then the encoding part of the traditional algorithm is used to output the compressed file. In the decoding stage, the decoding part of the traditional algorithm is first used to obtain the mapped image, and then the decoding part of the present invention is used to process it to obtain a higher quality restoration result than directly using the traditional algorithm.

[0055] Figure 3 The MS-SSIM rate-distortion curves of the present invention and the traditional algorithm on the Kodak data set. It can be seen that for traditional image compression algorithms that have not been exposed in the training phase, the present invention effectively transfers the model of the deep learning method to most traditional algorithms.

[0056] Figure 4 Figure 3 is the MS-SSIM rate-distortion curve of the present invention and the traditional algorithm on the Tecnick dataset. It can be seen that on a 1200x1200 high-resolution image, the present invention can still achieve a higher quality restoration result.

[0057] Figure 5 This is a diagram showing the enhancement effect of the present invention on the traditional method BPG. Figure 5 (a) is the original image. Figure 5 (c) To improve the performance of MS-SSIM indicators, Figure 5 (d) To improve the visual perception effect; it can be seen that for other indicators other than the original design indicator PSNR, the method of the present invention can improve the performance.

[0058] References

[0059] [1]Gregory K.Wallace.The jpeg still picture compressionstandard.Commun.ACM,34(4):30–44,1991.

[0060] [2]Google.Web picture format.[EB / OL],2010.https: / / chromium.googlesource.com / webm / libwebp.

[0061] [3]Fabrice Bellard.Bpg image format.[EB / OL],2015.https: / / bellard.org / bpg / .

[0062] [4]Alliance of Open Media(AOM).Av1.[EB / OL],2018.https: / / aomedia.googlesource.com / aom.

[0063] [5]Joint Video Experts Team(JVET).Vvc official test model vtm.[EB / OL],2021.https: / / vcgit.hhi.fraunhofer.de / jvet / VVCSoftware_VTM / - / tree / VTM-14.0.

[0064] [6]Johannes Ball′e,David Minnen,Saurabh Singh,Sung Jin Hwang,and NickJohnston.Variational image compression with a scale hyperprior.InInternational Conference on Learning Representations,2018.

[0065] [7]David Minnen,Johannes Ball′e,and George Toderici.Jointautoregressive and hierarchical priors for learned image compression.InAdvances in Neural Information Processing Systems,pages10794–10803,2018.

[0066] [8]Zhengxue Cheng,Heming Sun,Masaru Takeuchi,and Jiro Katto.Learnedimage compression with discretized gaussian mixture likelihoods and attentionmodules.In Conference on Computer Vision and Pattern Recognition,pages 7936–7945,2020.

[0067] [9]Yueqi Xie,Ka Leong Cheng,and Qifeng Chen.Enhanced invertibleencoding for learned image compression.In ACM International Conference onMultimedia,pages 162–170,2021.

[0068]

[10] Laurent Dinh,Jascha Sohl-Dickstein,and Samy Bengio.Densityestimation using real NVP.In International Conference on LearningRepresentations,2017.

[0069]

[11] Fabian Mentzer,George Toderici,Michael Tschannen,and EirikurAgustsson.High-fidelity generative image compression.In Advances in NeuralInformation Processing Systems,2020.

[0070]

[12] Jin Young Choi and Bohyung Han.Task-aware quantization networkfor JPEG image compression.In European Conference on Computer Vision,volume12365,pages 309–324,2020.

[0071]

[13] Jan P.Klopp,Keng-Chi Liu,Liang-Gee Chen,and Shao-Yi Chien.How toexploit the transferability of learned image compression to conventionalcodecs.In IEEE Conference on Computer Vision and Pattern Recognition,pages16165–16174,2021.

[0072]

[14] Laurent Dinh,Jascha Sohl-Dickstein,and Samy Bengio.Densityestimation using Real NVP.In International Conference on LearningRepresentations.2017.

[0073]

[15] Diederik P.Kingma and Prafulla Dhariwal.Glow:Generative flow withinvertible 1x1convolutions.In Advances in Neural Information ProcessingSystems,pages 10236–10245,2018.

[0074]

[16] Ze Liu,Yutong Lin,Yue Cao,Han Hu,Yixuan Wei,Zheng Zhang,StephenLin,and Baining Guo.Swin transformer:Hierarchical vision transformer usingshifted windows.In International Conference on Computer Vision,2021.

[0075]

[17] Justin Johnson and Alexandre Alahi and Li Fei-Fei.PerceptualLosses for Real-Time Style Transfer and Super-Resolution.In EuropeanConference on Computer Vision,2016.

[0076]

[18] Alec Radford,Luke Metz,and Soumith Chintala.UnsupervisedRepresentation Learning with Deep Convolutional Generative AdversarialNetworks.In International Conference on Learning Representations.2016.

[0077]

[19] Yazhou Xing,Zian Qian,and Qifeng Chen.Invertible image signalprocessing.In IEEE Conference on Computer Vision and Pattern Recognition,pages 6287–6296,2021。

Claims

1. A traditional image compression enhancement method based on a reversible tone mapping network, characterized in that: Based on the reversible deep neural network structure, the original image is mapped to a linear space in the encoding stage to reduce the number of bits occupied by pixels. Establish robust tone mapping restoration at the decoding stage; A filter module based on window self-attention is used to suppress or enhance the mapped image in the encoding and decoding stage, thereby reducing the impact of quantization loss information; the specific steps are: (1) Construct a reversible tone mapping network to establish a robust tone mapping between the original image and the linear space image. The forward operation obtains the mapped image I m , the inverse operation is used to obtain the output image I d ; (2) Construct a gradient proxy network to replace the non-differentiable traditional algorithm that cannot calculate gradients in deep learning, and obtain the gradient of the optimized neural network; (3) Construct a filtering module based on window self-attention to suppress or enhance the mapped image in the encoding and decoding stage, and obtain the filtered image before the traditional algorithm encoding and before the inverse operation of the tone mapping network; In step (1), the reversible tone mapping network is composed of 8 groups of affine coupling layers connected in series; each group of affine coupling layers includes 3 basic modules and 1 reversible 1x1 convolutional layer; wherein the first group of affine coupling layers input is the original image I, the output result of each group is passed to the next group as input, and the last group outputs the mapped image I m ; The three basic modules are s, t1, and t2 respectively, and each basic module consists of three convolutional layers; for the current input m of the affine coupling layer, it is divided into m1 and m2 in the channel dimension, and then the operation results n1 and n2 are combined to obtain the current output n. The positive operation process is: n1 = m1 + t1(m2); (1) n2 = m2 ⊙ exp(s(n1)) + t2(n1); (2) Where ⊙ represents the Hadamard product and exp represents the exponential operation; correspondingly, the inverse operation can be performed on each set of affine coupling layers, and the process is: m2 = (n2 – t2(n1)) ⊙ exp(-s(n1)); (3) m1 = n1 – t1(m2); (4) Among them, s(n1) represents the operation of basic module s on n1, t1(m2) represents the operation of basic module t1 on m2, and t2(n1) represents the operation of basic module t2 on n1; In step (2), the gradient proxy network is a pre-trained image compression network, which is used to replace the non-differentiable traditional algorithm that cannot calculate the gradient in deep learning, and obtain the gradient of the optimized neural network; in the test phase, the proxy network is switched back to the traditional algorithm, and the most suitable proxy network parameters for the target traditional algorithm quantization parameter QP are found by trying on a small amount of data; In step (3), the filtering module includes two basic modules and one window self-attention module; each basic module is composed of four convolutional layers, and the image features are up-sampled or down-sampled with step size or bilinear interpolation; the window self-attention module divides the image features into non-overlapping 16x16 windows by channel reorganization; the neural network features of each window are further obtained through the fully connected layer to obtain three intermediate features Q, K, and V. The calculation process of the window self-attention module is expressed as: Attention(Q, K, V) = Softmax(QK T +B) V, (5) Among them, B is the relative position offset of all elements in the window; The window self-attention is finally reorganized back to its original shape and added to the input neural network features; The filter module in the encoding stage inputs the mapping image I m , output the filtered image before encoding by the traditional algorithm; The filtering module in the decoding stage inputs the decoded image of the traditional algorithm and outputs the filtered image before the inverse operation of the tone mapping network.

2. The conventional image compression enhancement method according to claim 1, characterized in that: The first three convolutional layers of the proxy network are selected as the prior extraction module of the global de-redundancy mode of the deep learning method to extract prior features; the extracted prior features are adjusted to the number of channels and resolution through convolutional layers and nearest neighbor interpolation, and the self-attention is calculated together with the neural network feature cascade in the window self-attention module to realize the migration from deep learning methods to traditional image compression algorithms.

3. The conventional image compression enhancement method according to claim 1, characterized in that: In network training, the loss function has 4 parts: First, to bring the input and output images closer, the corresponding cycle loss is selected according to the improvement target; if the MS-SSIM index is improved, the DSSIM loss, i.e. 1-MS-SSIM, is used; if the visual perception is improved, a weighted combination of mean square error loss, perceptual loss and generative adversarial loss is used; Second, in order to map the original image to a linear space that can reduce the number of bits occupied by pixels, a tone mapping network is used in advance to generate a reference image I raw , it is expected that after pooling, it will be close to the mapping result as a whole; for the mapping image I m , mapping loss L map The calculation method is as follows: L map = | Pool(I m ) – Pool(I raw ) |, (6) Where Pool(·) represents the pooling operation, and |·| represents taking the absolute value; Third, in order to distinguish the functions of the tone mapping network and the two filtering modules as much as possible, the filtered image I f and the image before filtering I m The mean distance on each channel, it is expected that the filtering module will focus more on suppressing or enhancing local details; the filtering loss L flt The calculation method is as follows: THE flt = | And HW (THE m – I f ) |, (7) Among them, E HW (·) means retaining the channel dimension and taking the mean in the width and height dimensions; Fourth, the sum of the above three loss functions is taken as the distortion loss D, and the bit rate loss R is provided by the proxy network. The balance is controlled by the weight parameter λ, and the calculation method is as follows: L = R + λD, (8) Here, λ is set to be less than 0.

1.

4. The conventional image compression enhancement method according to claim 3, characterized in that: The training process of the network model is as follows: 20295 natural images were prepared, and the corresponding linear spatial reference images were generated through the tone mapping network for calculating the mapping loss. The image compression network was used as a proxy network to simulate the quantization behavior of the traditional image enhancement algorithm and provide the gradient of the optimized neural network during the network training phase. The rate-distortion parameter λ and the loss function corresponding to the target indicator of the enhancement network were selected, and the loss function L was minimized by the gradient descent method with adaptive learning rate plus momentum to obtain the network parameters of the reversible tone mapping network and the filtering module.

5. The conventional image compression enhancement method according to claim 4, characterized in that: The test process is as follows: Replace the proxy network with the traditional image compression algorithm. For a traditional algorithm that has not been used in the training phase, try it on a small amount of data to find the most suitable proxy network parameters for the target traditional algorithm quantization parameter QP; In the encoding stage, after the original image is input, the encoding part of the present invention is first used to map it into a linear space, and then the encoding part of the traditional algorithm is used to output a compressed file; In the decoding stage, the decoding part of the traditional algorithm is first used to obtain the mapping image, and then the decoding part of the present invention is used to process to obtain a restoration result with higher quality than that directly using the traditional algorithm.

Images