An image compression method based on multi-scale space and context information fusion

Abstract

The present invention relates to an image compression method based on multi-scale space and context information fusion, the method comprising the following steps: 1) constructing an image compression model based on multi-scale space and context information fusion, extracting hidden features from the original image through a main encoder, and using a multi-scale information fusion module to reduce the loss of effective information propagated forward; 2) a super priori module combines super prior information with multi-scale context information, obtains parameters and weights of three Gaussian functions, and obtains the probability distribution of hidden features by a Gaussian mixture model after adding the weights; 3) based on the probability distribution of hidden features, an arithmetic encoder encodes and decodes the hidden features; 4) a main decoder reconstructs the hidden features into a picture to complete image compression. Compared with the prior art, the present invention can achieve better image reconstruction quality at a lower compression code rate.

Description

Technical Field

[0001] The present invention relates to the technical field of image compression, and in particular to an image compression method based on multi-scale space and context information fusion. Background Art

[0002] After the third information revolution, a large amount of digital information was transmitted between various terminals. However, due to the limitations of the digital information acquisition methods at that time, most of the information remained in the form of text. However, with the birth and popularization of various electronic digital products, especially the advent of the mobile Internet era, everyone can become a photographer. Electronic devices that can be easily used have enabled a large amount of pictures and video information to be transmitted on the Internet. The large amount of data has also led to a rapid increase in the requirements for data speed and storage space. Therefore, the necessity of data compression is also reflected. How to compress image data, which occupies a considerable part of Internet data, has become a hot research topic.

[0003] Before the emergence of deep learning-based image coding methods, there were a large number of traditional methods, including JPEG, JPEG2000, BPG, etc. These methods are still widely used today, but traditional methods have many manually designed components, generally including block segmentation, linear transformation, quantization and entropy coding. Due to the rapid development of deep learning and its wide application in many computer vision fields, a large number of end-to-end image compression methods based on deep learning have been proposed. Most of the existing methods are based on relatively mature deep learning models, such as image compression based on convolutional neural networks, image compression based on generative adversarial networks, and image compression based on graph convolutional neural networks. Among them, the image compression algorithm based on generative adversarial networks uses adversarial training between the generator and the discriminator to improve the human eye perception of image reconstruction at low bit rates, but the restored images perform poorly in terms of peak signal-to-noise ratio (PSNR) and multi-level structural similarity (MS-SSIM) indicators. Because of the advantages of convolutional neural networks in image feature extraction, most deep learning-based image compression generally uses the structure of convolutional neural networks. The existing convolutional neural network-based methods first extract hidden features from the image through the main encoder, and then use the super-prior autoencoder to extract the edge information in the hidden features as super-prior features. After that, the probability distribution of the hidden features is estimated by combining the super-prior features with the context features obtained by the context model, so as to arithmetically encode the hidden features. Finally, the main encoder is used to restore the hidden features into an image. However, the existing methods still have many shortcomings. First, the main encoder removes the spatial redundancy in the hidden features, but also loses some effective spatial information, especially the information of the area with complex texture is lost in the forward propagation, which limits the quality of image reconstruction. On the other hand, since the content scale of the compressed image is uncertain, the fixed-size mask convolution kernel in the context model cannot effectively obtain effective correlation information from the hidden features, which makes the accuracy of the entropy model of the existing method still insufficient. Summary of the invention

[0004] The purpose of the present invention is to provide an image compression method based on multi-scale space and context information fusion in order to overcome the defects of the above-mentioned prior art.

[0005] The purpose of the present invention can be achieved by the following technical solutions:

[0006] An image compression method based on multi-scale space and context information fusion, the method comprising the following steps:

[0007] 1) Construct an image compression model based on multi-scale space and context information fusion, extract hidden features from the original image through the main encoder, and use a multi-scale information fusion module to reduce the loss of effective information in forward propagation;

[0008] 2) The hyper-prior module combines the hyper-prior information with the multi-scale context information to obtain the parameters and weights of the three Gaussian functions, and obtains the probability distribution of the hidden features by adding the weights to obtain the Gaussian mixture model;

[0009] 3) Based on the probability distribution of hidden features, the arithmetic encoder encodes and decodes the hidden features;

[0010] 4) The main decoder reconstructs the hidden features into a picture and completes the image compression.

[0011] The step 1) specifically comprises the following steps:

[0012] 11) The original image is extracted and downsampled by the residual block, attention module and multi-scale information fusion module to obtain the hidden feature y. In order to entropy encode y, it is quantized with a step size of 1 to obtain the quantized hidden feature Then we have:

[0013]

[0014]

[0015] Among them, x is the original image, is the parameter of the main encoder, Q(·) represents the quantization process, g a (·) indicates the main encoder;

[0016] 12) The feature y downsampled i times is fused through the multi-scale information fusion module in the main encoder (i) And the feature y downsampled i+2 times (i+2) In order to reduce the consumption of computing resources, the main encoder only uses two multi-scale information modules through the form of attention mechanism, and then:

[0017] y (i+2) =y (i+2) +y (i+2) *sigmoid(Res(y (i) )).

[0018] Among them, Res(·) represents the residual block.

[0019] The step 2) specifically comprises the following steps:

[0020] 21) The super-prior encoder calculates the super-prior feature z from the hidden feature y, and then quantizes it to obtain the quantized super-prior feature To assist in extracting spatial redundancy in hidden features and improve the accuracy of hidden feature probability distribution estimation, we have:

[0021]

[0022]

[0023] Among them, h a (·) represents the super prior encoder, are the parameters of the super prior encoder;

[0024] 22) Use the multi-scale 3D context module to extract the hidden features from the quantized The multi-scale context features obtained in Then we have:

[0025]

[0026] Among them, downsample means downsampling, represents a 3D context model with a convolution kernel size of 5×5×5. represents a three-dimensional context model with a convolution kernel size of 7×7×7, Represents a three-dimensional context model with a convolution kernel size of 9×9×9;

[0027] 23) Multi-scale context features and super prior features After the combination, the model parameters and weights of the Gaussian mixture model are solved using the super prior decoder, and then:

[0028]

[0029] Among them, ω i , μ i , They represent the weight, mean and variance of the i-th Gaussian model in the Gaussian mixture model, represents the i-th super-prior decoder;

[0030] 24) According to the weights, the three Gaussian functions are combined into a Gaussian mixture model as the entropy model, and the estimated probability distribution of the hidden features is calculated, then:

[0031]

[0032] in, Based on super prior features Hidden features of The conditional probability distribution of Based on the parameter ω i , μ i The Gaussian probability distribution of For the range from arrive Uniformly distributed noise.

[0033] The step 3) specifically comprises the following steps:

[0034] 31) To prevent the gradient vanishing phenomenon during model training, the quantization process is replaced by adding independent and identically distributed uniform noise during the image compression model training stage;

[0035] 32) During the use of the image compression model, the hidden features are quantized, and the probability distribution of the features is calculated based on the entropy model obtained by the super-prior encoder, and the quantized features are encoded using arithmetic coding in entropy coding.

[0036] The step 4) specifically comprises the following steps:

[0037] 41) The hidden features are transformed back into images through the residual block and upsampling in the main decoder, then:

[0038]

[0039] Among them, g s (·) indicates the main decoder, are the parameters of the main decoder, For the reconstructed picture;

[0040] 42) The reconstructed image Compare the objective and subjective indicators with the original image x to evaluate the compression and reconstruction effects of the model.

[0041] In the step 42), the objective and subjective indicators include PSNR and MS-SSIM.

[0042] In the step 1), the image compression model based on the fusion of multi-scale space and context information consists of a main encoder, a super-prior encoder, a super-prior decoder, a main decoder and a multi-scale three-dimensional context module.

[0043] When training the image compression model, in order to balance the relationship between bit rate and image reconstruction quality, the training objective function is set to:

[0044]

[0045] Among them, λ is a hyperparameter that balances bit rate and image reconstruction quality, and They are the quantized hidden features. and the quantized hyper-prior features The bit rate of the model is , D(·) represents the difference between the original image and the reconstructed image. MSE and MS-SSIM are used as the measurement criteria. When MSE is used to optimize the model, the evaluation criterion of the model is PSNR. When MS-SSIM is used to optimize the model, the evaluation criterion of the model is MS-SSM.

[0046] To speed up the convergence of the model, pre-training is first performed at a high bit rate, and then the value of λ is modified to adjust the bit rate of the model to other values.

[0047] When training the pre-trained model, the learning rate decreases with the number of iterations. When training models with other bit rates, the initial value of the learning rate increases and decreases with the number of iterations.

[0048] Compared with the prior art, the present invention has the following advantages:

[0049] 1. The multi-scale information fusion module used in the main encoder uses the attention mechanism to fuse image features of different scales together. This method preserves the spatial information of complex areas while avoiding adding additional spatial redundancy in hidden features.

[0050] 2. The multi-scale three-dimensional context module used in the context model fuses the correlation information in different scale spaces of the hidden features by using masked three-dimensional convolution kernels of different sizes in parallel, thereby improving the accuracy of the entropy model and improving the model compression efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 Schematic diagram of an image compression method based on multi-scale fusion of spatial and contextual information.

[0052] Figure 2 Schematic diagram of the multi-scale information fusion module.

[0053] Figure 3 Schematic diagram of the multi-scale three-dimensional context module.

[0054] Figure 4 The figure is a comparison chart of the effects of the present invention and several other methods.

[0055] Figure 5 The figure is a comparison chart of the effects of the present invention and several other methods. DETAILED DESCRIPTION

[0056] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0057] Example

[0058] The present invention provides an image compression method based on multi-scale spatial and contextual information fusion, such as Figure 1 As shown, the following steps are included:

[0059] 11) The training stage of the present invention uses the COCO2014 training set, all training images are randomly cropped to a size of 256x256, the batch size during training is set to 16, and Kodak24 is used as the test set for model training;

[0060] 12) In order to balance the relationship between bit rate and image reconstruction quality, the objective function of model training is set to:

[0061]

[0062] Among them, λ is a hyperparameter that balances bitrate and image reconstruction quality, and Respectively and bit rate, D(·) represents the difference between the original image and the reconstructed image, and MSE and MS-SSIM can be used as measurement criteria. When MSE is used to optimize the model, the evaluation criterion of the model is PSNR. At this time, the values ​​of λ are {0.0035, 0.0067, 0.0130, 0.0250, 0.0483} according to the bit rate. If MS-SSIM is used to optimize the model, the evaluation criterion of the model is MS-SSM, and the values ​​of λ are {4.58, 8.73, 16.64, 31.73, 60.50} according to the bit rate. In order to speed up the convergence of the model, the model is first pre-trained at a high bit rate, and then the value of λ is modified to adjust the bit rate of the model to other values. When training the pre-trained model, the learning rate is initially set to 10 -5 , and then every 100,000 iterations, the learning rate is reduced to 1 / 2 of the original. After that, when training models with other bit rates, the learning rate is initially set to 5×10 -5 , and then every 100,000 iterations, the learning rate drops to 1 / 2 of the original.

[0063] 13) The original image is extracted and downsampled by the residual block, attention module and multi-scale information fusion module to obtain the hidden feature y. In order to entropy encode y, it is quantized with a step size of 1 to obtain The above operations can be expressed as follows:

[0064]

[0065]

[0066] 14) The multi-scale information fusion module in the main encoder downsamples the feature y i times (i) And the feature y downsampled i+2 times (i+2) They are fused together in the form of an attention mechanism, as follows: Figure 2 In order to reduce the consumption of computing resources, the main encoder only uses two multi-scale information modules. This operation can be expressed by the following formula:

[0067] y (i+2) =y (i+2) +y (i+2) *sigmoid(Res(y (i) ))

[0068] Where Res(·) represents the residual block.

[0069] 21) The super-prior encoder calculates the super-prior feature z from the hidden feature, and then obtains it through quantization It is used to help extract spatial redundancy in hidden features and improve the accuracy of hidden feature probability distribution estimation. The above steps can be expressed by the following formula:

[0070]

[0071]

[0072] 22) The multi-scale context features calculated from the quantized hidden features using the multi-scale 3D context model are as follows: Figure 3 The above steps can be expressed by the formula:

[0073]

[0074] 23) After combining the contextual features and the super prior features, the super prior decoder is used to calculate the model parameters and weights of the Gaussian mixture model, which can be expressed as:

[0075]

[0076] 24) According to the weights, the three Gaussian functions are combined into a Gaussian mixture model as the entropy model, and the estimated probability distribution of the hidden features is calculated, which can be expressed as:

[0077]

[0078] 31) Since the hidden features after quantization are discrete, the derivative of the discrete function is 0 everywhere, which will cause the gradient to disappear during model training. In order to train the image compression model, during the training phase, the quantization process will be replaced by adding independent and identically distributed uniform noise.

[0079] 32) During the use of the model, the hidden features will be quantized, and the entropy model obtained based on the super priori automatic encoder can calculate the probability distribution of the features, and the quantized features are encoded using entropy coding, generally using arithmetic coding in entropy coding.

[0080] 41) The hidden features are transformed back into images through the residual block and upsampling in the main decoder. The above steps can be expressed as:

[0081]

[0082] 42) Compare the reconstructed image with the original image in terms of objective and subjective indicators to evaluate the compression and reconstruction effects of the model.

[0083] In order to verify the effectiveness of this method, this method is compared with traditional methods such as JPEG, JPEG2000, BPG, VCC and some end-to-end image compression methods. The Kodak24 public test set is used as the test data, and the original image and the reconstructed image after algorithm compression are compared. The difference in PSNR and MS-SSIM between the two indicators is calculated, and two curves are obtained respectively as shown in the figure below. Figure 4 and 5 shown.

[0084] The present invention uses a multi-scale information fusion module in the main encoder to avoid adding spatial redundancy in hidden features while retaining effective spatial information. The multi-scale three-dimensional context module used in the context model can fuse context information in different scale spaces to make the entropy model more accurate. The test results of the present invention on the Kodak24 public test set show that in terms of PSNR index, the effect of this method is 0.15dB higher than the latest traditional image compression standard VVC.

Claims

1. An image compression method based on multi-scale space and context information fusion, characterized in that: The method comprises the following steps: 1) Construct an image compression model based on multi-scale space and context information fusion, extract hidden features from the original image through the main encoder, and use a multi-scale information fusion module to reduce the loss of effective information in forward propagation; 2) The hyper-prior module combines the hyper-prior information with the multi-scale context information to obtain the parameters and weights of the three Gaussian functions, and obtains the probability distribution of the hidden features by adding the weights to obtain the Gaussian mixture model; 3) Based on the probability distribution of hidden features, the arithmetic encoder encodes and decodes the hidden features; 4) The main decoder reconstructs the hidden features into a picture and completes the image compression; The step 1) specifically comprises the following steps: 11) The original image is extracted and downsampled by the residual block, attention module and multi-scale information fusion module to obtain the hidden feature y. In order to entropy encode y, it is quantized with a step size of 1 to obtain the quantized hidden feature Then we have: Among them, x is the original image, θ ga is the parameter of the main encoder, Q(·) represents the quantization process, g a (·) indicates the main encoder; 12) The feature y downsampled i times is fused through the multi-scale information fusion module in the main encoder (i) And the feature y downsampled i+2 times (i+2) In order to reduce the consumption of computing resources, the main encoder only uses two multi-scale information modules through the form of attention mechanism, and then: Among them, Res(·) represents the residual block; The step 2) specifically comprises the following steps: 21) The super-prior encoder calculates the super-prior feature z from the hidden feature y, and then quantizes it to obtain the quantized super-prior feature To assist in extracting spatial redundancy in hidden features and improve the accuracy of hidden feature probability distribution estimation, we have: Among them, h a (·) represents the super prior encoder, are the parameters of the super prior encoder; 22) Use the multi-scale 3D context module to extract the hidden features from the quantized The multi-scale context features obtained in Then we have: Among them, downsample means downsampling, represents a 3D context model with a convolution kernel size of 5×5×5. represents a three-dimensional context model with a convolution kernel size of 7×7×7, Represents a three-dimensional context model with a convolution kernel size of 9×9×9; 23) Multi-scale context features and super prior features After the combination, the model parameters and weights of the Gaussian mixture model are solved using the super prior decoder, and then: Among them, ω i ,μ i , They represent the weight, mean and variance of the i-th Gaussian model in the Gaussian mixture model, represents the i-th super-prior decoder; 24) According to the weights, the three Gaussian functions are combined into a Gaussian mixture model as the entropy model, and the estimated probability distribution of the hidden features is calculated, then: in, Based on super prior features Hidden features of The conditional probability distribution of Based on the parameter ω i ,μ i The Gaussian probability distribution of For the range from arrive Uniformly distributed noise.

2. The image compression method based on multi-scale space and context information fusion according to claim 1, characterized in that: The step 3) specifically comprises the following steps: 31) To prevent the gradient vanishing phenomenon during model training, the quantization process is replaced by adding independent and identically distributed uniform noise during the image compression model training stage; 32) During the use of the image compression model, the hidden features are quantized, and the probability distribution of the features is calculated based on the entropy model obtained by the super-prior encoder, and the quantized features are encoded using arithmetic coding in entropy coding.

3. The image compression method based on multi-scale space and context information fusion according to claim 1, characterized in that: The step 4) specifically comprises the following steps: 41) The hidden features are transformed back into images through the residual block and upsampling in the main decoder, then: Among them, g s (·) indicates the main decoder, are the parameters of the main decoder, For the reconstructed picture; 42) The reconstructed image Compare the objective and subjective indicators with the original image x to evaluate the compression and reconstruction effects of the model.

4. The image compression method based on multi-scale space and context information fusion according to claim 3, characterized in that: In the step 42), the objective and subjective indicators include PSNR and MS-SSIM.

5. The image compression method based on multi-scale space and context information fusion according to claim 1, characterized in that: In the step 1), the image compression model based on the fusion of multi-scale space and context information consists of a main encoder, a super-prior encoder, a super-prior decoder, a main decoder and a multi-scale three-dimensional context module.

6. The image compression method based on multi-scale space and context information fusion according to claim 5, characterized in that: When training the image compression model, in order to balance the relationship between bit rate and image reconstruction quality, the training objective function is set to: Among them, λ is a hyperparameter that balances bit rate and image reconstruction quality, and They are the quantized hidden features. and the quantized hyper-prior features The bit rate of the model is , D(·) represents the difference between the original image and the reconstructed image. MSE and MS-SSIM are used as the measurement criteria. When MSE is used to optimize the model, the evaluation criterion of the model is PSNR. When MS-SSIM is used to optimize the model, the evaluation criterion of the model is MS-SSM.

7. The image compression method based on multi-scale space and context information fusion according to claim 6, characterized in that: To speed up the convergence of the model, pre-training is first performed at a high bit rate, and then the value of λ is modified to adjust the bit rate of the model to other values.

8. The image compression method based on multi-scale space and context information fusion according to claim 6, characterized in that: When training the pre-trained model, the learning rate decreases with the number of iterations. When training models with other bit rates, the initial value of the learning rate increases and decreases with the number of iterations.

Images