Image decompression effect method based on enhanced separable convolutional network

Abstract

This invention discloses an image decompression method based on an enhanced separable convolutional network. The main steps include: constructing a convolutional neural network (CNN) for removing image compression effects using enhanced separable convolutions as the primary building blocks; training image decompression models under different quality factors using the CNN built in the previous step; inputting the compressed image into the trained image decompression models, and outputting the decompressed image. The method described in this invention has fewer model parameters and lower computational cost, effectively mining and fusing multi-scale information of the image, and allowing multi-scale information to flow more efficiently from shallow to deep networks for accurate pixel value prediction. Therefore, this invention achieves good subjective and objective results and is an efficient image decompression method.

Description

Technical Field

[0001] This invention relates to image decompression techniques, specifically to an image decompression method based on an enhanced separable convolutional network, belonging to the field of digital image processing. Background Technology

[0002] Lossy image compression algorithms can effectively save transmission bandwidth and storage space and have been widely used in various mobile devices and systems. However, lossy compression inevitably leads to a decrease in image quality. For example, the JPEG coding standard typically coarsely quantizes the discrete cosine transform coefficients of non-overlapping 8×8 image blocks, thereby introducing various compression effects. These compression effects not only degrade the visual experience but also significantly increase the difficulty of subsequent computer vision tasks such as object detection, semantic segmentation, and super-resolution reconstruction.

[0003] To remove image compression artifacts and improve image quality, researchers have proposed numerous methods. These methods mainly include enhancement-based, reconstruction-based, and learning-based methods. Among them, deep learning-based methods have received increasing attention and research due to their superior performance. However, most existing deep learning-based methods have a large number of model parameters and computational cost, making them difficult to deploy on mobile devices and systems. Therefore, this invention constructs an enhanced separable convolutional network that achieves good image decompression effects while significantly reducing the number of model parameters and computational cost. Summary of the Invention

[0004] This invention further develops the advantages of depth-wise separable convolution (DWSConv) by designing an enhanced separable convolution (ESConv), thereby constructing a lightweight deep convolutional neural network for efficient image decompression effects.

[0005] The image decompression method based on enhanced separable convolutional networks proposed in this invention mainly includes the following steps:

[0006] (1) Convolutional neural networks for removing image compression effects are constructed using enhanced separable convolutions as the main building blocks.

[0007] (2) Using the convolutional neural network in step one, train image decompression effect models under different quality factors respectively;

[0008] (3) Input the compressed image into the trained image decompression effect model and output the decompression effect image. Attached Figure Description

[0009] Figure 1 This is a block diagram illustrating the principle of the image decompression method based on enhanced separable convolutional networks of this invention.

[0010] Figure 2 Block diagram of the enhanced weighted residual module.

[0011] Figure 3 For enhanced separable convolutional bounding boxes.

[0012] Figure 4 This is a comparison chart (quality factor 10) of the decompression effect results of the compressed image "Bikes" by the present invention and five other methods: where (a) is the original image, (b) is the JPEG compressed image, and (c)(d)(e)(f)(g)(h) are the decompression effect results of method 1, method 2, method 3, method 4, method 5 and the present invention, respectively. Detailed Implementation

[0013] The present invention will be further described below with reference to the accompanying drawings:

[0014] Figure 1 In this context, the image decompression method based on enhanced separable convolutional networks can be divided into the following steps:

[0015] (1) Convolutional neural networks for removing image compression effects are constructed using enhanced separable convolutions as the main building blocks.

[0016] (2) Using the convolutional neural network in step one, train image decompression effect models under different quality factors respectively;

[0017] (3) Input the compressed image into the trained image decompression effect model and output the decompression effect image.

[0018] Specifically, in step (1), the constructed enhanced separable convolutional network model mainly consists of three parts: low-level feature extraction, high-level feature extraction, and image reconstruction. Low-level feature extraction and image reconstruction are both implemented by a 3×3 convolutional layer with a kernel size of 3×3, while high-level feature extraction is implemented by a weighted shared source residual group.

[0019] The weighted shared-source residual group consists of n enhanced weighted residual blockgroups (IWRBGs) with weighted share-source skip connections (WSSCs), which can be expressed by the formula:

[0020]

[0021] in, F LL F HL Let represent the input of the nth enhanced weighted residual module group, the low-level features extracted by the 3×3 convolutional layer, and the high-level features extracted by the weighted shared source residual group, respectively. This represents the operation of the nth enhanced weighted residual module group. These represent the weighting coefficients of the nth enhanced weighted residual block group and its shared source hop connection, respectively. Each enhanced weighted residual block group contains m enhanced weighted residual blocks (IWRBs). Figure 2 The structure of each enhanced weighted residual module is shown, and can be expressed by the formula:

[0022]

[0023] in, These represent the input and output of the enhanced weighted residual module, β and β, respectively. res β ssc H represents the weighting coefficients of the residual branch and the jump connection branch of the enhanced weighted residual module, respectively. ESC (·) indicates an enhanced separable convolution operation.

[0024] Figure 3 This paper showcases an enhanced separable convolution architecture, comprising two parts: Grouped multi-scale dual depth-wise convolution (GMDDConv) and Wide-activated dual point-wise convolution (WDPConv). GMDDConv allows us to extract rich multi-scale information using network structures with many branches and large kernels, while increasing the number of parameters and computational cost by only a small amount. WDPConv enhances the inter-channel information fusion, allowing more feature information to pass through the ReLU activation function while maintaining a high degree of nonlinearity. Specifically, GMDDConv first processes an input tensor with c channels... Dividing the data into g groups along the channel dimension can be expressed by the following formula:

[0025]

[0026] Among them, H G (·) indicates a grouping operation. Let g represent the feature tensor of the g-th group. Next, the grouped feature tensors are fed into the corresponding dual depth-wise convolution (DDConv) to extract multi-scale features. Finally, the multi-scale features are concatenated along the channel dimension to obtain the output of the grouped multi-scale dual depth-wise convolution. This process can be expressed by the following formula:

[0027]

[0028] Among them, H Concat (·) indicates a channel cascading operation. Let represent the g-th dual depthwise convolution. Each dual depthwise convolution consists of two cascaded depthwise convolutions (DWConv) with kernel sizes of k×1 and 1×k, respectively, where k = 2g + 1.

[0029] Wide activation dual pointwise convolution consists of two pointwise convolutions (PWConv) and an activation function ReLU. Its operation can be expressed by the following formula:

[0030]

[0031] in, Let σ(·) represent the input and output of the wide activation dual pointwise convolution, respectively, and let σ(·) represent the activation function ReLU. This represents a pointwise convolution where the number of output channels is r times the number of input channels (r≥1). This represents a pointwise convolution where the number of output channels is 1 / r times the number of input channels.

[0032] It is worth noting that grouped multi-scale dual depthwise convolution and wide activation dual pointwise convolution can mutually promote each other, making the enhanced separable convolution more efficient. On the one hand, grouped multi-scale dual depthwise convolution can provide more complementary features, making the nonlinear feature fusion of wide activation dual pointwise convolution more effective. On the other hand, wide activation dual pointwise convolution can enhance the nonlinear fusion of features, allowing the multi-scale information extracted by grouped multi-scale dual depthwise convolution to be fully utilized.

[0033] This invention uses the absolute value error criterion as the loss function for training and enhancing separable convolutional network models, expressed by the formula:

[0034]

[0035] Where Θ represents the parameters of the network model, H(·) represents the operation of the enhanced separable convolutional network model, and Y i Xi Let represent the i-th compressed image and its corresponding original image, respectively, and N be the number of training samples in each batch, which is set to 16 in this invention. Furthermore, the hyperparameters n, m, c, g, and r involved in the enhanced separable convolutional network proposed in this invention are set to 4, 2, 64, 4, and 2, respectively.

[0036] To verify the effectiveness of this invention, extensive comparative experiments were conducted on the commonly used test image libraries Classic5 (containing 5 images) and LIVE1 (containing 29 images). In the experiments, this invention was compared with five typical image decompression techniques based on convolutional neural networks. These five image decompression techniques used for comparison are:

[0037] Method 1: The method proposed by Dong et al., see reference "Dong C, Deng Y, Change Loy C, et al. Compression artifacts reduction by a deep convolutional network[C]. IEEE International Conference on Computer Vision, 2015:576-584."

[0038] Method 2: The method proposed by Zhang et al., see reference "Zhang K, Zuo W, Chen Y, et al. Beyonda gaussian denoiser: Residual learning of deep cnn for image denoising[J].IEEE Transactions on Image Processing,2017,26(7):3142-3155."

[0039] Method 3: The method proposed by Tai et al., see reference "Tai Y, Yang J, Liu X, et al. Memnet: A persistent memory network for image restoration[C]. IEEE international conference on computer vision, 2017:4539-4547."

[0040] Method 4: The method proposed by Chen et al., see reference "Chen H, He X, Qing L, et al. DPW-SDNet: Dual pixel-wavelet domain deep CNNs for soft decoding of JPEG-compressedimages[C]. IEEE Conference on Computer Vision and Pattern Recognition Workshops,2018:711-720."

[0041] Method 5: The method proposed by Sun et al., see reference "Sun M, He X, Xiong S, et al. Reduction of JPEG compression artifacts based on DCT coefficients prediction[J].Neurocomputing,2020,384:335-345."

[0042] The comparative experiment is as follows:

[0043] Experiment 1: First, the `save` function of the PIL (Python Image Library) module was used to perform JPEG compression on five test images from the Classic5 dataset with quality factors (QF) of 10, 20, 30, and 40, respectively. Then, methods 1 to 5, as well as this invention, were used to decompress the compressed images. Table 1 shows the average evaluation metrics of the comparison methods and this invention on the Classic5 dataset. Note that the authors of method 3 did not provide the training model for QFs of 30 and 40, so Table 1 only shows the test results for QFs of 10 and 20. Objective evaluation metrics included PSNR (PeakSignal to Noise Ratio), SSIM (Structure Similarity Index), and PSNR-B. Higher values ​​for these three metrics indicate better quality of the decompressed image.

[0044] Table 1

[0045]

[0046] Experiment 2: First, the `save` function of the PIL (Python Image Library) module was used to perform JPEG compression on 29 test images from the LIVE1 dataset with quality factors (QF) of 10, 20, 30, and 40, respectively. Then, methods 1 to 5, as well as the present invention, were used to remove compression effects from the compressed images. Table 2 shows the average PSNR, SSIM, and PSNR-B values ​​obtained by the comparison methods and the present invention on the LIVE1 test image library. Furthermore, for subjective visual comparison, Figure 4 The decompression effect results for the image "Bikes" in the LIVE1 test image library are presented at a QF of 10. The decompression effect results for the original image, the JPEG compressed image, and methods 1, 2, 3, 4, 5, and the present invention are shown below. Figure 4 (a) Figure 4 (b) Figure 4 (c) Figure 4 (d) Figure 4 (e) Figure 4 (f) Figure 4 (g) and Figure 4 As shown in (h).

[0047] Table 2

[0048]

[0049] Experiment 3 uses the `profile` function from the PyTorch third-party library `thop` to calculate the model parameters `Params` and `Mult-Adds` of methods 1 to 5, as well as the model of this invention, to compare their model complexity. When calculating the `Mult-Adds`, the resolution of the input compressed image is assumed to be 1280×720. Table 3 shows the calculation results.

[0050] Table 3

[0051] Evaluation indicators Method 1 Method 2 Method 3 Method 4 Method 5 This invention Params(K) 106 667 677 1343 3002 277 Mult-Adds(G) 98.2 614.8 2665.0 309.6 686.5 255.9

[0052] As can be seen from Tables 1 and 2, compared with the other five image decompression methods, except that the average PSNR-B value of this invention is slightly lower than that of method 5 when the quality factor QF of the Classic5 dataset is 40, this invention achieves the highest average PSNR, SSIM, and PSNR-B values ​​in all other cases. Figure 4As can be seen, JPEG images suffer from severe compression effects such as fragmentation, ringing, and blurring. While methods 1, 2, 3, 4, and 5 effectively suppress these compression effects, some blurring remains, and some lost details are not well recovered. In contrast, this invention effectively removes the compression effects of JPEG images while restoring a clearer texture structure, achieving a visual effect closer to the original image. Table 3 shows that this invention has lower model complexity compared to methods 2, 3, 4, and 5.

[0053] In summary, compared to the other five methods, the image decompression effect method of this invention has significant advantages in both subjective and objective evaluation, and the model complexity of this invention is relatively low. Therefore, this invention is a highly efficient image decompression effect method.

Claims

1. A method for image decompression effect based on enhanced separable convolutional network, characterized in that Includes the following steps: Step 1: Using enhanced separable convolutions as the main building blocks, build a convolutional neural network model to remove image compression effects. This network model mainly includes three parts: low-level feature extraction, high-level feature extraction, and image reconstruction. Step 2: Using the convolutional neural network model from Step 1, train image decompression effect models under different quality factors respectively; Step 3: Input the compressed image into the trained image decompression effect model, and output the decompression effect image; For the network model in step one, low-level feature extraction and image reconstruction are both implemented by a convolution layer with a kernel size of 3x3 , and high-level feature extraction is implemented by weighted shared source residual groups; The weighted shared source residual group is composed of An enhanced weighted residual module group with a weighted shared source skip connection, which is expressed by a formula: in, , , They represent the first An enhanced weighted residual module group's input, convolutional layer Low-level features extracted by conv, and high-level features extracted by weighted shared source residual groups. Indicates the first The operation of an enhanced weighted residual module group, , They represent the first A set of enhanced weighted residual modules and their shared source hop connections, each enhanced weighted residual module group containing... Each enhanced weighted residual module has a structure, expressed by the formula: in, , These represent the input and output of the enhanced weighted residual module, respectively. , These represent the weighting coefficients of the residual branches and jump connection branches in the enhanced weighted residual module, respectively. This indicates an enhanced separable convolution operation; The enhanced separable convolution structure consists of two parts: grouped multi-scale dual depthwise convolution and wide activation dual pointwise convolution. The grouped multi-scale dual depthwise convolution first divides the channel number into two parts. Input tensor Divided along the channel dimension Group, this process can be expressed by the formula: in, Indicates grouping operation. Indicates the first Grouped feature tensors; these grouped feature tensors are fed into corresponding dual depthwise convolutions to extract multi-scale features; the multi-scale features are concatenated along the channel dimension to obtain the output of the grouped multi-scale dual depthwise convolution. This process can be expressed by the following formula: in, This indicates a channel cascading operation. Indicates the first Each dual depthwise convolution consists of two cascaded depthwise convolutions with kernel sizes of [sizes missing]. and ,in ; Wide activation dual pointwise convolution consists of two pointwise convolutions and an activation function ReLU. Its operation is expressed by the following formula: in, , Let represent the input and output of the wide activation dual pointwise convolution, respectively. ReLU represents the activation function. This indicates that the number of output channels is equal to the number of input channels. times Pointwise convolution, This indicates that the number of output channels is equal to the number of input channels. A pointwise convolution of multiples.

2. The image decompression method based on enhanced separable convolutional networks according to claim 1, characterized in that... The enhanced separable convolution in step one mainly includes a grouped multi-scale dual depthwise convolution and a wide activation dual pointwise convolution. The grouped multi-scale dual depthwise convolution is responsible for extracting rich multi-scale spatial features, while the wide activation dual pointwise convolution is dedicated to effectively nonlinearly fusing multi-scale spatial features. The grouped multi-scale dual depthwise convolution and the wide activation dual pointwise convolution complement each other and promote each other, further enhancing the efficiency of the enhanced separable convolution.

3. The image decompression method based on enhanced separable convolutional networks according to claim 1, characterized in that... The grouped multi-scale dual depthwise convolution in step one splits a depthwise convolution into several dual depthwise convolutions with different kernel sizes to extract rich multi-scale spatial features, while having the characteristics of low parameter quantity and low computational cost.

Images