Color compressed image restoration method based on two-stage feedback convolutional neural network

Abstract

The application discloses a color compressed image restoration method based on a two-stage feedback convolutional neural network. Mainly includes the following steps: building a two-stage feedback convolutional neural network for restoring color compressed images; using the two-stage feedback convolutional neural network built in the previous step, training a color compressed image restoration model for different quality factors; inputting the color compressed image into the trained two-stage feedback convolutional neural network model, and outputting a high-quality color image. The method uses a feedback mechanism to use the high-level features extracted in the first stage to guide the network to better restore the color compressed image in the second stage, and shares the network weights of different stages to reduce the parameter amount, while using a curriculum learning strategy to encourage the network to preferentially process the luminance component containing more detailed information, and improve the effectiveness of the feedback mechanism. Therefore, the application achieves good performance in objective indicators and subjective effects, and is a high-efficiency color compressed image restoration method.

Description

Technical Field

[0001] This invention relates to color compressed image restoration technology, specifically to a color compressed image restoration method based on a two-stage feedback convolutional neural network, belonging to the field of digital image processing. Background Technology

[0002] Lossy image coding techniques can achieve high compression ratios, effectively saving transmission bandwidth and storage capacity, but inevitably leading to a degradation in image quality. Typically, the JPEG image coding method, by downsampling the chroma components by a factor of 2 in both height and width and independently quantizing the Discrete Cosine Transform (DCT) coefficients of each 8×8 image block, often introduces complex compression artifacts, including blockiness, ringing, blurring, and bleeding. Suppressing these compression artifacts to restore high-quality images not only improves the user's visual experience but also facilitates the processing and analysis of various computer vision methods that require compressed images as input. Therefore, research on compressed image restoration methods has significant practical application value.

[0003] Currently, researchers have proposed numerous methods for restoring compressed images. However, most methods only target grayscale compressed images, while color compressed images have broader practical applications. Therefore, this invention constructs a color compressed image restoration method based on a two-stage feedback convolutional neural network, achieving excellent suppression of color image compression artifacts. Summary of the Invention

[0004] This invention integrates the advantages of feedback mechanism, parameter sharing, and curriculum learning strategies, and designs a feedback module to construct a two-stage feedback convolutional neural network for efficient color compressed image restoration.

[0005] The color compressed image restoration method based on a two-stage feedback convolutional neural network proposed in this invention mainly includes the following steps:

[0006] (1) Construct a two-stage feedback convolutional neural network to restore the color compressed image;

[0007] (2) Using the two-stage feedback convolutional neural network in step one, train a color compressed image restoration model for different quality factors.

[0008] (3) Input the color compressed image into the trained two-stage feedback convolutional neural network model to output a high-quality color image. Attached Figure Description

[0009] Figure 1 This is a block diagram illustrating the principle of the color compressed image restoration method based on a two-stage feedback convolutional neural network according to the present invention.

[0010] Figure 2 This is a block diagram of the feedback module.

[0011] Figure 3 This is a block diagram of the implementation of the proximal operator.

[0012] Figure 4 This is a block diagram of the residual element.

[0013] Figure 5 For enhanced separable convolutional bounding boxes.

[0014] Figure 6 The g-th dual separable convolution block diagram for enhanced separable convolution.

[0015] Figure 7 The image shows a comparison of the results of restoring the color compressed image "Barbara" using the present invention and five other methods (quality factor 10): (a) is the original image, (b) is the JPEG compressed image, and (c)(d)(e)(f)(g)(h) are the restoration results of method 1, method 2, method 3, method 4, method 5 and the present invention, respectively. Detailed Implementation

[0016] The present invention will be further described below with reference to the accompanying drawings:

[0017] Figure 1 In this paper, the color compressed image restoration method based on a two-stage feedback convolutional neural network can be divided into the following steps:

[0018] (1) Construct a two-stage feedback convolutional neural network to restore the color compressed image;

[0019] (2) Using the two-stage feedback convolutional neural network in step one, train a color compressed image restoration model for different quality factors.

[0020] (3) Input the color compressed image into the trained two-stage feedback convolutional neural network model to output a high-quality color image.

[0021] Specifically, in step (1), the constructed two-stage feedback convolutional neural network is expanded into two stages through a feedback mechanism and parameter sharing, and simultaneously restores the color compressed image from the luminance channel to the RGB channel under the drive of the course learning strategy. In the first stage, the two-stage feedback convolutional neural network focuses on reconstructing the luminance channel, while in the second stage, it utilizes high-level features from the first stage to guide the restoration of the RGB color image. The two-stage feedback convolutional neural network mainly includes an input module, a feedback module, and an output module, where the input module and the feedback module share weights in different stages. Let X rgb For a color JPEG compressed image, its luminance channel X can be obtained through color space conversion. y and chroma channel X cbcr Since the chroma channel may contain information that helps in reconstructing the luminance channel, X... y and X cbcr The inputs are cascaded along the channel dimension as input for the first stage. First, the input module extracts low-level features from the color JPEG compressed image. and Expressed as a formula:

[0022]

[0023]

[0024] Among them, H FE (·) represents the feature extraction function, H Concat (·) indicates a channel cascading operation. Next, and Input feedback blocks to extract high-level features and Expressed as a formula:

[0025]

[0026]

[0027] Among them, H FB (·) indicates the feedback module function. Finally, the output module and long-spin connection are used to reconstruct the luminance channel. or RGB color image Expressed as a formula:

[0028]

[0029]

[0030] in, and This indicates a rebuild operation.

[0031] The input and output modules are implemented as a single convolutional layer with a kernel size of 3×3. For example... Figure 2 As shown, the feedback module includes an initial enhancement module and an adaptive iterative self-enhancement module. First, the initial enhancement module fuses low-level features F... LL and high-level features F HL Generate enhanced low-level features It can also be viewed as initial high-level features The initial enhancement module is implemented as a convolutional layer with a kernel size of 1×1. Then, the adaptive iterative self-enhancement module further processes the data. To obtain the final high-level features This process can be expressed by the following formula:

[0032]

[0033] in, This represents the prior term weighted by the tradeoff parameter λ. To facilitate optimization, this invention uses auxiliary variables... By introducing the above equation, we get:

[0034]

[0035] Where μ is the penalty parameter, the above equation is then solved through the following iterative steps:

[0036]

[0037]

[0038] Where i represents the iteration index from 1 to t. The optimization problems involved in the above two formulas are respectively about Proximal operators and about The quadratic optimization function can be solved implicitly through a convolutional neural network and explicitly through the zeros of the first derivative to obtain a closed-form solution. This invention implements the near-end operator as n residual units, such as... Figure 3 As shown. Therefore, the above iterative steps can be written as:

[0039]

[0040]

[0041] in, and Let i represent the functions of the i-th proximal operator and the quadratic optimizer, respectively, and let their outputs be expressed as follows: and Let α represent the nth residual unit of the i-th proximal operator, where α = 1 / (1+μ), which can be adaptively learned during network training.

[0042] Figure 4 The structure of the residual unit is shown, which mainly consists of a short-jump connection and two enhanced separable convolutions (ESConv). Figure 5 It demonstrates an enhanced separable convolution architecture, incorporating a Grouped Multi-scale Dual Depth-wise Convolution (GMD). 2 A multi-scale dual point-wise convolution (WGDPConv) and a wide GELU-activated dual point-wise convolution (WGDPConv) are performed. First, a grouped multi-scale dual depthwise convolution is applied to the input tensor with c channels. Divide the channel into g groups equally along the channel dimension, as expressed by the formula:

[0043]

[0044] Among them, H G (·) indicates a channel grouping operation. Let g represent the feature tensor of the g-th group. Then, the feature tensors of different groups are input into g dual depthwise convolutions (D...) with different kernel sizes. 2 Conv) is used to extract features at multiple scales. Finally, the outputs of g dual depthwise convolutions are concatenated along the channel dimension to obtain the multi-scale features. This process can be expressed by the following formula:

[0045]

[0046] in, Let represent the g-th dual depthwise convolution, which consists of two depthwise convolutions (DWConv) with kernel sizes of (2g+1)×1 and 1×(2g+1) respectively, as shown below. Figure 6 As shown. The wide GELU activation dual pointwise convolution consists of two pointwise convolutions (PWConv) and one GELU activation function. Its operation is expressed by the following formula:

[0047]

[0048] in, H represents the output of the enhanced separable convolution. GELU (·) represents the GELU activation function. This represents a pointwise convolution where the number of output channels is r times the number of input channels. This represents a pointwise convolution where the number of output channels is 1 / r times the number of input channels.

[0049] This invention trains a two-stage feedback convolutional neural network by minimizing the two-domain absolute value error loss function, thereby implementing a learning strategy that prioritizes restoring the luminance channel, which contains more detailed information, before restoring the RGB color compressed image. This can be expressed by the following formula:

[0050]

[0051]

[0052]

[0053] Where, λ l It is a trade-off parameter. and These represent the calculation of absolute value error loss in the pixel domain and DCT domain, respectively, where N is the number of training samples in each batch, and γ(·) represents the DCT operation. and Let i represent the i-th uncompressed RGB color image and the luminance channel, respectively.

[0054] This invention uses t, n, c, g, r, and λ respectively. l And N is set to 4, 2, 64, 4, 2, 0.05 and 16.

[0055] To verify the superiority of this invention, the following five typical image compression restoration methods based on convolutional neural networks were selected for comparison:

[0056] Method 1: The method proposed by Dong et al., see reference "Dong C, Deng Y, Change Loy C, et al. Compression artifacts reduction by a deep convolutional network[C]. IEEE International Conference on Computer Vision, 2015:576-584."

[0057] Method 2: The method proposed by Zhang et al., see reference "Zhang K, Zuo W, Chen Y, et al. Beyonda gaussian denoiser: Residual learning of deep cnn for image denoising[J].IEEE Transactions on Image Processing,2017,26(7):3142-3155."

[0058] Method 3: The method proposed by Chen et al., see reference "Chen H, He X, An C, et al. Deep wide-activated residual network based joint blocking and color bleeding artifacts reduction for 4:2:0 JPEG-compressed images[J].IEEE Signal Processing Letters,2018,26(1):79-83."

[0059] Method 4: The method proposed by Li et al., see reference "Li J, Wang Y, Xie H, et al. Learning asingle model with a wide range of quality factors for JPEG image artifacts removal[J].IEEE Transactions on Image Processing,2020,29:8842-8854."

[0060] Method 5: The method proposed by Ehrlich et al., see reference "Ehrlich M, Davis L, Lim SN, et al. Quantization guided jpeg artifact correction[C] / / European Conference on Computer Vision. Springer, Cham, 2020:293-309.".

[0061] The comparative experiments are as follows: First, the save function of the PIL (Python Image Library) module was used to perform JPEG compression on color images from the commonly used benchmark image library Classic5 with quality factors (QF) of 10, 20, 30, and 40, respectively. Then, the five comparative methods mentioned above and the present invention were used to restore the color JPEG compressed images. Table 1 shows the average evaluation metrics of the five comparative methods and the present invention on the Classic5 dataset. Since the authors of methods 1 and 2 only provided models for restoring grayscale JPEG compressed images, these two models were retrained on color RGB images for fair comparison. The authors of method 3 did not provide training models with QF of 30 and 40, so Table 1 only shows the test results of this method with QF of 10 and 20. PSNR (Peak Signal to Noise Ratio), SSIM (Structure Similarity Index), and PSNR-B were used as objective metrics to evaluate the restoration results. The higher the values ​​of these three metrics, the better the quality of the restored color compressed image. In addition, for subjective visual comparison, Figure 7 The restored image "Barbara" with a QF of 10 is presented. The restored images of the original image, the JPEG compressed image, and the images restored using methods 1, 2, 3, 4, 5, and the color compressed image of this invention are shown below. Figure 7 (a) Figure 7 (b) Figure 7 (c) Figure 7 (d) Figure 7 (e) Figure 7 (f) Figure 7 (g) and Figure 7 As shown in (h).

[0062] Table 1

[0063]

[0064] As can be seen from Table 1, compared with the five comparison methods, this invention achieved the highest average PSNR, SSIM, and PSNR-B values ​​across all quality factors. Figure 7 As can be seen, compared with the JPEG compressed image restoration results of methods 1, 2, 3, 4 and 5, the restored image output by the present invention contains fewer compression artifacts and clearer texture structure, and is more visually similar to the original image.

[0065] In summary, compared to the five comparison methods, the color compressed image restoration effect of the present invention has significant advantages in both subjective and objective evaluation. Therefore, the present invention is a highly efficient method for color compressed image restoration.

Claims

1. A method for restoring color compressed images based on a two-stage feedback convolutional neural network, characterized in that... Includes the following steps: Step 1: Construct a two-stage feedback convolutional neural network to restore the color compressed image. It is divided into two stages through feedback mechanism and parameter sharing, and restores the color compressed image in the manner of luminance channel to RGB channel under the drive of course learning strategy. In the first stage, the two-stage feedback convolutional neural network focuses on reconstructing the luminance channel, while in the second stage, it utilizes high-level features from the first stage to guide RGB color image restoration. The two-stage feedback convolutional neural network mainly consists of an input module, a feedback module, and an output module, where the input and feedback modules share weights at different stages. Let X... rgb For a color JPEG compressed image, its luminance channel X can be obtained through color space conversion. y and chroma channel X cbcr Since the chroma channel may contain information that helps in restoring the luminance channel, X... y and X cbcr The inputs are cascaded along the channel dimension as input for the first stage; firstly, the input module extracts low-level features from the color JPEG compressed image. and Expressed as a formula: Among them, H FE (·) represents the feature extraction function, H Concat (·) indicates a channel cascading operation; then, and Input feedback blocks to extract high-level features and Expressed as a formula: Among them, H FB (·) indicates the feedback module function; finally, the output module and long-spin connection are used to reconstruct the luminance channel. or RGB color image Expressed as a formula: in, and This represents the reconstruction operation; when training a two-stage feedback convolutional neural network, a learning strategy is implemented by minimizing the two-domain absolute value error loss function to prioritize the restoration of the luminance channel, which contains more detailed information, before restoring the RGB color compressed image. This strategy is expressed by the formula: Where, λ l It is a trade-off parameter. and These represent the calculation of absolute value error loss in the pixel domain and DCT domain, respectively, where N is the number of training samples in each batch, and γ(·) represents the DCT operation. and These represent the i-th uncompressed RGB color image and the luminance channel, respectively. Step 2: Using the two-stage feedback convolutional neural network from Step 1, train color compressed image restoration models for different quality factors. Step 3: Input the color compressed image into the trained two-stage feedback convolutional neural network model to output a high-quality color image.

2. The color compressed image restoration method based on a two-stage feedback convolutional neural network according to claim 1, characterized in that... The feedback module in step one includes an initial enhancement module and an adaptive iterative self-enhancing module; firstly, the initial enhancement module fuses low-level features F LL and high-level features F HL Generate enhanced low-level features It can be regarded as the initial high-level features Next, the adaptive iterative self-enhancing module alternately uses t proximal operators and t quadratic optimizers to further process the data. To obtain the final high-level features This process can be expressed by the following formula: Where i represents the iteration index from 1 to t. and Let i represent the functions of the i-th proximal operator and the quadratic optimizer, respectively, and let their outputs be expressed as follows: and Let represent the nth residual unit of the i-th proximal operator, and α be an adaptively learnable parameter.

Images