Real image super-division variable rate method based on degradation prediction

Abstract

The invention discloses a real image super-division variable rate method based on degradation prediction. Firstly, a data set is acquired and preprocessed; extracting degradation feature information from the low-resolution image through a degradation prediction network; and the obtained degradation feature information and the low-resolution image are sent to a reconstruction network for enhancement, and a final high-resolution image is generated. According to the method, a degradation perception reconstruction module is used and embedded into a Transform architecture, so that high-level semantic information can be better mined and texture details can be recovered by utilizing learned degradation information and through a self-attention mechanism; degradation features of the LR image can be predicted from the LR image, a real degraded low-resolution image is processed, and the method is not limited by synthesis degradation.

Description

Technical Field

[0001] The invention belongs to the field of computer vision, and in particular relates to a real image super-resolution variation method based on degradation prediction. Background Art

[0002] Image super-resolution is a long-standing problem in the research community, which aims to reconstruct high-resolution images from low-resolution images. It has a wide range of applications in image restoration and medical imaging. In recent years, super-resolution methods based on convolutional neural networks have made significant progress with the development of deep learning technology. Researchers have achieved good reconstruction results on synthetic datasets by enhancing the network representation capabilities.

[0003] However, actual degradation usually involves multiple factors such as blur, noise, and downsampling, which goes far beyond the simple downsampling assumption. Although existing single image super-resolution methods based on convolutional neural networks have shown good results on synthetic benchmark datasets, these methods are difficult to generalize to real-world low-resolution images due to their reliance on specific degradation scenarios, and their performance will be severely degraded when faced with practical applications. Therefore, there are still major challenges in solving the problem of super-resolution of actual degraded images. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present invention provides a real image super-resolution variation method based on degradation prediction. The present invention aims to solve how to predict degradation from a real image and reconstruct a high-resolution image based on the extracted degradation features.

[0005] A real image super-resolution rate-of-change method based on degradation prediction comprises the following steps:

[0006] Step 1: Dataset acquisition and preprocessing.

[0007] Step 2: Extract degradation feature information from the low-resolution image through the degradation prediction network.

[0008] Step 3: Send the obtained degradation feature information and the low-resolution image to the reconstruction network for enhancement to generate the final high-resolution image.

[0009] Step 4: Train a real image super-resolution variability network based on degradation prediction based on the preprocessed dataset.

[0010] The real image super-resolution variability network consists of two parts: a degradation prediction network and a reconstruction network.

[0011] Step 5: Test. Input the test set images into the trained real image super-resolution network to obtain the reconstructed high-resolution image SR, and compare it with the true value GT to calculate various evaluation indicators.

[0012] The following is a detailed description of the above steps:

[0013] Preferably, step 1 is as follows:

[0014] The real super-resolution image dataset RealSR is used for training and testing. RealSR consists of 3147 images collected from 559 scenes. 400 pairs of low-resolution and high-resolution (LR-GT) images at a scale of 2× (2 times magnification) are used for training, and 100 pairs of images are used for testing.

[0015] The preprocessing of the training set includes: rotating the images 30°, 45°, and 60° clockwise and counterclockwise, flipping them horizontally and vertically, and cropping the training low-resolution images to 48×48.

[0016] No special processing is performed on the test set images.

[0017] Preferably, the degradation prediction network described in step 2 is composed of a GT degradation prediction network and an LR degradation prediction network, and only the LR degradation prediction network participates in the test. In the GT degradation prediction network, the true value GT is concat (channel splicing) with the low-resolution image LR after pixel unshuffle downsampling, and the obtained output is sequentially passed through 9 ResBlocks (residual blocks), Conv (convolution)-Relu (activation function)-Conv (convolution)-Relu (activation function)-AvgPool (adaptive average pooling) to obtain the representation, and the low-resolution image LR is passed through 9 ResBlocks (residual blocks), Conv (convolution)-Relu (activation function)-Conv (convolution)-Relu (activation function)-AvgPool (adaptive average pooling) to obtain the representation, and then processed through the Liner (linear layer)-Relu (activation function)-Liner (linear layer)-Relu (activation function) layer to obtain the GT degradation prediction feature. In the LR degradation prediction network, the low-resolution image LR is sequentially processed through 9 ResBlocks (residual blocks), Conv (convolution)-Relu (activation function)-Conv (convolution)-Relu (activation function)-AvgPool (adaptive average pooling), and the representation obtained is processed through the Liner (linear layer)-Relu (activation function)-Liner (linear layer)-Relu (activation function) layer to obtain the LR degradation prediction feature.

[0018] Preferably, the input of the reconstruction network in step 3 is the low-resolution image LR and the degradation prediction feature. LR first passes through a convolutional layer to extract the low-resolution feature, and then passes through N degradation-aware reconstruction modules, wherein the input of the first reconstruction block is the degradation prediction feature and the low-resolution feature, and the input of the subsequent reconstruction block is the degradation prediction feature and the output feature of the previous reconstruction block, and then passes through a convolutional layer and a PixelShuffle (up-sampling block) for up-sampling to obtain the final output, i.e., the reconstructed image SR. The reconstruction module can be expressed by the following formula:

[0019]

[0020] Where D represents the degradation prediction feature, DA represents DABlock (degradation perception module), F is the output of the reconstruction module, and X is the input of the reconstruction module. The specific operation of DABlock (degradation perception module) is as follows: the degradation prediction feature passes through the Liner layer and then multiplies the low-resolution feature or the output feature of the previous reconstruction block through the Norm layer to obtain a normalized vector , and then processed by a multi-head self-attention network. Specifically, for the normalized vector , by applying 1×1 convolution to aggregate pixel-level cross-channel context, and then applying 3×3 depth-wise convolution to encode channel-level spatial context, generating three matrices: query vector matrix, key vector matrix, and value vector matrix:

[0021]

[0022]

[0023]

[0024] in, is a 1×1 point-wise convolution, It is a 3×3 depth-wise (depth-separable) convolution. The three matrices are then fed into the multi-head attention mechanism operation to calculate the attention between each other:

[0025]

[0026] in is the obtained attention value, Q, K, V are the query vector matrix, key vector matrix and value vector matrix respectively. The output of input Y after passing through the multi-head self-attention network is as follows:

[0027]

[0028] Finally, the output of the multi-head self-attention network is processed by the Norm layer and then multiplied pixel-wise with the features of the degradation prediction features after the Liner layer. Then, it passes through a feedforward network to obtain the output of the DABlock (degradation perception module), which is expressed as follows:

[0029]

[0030] in represents a feed-forward network, is the feature of the input FFN, and D is the degradation prediction feature.

[0031] Preferably, step 4 is as follows:

[0032] The preprocessed training set images are input into the real image super-resolution variability network based on degradation prediction. The training is carried out in two stages. In the first stage, GT and LR are input into the GT degradation prediction network, and the obtained GT degradation feature information is sent to the reconstruction network for enhancement. The loss of GT and reconstructed image SR is calculated using the L1 loss function, and back propagation is performed; the first stage training stops when convergence.

[0033]

[0034] In the second stage, based on the weights trained in the first stage, the GT degradation prediction network is frozen and the LR is input into the LR degradation prediction network to obtain the degradation features. , input GT and LR into the GT degradation prediction network to obtain , and The loss is calculated by the L1 loss function, and It is sent to the reconstruction network for enhancement, and the L1 loss function is used to calculate the loss of GT and reconstructed image SR, and back propagation is performed.

[0035]

[0036]

[0037] The loss function at this stage is ,in, is the pixel loss, is the degradation loss. The network is optimized by the optimizer Adam, and the exponential decay rates β1 and β2 are set to 0.9 and 0.999 respectively, and the initial learning rate is 0.0001. After multiple rounds of training, the final network model is obtained.

[0038] Preferably, the evaluation index in step 5 is as follows:

[0039] The evaluation index is PSNR, which is a commonly used index to measure the quality of images or videos, especially in the quality evaluation of compressed and restored images. Its calculation formula is as follows:

[0040]

[0041] Where MAX is the maximum pixel value in the image. MSE is the mean square error, which represents the average value of the difference between the original image and the processed image. The calculation formula is:

[0042]

[0043] in, and Respectively represent the original image and the compressed image at position The pixel value of .

[0044] The beneficial effects of the present invention are as follows:

[0045] The present invention uses a degradation-aware reconstruction module and embeds it into the Transformer architecture, which can utilize the learned degradation information and better mine high-level semantic information and restore texture details through the self-attention mechanism. It can predict the degradation characteristics from the LR image and process the real degraded low-resolution image without being restricted by synthetic degradation. At the same time, the generated reconstructed image can have higher quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 This is the GT degradation prediction network structure diagram.

[0047] Figure 2 This is the LR degradation prediction network structure diagram.

[0048] Figure 3 This is the network structure diagram of ResBlock.

[0049] Figure 4 To reconstruct the network structure diagram.

[0050] Figure 5 This is the network structure diagram of DABlock. DETAILED DESCRIPTION

[0051] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0052] A real image super-resolution rate-of-change method based on degradation prediction comprises the following steps:

[0053] Step 1: Preprocessing of the dataset.

[0054] The real super-resolution image dataset RealSR is used for training and testing. The present invention uses 400 pairs of (LR-GT) images of 2× scale in the dataset for training and 100 pairs of images for testing.

[0055] The preprocessing of the training set includes: rotating the images 30°, 45°, and 60° clockwise and counterclockwise, flipping them horizontally and vertically, and cropping the training low-resolution images to 48×48.

[0056] No special processing is performed on the test set images.

[0057] Step 2: Extract degradation feature information from the low-resolution image through the degradation prediction network.

[0058] The degradation prediction network consists of a GT degradation prediction network and an LR degradation prediction network. Only the LR degradation prediction network is involved in the test. The GT degradation prediction network is as follows: Figure 1 As shown in the figure, the true value GT is concat (channel splicing) with the low-resolution image LR after Pixel unshuffle downsampling, and the output is passed through 9 ResBlocks (residual blocks), Conv (convolution)-Relu (activation function)-Conv (convolution)-Relu (activation function)-AvgPool (adaptive average pooling) to obtain the representation, and the low-resolution image LR is passed through 9 ResBlocks (residual blocks), Conv (convolution)-Relu (activation function)-Conv (convolution)-Relu (activation function)-AvgPool (adaptive average pooling) to obtain the representation, and then processed through the Liner (linear layer)-Relu (activation function)-Liner (linear layer)-Relu (activation function) layer to obtain the GT degradation prediction feature. The LR degradation prediction network is as follows Figure 2 As shown in the figure, the low-resolution image LR passes through 9 ResBlocks (residual blocks), Conv (convolution)-Relu (activation function)-Conv (convolution)-Relu (activation function)-AvgPool (adaptive average pooling) in sequence, and the representation obtained is processed by Liner (linear layer)-Relu (activation function)-Liner (linear layer)-Relu (activation function) layer to obtain the LR degradation prediction feature.

[0059] Step 3: Send the obtained degradation feature information and the low-resolution image to the reconstruction network for enhancement to generate the final high-resolution image.

[0060] Rebuilding the network Figure 4As shown in the figure, the input of the network is the low-resolution image LR and the degradation prediction features. LR first passes through a convolutional layer to extract the low-resolution features, and then passes through N degradation-aware reconstruction modules. The input of the first reconstruction block is the degradation prediction features and the low-resolution features, and the input of the subsequent reconstruction blocks is the degradation prediction features and the output features of the previous reconstruction block. Then, it passes through a convolutional layer and a PixelShuffle (up-sampling block) for up-sampling to obtain the final output, i.e., the reconstructed image SR. The reconstruction block can be expressed by the following formula:

[0061]

[0062] Where D represents the degradation prediction feature, DA represents DABlock (degradation perception module), F is the output of the reconstruction module, X is the input of the reconstruction module, and the structure of DABlock (degradation perception module) is as follows: Figure 5 The specific operation of DABlock (degradation perception module) is as follows: the degradation prediction feature passes through the Liner layer and then multiplies the low-resolution feature or the output feature of the previous reconstruction block through the Norm layer to obtain a normalized vector , and then processed by a multi-head self-attention network. Specifically, for the normalized vector , by applying 1×1 convolution to aggregate pixel-level cross-channel context, and then applying 3×3 depth-wise convolution to encode channel-level spatial context, generating three matrices: query vector matrix, key vector matrix, and value vector matrix:

[0063]

[0064]

[0065]

[0066] in, is a 1×1 point-wise convolution, It is a 3×3 depth-wise (depth-separable) convolution. The three matrices are then fed into the multi-head attention mechanism operation to calculate the attention between each other:

[0067]

[0068] in is the obtained attention value, Q, K, V are the query vector matrix, key vector matrix and value vector matrix respectively. The output of input Y after passing through the multi-head self-attention network is as follows:

[0069]

[0070] Finally, the output of the multi-head self-attention network is processed by the Norm layer and then multiplied pixel-wise with the features of the degradation prediction features after the Liner layer. Then, it passes through a feedforward network to obtain the output of the DABlock (degradation perception module), which is expressed as follows:

[0071]

[0072] in represents a feed-forward network, is the feature of the input FFN, and D is the degradation prediction feature.

[0073] Step 4: Train a real image super-resolution variability network based on degradation prediction based on the preprocessed dataset.

[0074] The preprocessed training set images are input into the real image super-resolution variability network based on degradation prediction. The training is carried out in two stages. In the first stage, GT and LR are input into the GT degradation prediction network, and the obtained GT degradation feature information is sent to the reconstruction network for enhancement. The L1 loss function is used to calculate the loss of GT and reconstructed image SR, and back propagation is performed; the first stage training is stopped after 200 rounds.

[0075]

[0076] In the second stage, based on the weights trained in the first stage, the GT degradation prediction network is frozen and the LR is input into the LR degradation prediction network to obtain the degradation features. , input GT and LR into the GT degradation prediction network to obtain , and The loss is calculated by the L1 loss function, and It is sent to the reconstruction network for enhancement, and the L1 loss function is used to calculate the loss of GT and reconstructed image SR, and back propagation is performed.

[0077]

[0078]

[0079] The loss function at this stage is ,in, is the pixel loss, is the degradation loss. The network is optimized by the optimizer Adam, and the exponential decay rates β1 and β2 are set to 0.9 and 0.999 respectively, and the initial learning rate is 0.0001. After multiple rounds of training, the final network model is obtained.

[0080] Step 5: Test.

[0081] The evaluation index is PSNR, which is a commonly used index to measure the quality of images or videos, especially in the quality evaluation of compressed and restored images. Its calculation formula is as follows:

[0082]

[0083] Where MAX is the maximum pixel value in the image. MSE is the mean square error, which represents the average value of the difference between the original image and the processed image. The calculation formula is:

[0084]

[0085] in, and Respectively represent the original image and the compressed image at position The pixel value of .

[0086] Network training settings:

[0087] This example proposes to use PyTorch on a PC equipped with an Intel(R) Core(TM) i9-12900K CPU and RTX3090 GPU. During the training phase, each image is resized to 48×48 and a variety of data enhancement strategies are used, including random flipping and rotation. This example uses Adam to optimize our network, where the batch size and initial learning rate are set to 4 and .

[0088] In order to fully verify the model of the present invention, we tested it on a public real image super-resolution dataset RealSR. The experimental data results are as follows:

[0089] method BicuBic EDSR CDC TNN SASR DASR The present invention PSNR 31.67 33.88 33.96 34.00 34.04 33.90 34.16

[0090] The above contents are further detailed descriptions of the present invention in combination with specific / preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, they can also make several substitutions or modifications to these described embodiments without departing from the concept of the present invention, and these substitutions or modifications should be regarded as belonging to the protection scope of the present invention.

[0091] Parts of the present invention that are not described in detail belong to the well-known technologies of those skilled in the art.

Claims

1. A real image super-resolution rate method based on degradation prediction, characterized in that: The steps include: Step 1: Dataset acquisition and preprocessing; Step 2: Extract degradation feature information from the low-resolution image through a degradation prediction network; Step 3: Send the obtained degradation feature information and the low-resolution image to the reconstruction network for enhancement to generate the final high-resolution image; Step 4: training a real image super-resolution variability network based on degradation prediction based on the preprocessed data set; The real image super-resolution variability network is composed of two parts: a degradation prediction network and a reconstruction network; Step 5: Test: Input the test set images into the trained real image super-resolution rate network to obtain the reconstructed high-resolution image SR, and compare it with the true value GT to calculate various evaluation indicators.

2. The real image super-resolution rate method based on degradation prediction according to claim 1, characterized in that: Step 1 is as follows: The real super-resolution image dataset RealSR is used for training and testing. 400 pairs of low-resolution and high-resolution images with a scale of 2 times the dataset are used for training, and 100 pairs of images are used for testing. The preprocessing of the training set includes: rotating the images 30°, 45°, and 60° clockwise and counterclockwise, flipping them horizontally and vertically, and cropping the training low-resolution images to 48×48.

3. The real image super-resolution rate method based on degradation prediction according to claim 1 or 2, characterized in that: The degradation prediction network described in step 2 is composed of a GT degradation prediction network and an LR degradation prediction network; in the GT degradation prediction network, the true value GT is downsampled and then channel-joined with the low-resolution image LR, and the output is sequentially passed through 9 residual blocks, convolution-activation function-convolution-activation function-adaptive average pooling to obtain a representation, and the low-resolution image LR is passed through 9 residual blocks, convolution-activation function-convolution-activation function-adaptive average pooling to obtain a representation, and then processed through a linear layer-activation function-linear layer-activation function layer to obtain a GT degradation prediction feature; in the LR degradation prediction network, the low-resolution image LR is sequentially passed through 9 residual blocks, and the representation obtained after convolution-activation function-convolution-activation function-adaptive average pooling is processed through a linear layer-activation function-linear layer-activation function layer to obtain an LR degradation prediction feature.

4. The real image super-resolution rate method based on degradation prediction according to claim 3, characterized in that: Step 3: The input of the reconstruction network is the low-resolution image LR and the degradation prediction features. LR first passes through a convolutional layer to extract the low-resolution features, and then passes through N degradation-aware reconstruction modules. The input of the first reconstruction block is the degradation prediction features and the low-resolution features, and the input of the subsequent reconstruction blocks is the degradation prediction features and the output features of the previous reconstruction block. Then, it passes through a convolutional layer and an upsampling block for upsampling to obtain the final output, i.e., the reconstructed image SR. The reconstruction module can be expressed by the following formula: ； Where D represents the degradation prediction feature, DA represents the degradation perception module, F is the output of the reconstruction module, and X is the input of the reconstruction module. The specific operation of the degradation perception module is as follows: the degradation prediction feature passes through the linear layer and then multiplies the low-resolution feature or the output feature of the previous reconstruction block through the normalization layer to obtain a normalized vector , and then processed by a multi-head self-attention network; for the normalized vector , by applying 1×1 convolution to aggregate pixel-level cross-channel context, and then applying 3×3 depth-wise convolution to encode channel-level spatial context, generating three matrices: query vector matrix, key vector matrix, and value vector matrix: ； ； ； in, is a 1×1 point-wise convolution, It is a 3×3 depth-wise separable convolution; the three matrices are then fed into a multi-head attention mechanism to calculate the attention between each other: ； in is the obtained attention value, Q, K, V are the query vector matrix, key vector matrix and value vector matrix respectively; the output of input Y after the multi-head self-attention network is as follows: ； Finally, the output of the multi-head self-attention network is processed by the normalization layer and then multiplied pixel-wise with the features of the degradation prediction features after the linear layer. Then, it passes through a feedforward network to obtain the output of the degradation perception module. The formula is as follows: ； ； in represents a feed-forward network, is the feature of the input feedforward network, and D is the degradation prediction feature.

5. The real image super-resolution rate method based on degradation prediction according to claim 4, characterized in that: Step 4 is as follows: The preprocessed training set images are input into the real image super-resolution variability network based on degradation prediction. The training is carried out in two stages. In the first stage, GT and LR are input into the GT degradation prediction network, and the obtained GT degradation feature information is sent to the reconstruction network for enhancement. The loss of GT and reconstructed image SR is calculated using the L1 loss function, and back propagation is performed. The first stage training stops when convergence. ； In the second stage, based on the weights trained in the first stage, the GT degradation prediction network is frozen and the LR is input into the LR degradation prediction network to obtain the degradation features. , input GT and LR into the GT degradation prediction network to obtain , and The loss is calculated by the L1 loss function, and Send it to the reconstruction network for enhancement, use the L1 loss function to calculate the loss of GT and reconstructed image SR, and perform back propagation; ； ； The loss function at this stage is ,in, is the pixel loss, For degradation loss.

6. The real image super-resolution rate method based on degradation prediction according to claim 5, characterized in that: The real image super-resolution variability network based on degradation prediction is optimized by the optimizer Adam, and the exponential decay rates β1 and β2 are set to 0.9 and 0.999 respectively, and the initial learning rate is 0.0001; the final network model is obtained after multiple rounds of training.

7. The real image super-resolution rate method based on degradation prediction according to claim 1, characterized in that: The evaluation indicators in step 5 are as follows: The evaluation index is PSNR, and its calculation formula is as follows: ； Among them, MAX is the maximum pixel value in the image; MSE is the mean square error, which represents the average value of the difference between the original image and the processed image. Its calculation formula is: ； in, and Respectively represent the original image and the compressed image at position The pixel value of .

Images