An image blind super-resolution reconstruction method based on weakly supervised contrast learning

Abstract

The application provides an image blind super-resolution reconstruction method based on weakly supervised contrast learning. In the training stage, a degradation representation vector is obtained through weakly supervised contrast learning from an input low-resolution image (LR), so as to adapt to different input low-resolution images; meanwhile, the input low-resolution image and the corresponding degradation representation vector are jointly used as the input of a super-resolution network, so that the degradation information is fully utilized to guide the reconstruction of the super-resolution network; finally, in the test stage, image reconstruction is performed through image patching operation. The application is mainly applied to image processing software and has broad application prospects in the fields of video monitoring, medicine, target detection and the like.

Description

Technical Field

[0001] This invention relates to a blind super-resolution reconstruction method for images based on weakly supervised contrastive learning, and to a novel super-resolution reconstruction network method using image degradation as auxiliary information. It relates to the problem of single-image super-resolution reconstruction in the field of image restoration and belongs to the field of computer vision and intelligent information processing. Background Technology

[0002] Single Image Super Resolution (SISR) is a subtask of image restoration, aiming to recover a high-resolution (HR) image from a low-resolution (LR) image. As a fundamental problem in computer vision, its enormous application potential in fields such as remote sensing, medical imaging, and security monitoring is attracting increasing attention.

[0003] Over the past decade, SISR has become an active topic in the field of low-level computer vision, leading to numerous research proposals. These methods can be broadly categorized into three types: interpolation-based methods, reconstruction-based methods, and learning-based methods. Among learning-based methods, deep learning-based SISR methods, in particular, have elevated super-resolution performance to a new level. However, most existing methods assume that the degradation of LR images is known, such as bicubic downsampling, which is not applicable in real-world scenarios. Worse still, the mismatch between the inherent degradation of LR images and the assumed degradation often results in artifacts in the reconstructed images. For cases where image degradation is unknown (i.e., blind super-resolution), this invention proposes a blind super-resolution reconstruction method based on weakly supervised contrastive learning, aiming to solve the problem of image reconstruction with unknown degradation types. Summary of the Invention

[0004] This invention proposes a weakly-supervised contrastive learning-based blind image super-resolution (WCBSR) method, which consists of two networks: an implicit degradation modeling network (IDMN) and a degradation-guided super-resolution network (DGSRN). Specifically, IDMN is responsible for degradation encoding the LR image y to generate a degradation representation vector, while DGSRN uses the LR image and its corresponding degradation representation vector as input to reconstruct the high-resolution image x. The coordination between IDMN and DGSRN enables this invention to exhibit excellent performance in processing images with unknown degradation types.

[0005] A blind image super-resolution reconstruction method based on weakly supervised contrastive learning includes the following steps:

[0006] (1) Training phase: N train A low-resolution image is fed into the IDMN network, encoded into a degenerate representation vector d, and subjected to weakly supervised contrastive learning.

[0007] (2) Training phase: Then the degenerate representation vector d is reduced in dimension to d through a fully connected layer. r Each LR image and its corresponding dimensionality-reduced degenerate representation vector d r The images are fed into the DGSRN network for joint optimization, and finally the reconstructed HR image is obtained; (steps (1) and (2) are both training processes)

[0008] (3) Testing phase: Two testing methods are used to obtain the degenerate representation vector and combine it with the image for reconstruction. Attached Figure Description

[0009] Figure 1 This is an overall block diagram of an image blind super-resolution reconstruction method based on weakly supervised contrastive learning according to the present invention;

[0010] Figure 2 A schematic diagram illustrating the addition of positive samples to the weakly supervised contrastive learning method of this invention;

[0011] Figure 3 This is a schematic diagram of the distance-weighted weak-supervised contrastive learning method of the present invention;

[0012] Figure 4 This is a schematic diagram of test method 2 of the present invention;

[0013] Figure 5 The image shows the test results of this invention compared with other methods. The test method is Method 1 in the test phase. The selected image is img_044 from the Urban100 dataset. The input image degradation information is a blur kernel width of 0.6 and a noise level of 5.

[0014] Figure 6 The image shows the test results of this invention compared with other methods. The test method is Method 2 in the test phase. The selected image is 3096 from the B100 dataset. The input image degradation information is a blur kernel width of 1.8 and a noise level of 5. Detailed Implementation

[0015] The following is in conjunction with the appendix Figure 1 Appendix Figure 2 Appendix Figure 3 Appendix Figure 4 Further explanation of the present invention:

[0016] Figure 1 This is the overall block diagram of the present invention, which mainly consists of two parts: an implicit degradation modeling network (IDMN) and a degradation-guided super-resolution network (DGSRN). The DGSRN mainly consists of three parts: LR image feature extraction, degradation-guided feature transformation, and HR image reconstruction.

[0017] Overall network architecture:

[0018] First, the input LR image y is contrastively learned through an IDMN network to obtain the degenerate representation vector d; then, y is processed by LR feature extraction (3×3 convolution, 64 channels) to obtain F. IF After that, F IF With the degenerate representation vector d after dimensionality reduction r Together, they are fed into a feature transformation network guided by degradation information to obtain the output feature F. RF As shown in formula (1):

[0019] F RF =T FTN (F IF ,d r (1)

[0020] Among them, T FTN (·) represents the feature transformation network, which contains K g A degenerate information-guided feature transformation group (DGFTG), d rThe dimensionality reduction is achieved through a fully connected layer, as shown in Equation (2):

[0021] d r =T DRN (d) (2)

[0022] Among them, T DRN (·) indicates a dimensionality reduction operation performed through a fully connected layer with 256 input and 64 output channels. Finally, F RF High-resolution output images are obtained through the HR image reconstruction module. As shown in formula (3):

[0023]

[0024] in, It is the output high-resolution image, T IRN (·) represents the image reconstruction process, in which the L1 loss function is used to optimize the parameters throughout the reconstruction process.

[0025] Degradation Information Guided Super-Resolution Network (DGSRN):

[0026] DGSRN contains k g Each Degeneracy Information Guided Feature Transform Group (DGFTG) and skip connections. Each DGFTG contains k... m The DGFTM consists of a Degradation Information Guided Feature Transform Module (DGFTM) and jump connections. Internally, the DGFTM contains two Degradation Information Guided Feature Modulation Blocks (DGFMBs), which are the core of the DGFTM.

[0027] More specifically, the final output feature F of DGSRN RF This can be represented as shown in formula (4):

[0028]

[0029] in, Indicates the Kth g One DGFTG module, T NC This represents the last 3×3 convolution of DGFTG. The input's... The kth DGFTG is the (kth)th g -1) th DGFTG and d r The output is shown in formula (5):

[0030]

[0031] in, and They are respectively the (k)th g -1)th and the The output of DGFTG.

[0032] Degradation Information Guided Feature Transformation Group (DGFTG):

[0033] DGFTG is similar to DGSRN, and includes K m A Degradation Information Guided Feature Transformation Module (DGTFM) and a jump connection. One DGFTG can be represented as As shown in formula (6):

[0034]

[0035] in, Indicates the kth m One DGFTM, T GC This represents the last 3×3 convolution of DGFTG.

[0036] Degradation Information Guided Feature Transformation Module (DGFTM):

[0037] DGFTM consists of two Degenerate Guided Feature Modulation Blocks (DGFMB) and two 3×3 convolutions. Each DGFTM can be represented as shown in formula (7):

[0038]

[0039] in, They represent the first and the The output of each DGFTM, and Let i represent the i-th DGFMB and the 3×3 convolution, respectively.

[0040] Degradation Information Guided Feature Modulation Block (DGFMB):

[0041] Given the features F of the input IN and the degenerate representation vector d of the dimension reduction r First, put F IN Perform global average pooling to obtain F SAP As shown in formula (8):

[0042] F SAP =T SAP (F IN (8)

[0043] Where T SAP This indicates a global average pooling operation. Then, F... SAP With d rPerform a concatenation operation to obtain the final modulation coefficient F. ATM As shown in formula (9),

[0044]

[0045] Where [·] represents the concatenation operation. Indicates two fully connected layers, T SMD (·) represents the sigmoid activation function. Finally, the output of DGFMB is F. OUT As shown in formula (10),

[0046]

[0047] in, This indicates a multiplication operation.

[0048] Weakly supervised comparative learning:

[0049] In the comparative learning concept of this invention, given N train N HR images, through random image degradation operation, to N train Zhang HR image degraded to N train We extract LR images with different degradation types and use the blur kernel width δ and additive Gaussian noise σ to distinguish whether these degradation types are the same. Then, we obtain N... train Two blocks are randomly cropped from the LR image to obtain 2N. train Zhang LR image patches, 2N train Zhang LR image patches are used to generate degradation vectors through an IDMN network. 2N train Degenerate vectors Degenerate representation is obtained through two fully connected layers. use We will conduct comparative learning to distinguish different types of image degradation. We assume that the degradation process of two blocks in the same LR image is the same. Taking a certain LR image m as an example... i For example, m i One of the image patches For reference, there is another image patch corresponding to it. yes A positive sample, and another image m with a different degradation type than LR. j Two image patches and yes The negative samples. The purpose of contrastive learning is to reduce the distance between positive samples and negative samples. Increase the distance of negative samples ( and ).

[0050] However, in the contrastive learning process, in order to fully utilize contrastive learning to distinguish differences in image degradation types rather than differences in image content, multiple images with different content are subjected to the same degradation, thereby obtaining multiple positive samples and improving the efficiency of contrastive learning in learning the image degradation process. For example... Figure 2 As shown: In the three low-resolution images on the left, the first and second images (counting from top to bottom) are two different HR images obtained through the same degradation process (blur kernel width 1.8, additive Gaussian noise intensity 10). The degradation process for the third image is different from the first and second images; that is, the degradation parameters are a blur kernel width of 0.6 and an additive Gaussian noise intensity of 5. After cropping the three LR images on the left, the encoder produces the degraded representation vector d. q_p d q_n d q Then, a degenerate representation vector r is generated through two fully connected layers. - r + Taking the middle degenerate representation vector r as a reference, since the degradation process is the same in the first and second images on the left, the two degenerate representation vectors r obtained from the first image on the left are... + All are considered positive samples of the reference sample r, while the degradation process in the third figure on the left is different from the previous two figures, so it is considered a negative sample r. - Therefore, the contrastive learning loss function designed to increase positive samples is shown in Equation (11):

[0051]

[0052] Where, N neg N pos These represent the number of negative samples and the number of positive samples, respectively. It is the i-th positive sample, and r is The corresponding reference sample, It is the j-th negative sample, and τ is set to a constant of 0.07.

[0053] Secondly, because image degradation is random, some Lightroom images undergo different degradation processes, while others undergo similar degradation processes. For example... Figure 3 As shown in the three LR images on the left, observation reveals that the first image (counting from top to bottom) shows a significant difference in degradation compared to the second image (i.e., a large difference in the blur kernel width δ and additive Gaussian noise σ), while the third image exhibits a similar degradation process to the second image (i.e., the blur kernel width δ and additive Gaussian noise σ are similar). To better distinguish the differences in degradation types, a weighted contrastive learning loss function was designed. Figure 3 In the middle, the three LR images on the left are cropped and then encoded to obtain the degenerate representation vector d. q_nd q_p d q Then, a degenerate representation vector r is generated through two fully connected layers. - r + r. - r + Let r represent the negative sample, positive sample, and reference sample, respectively. Here, the reference sample r corresponds to the degraded representation vector generated from the second LR image. Since the degradation difference between the second and third images among the three LR images on the left is small, while the degradation difference between the first and second images is large, r is reduced during the contrastive learning process. + The distance between r and the negative sample r of the third LR image from the left is also considered. - The distance is slightly increased, r and the negative sample r of the first LR image on the left. - The distance increases significantly. This plays a certain role in distinguishing the differences between different negative samples. The distance weighting method implemented here is Euclidean distance (other distance calculation methods can also be selected, it is not unique), as shown in formula (12):

[0054]

[0055] Among them, δ, The fuzzy kernel widths for the reference and negative samples are σ and σ. Let be the additive Gaussian noise intensity of the reference sample and the negative sample. Equation (13) is the contrastive learning loss function using distance weighting:

[0056]

[0057] in, Let r and r' be the weight coefficients corresponding to different negative samples. + , Let N represent the reference sample, the positive sample, and the j-th negative sample, respectively. neg This represents the number of negative samples, and τ is set to a constant of 0.07.

[0058] Therefore, by combining the increase of positive samples and the increase of weights in the loss function, the final weakly supervised contrastive learning loss function is obtained as shown in formula (14):

[0059]

[0060] in, r represents the i-th positive sample, the j-th negative sample, and the reference sample, respectively, and w j It is the distance weighting coefficient, N pos N neg These represent the number of positive and negative samples, respectively, and τ is set to a constant of 0.07.

[0061] Furthermore, to better encode image patches and generate degradation representation vectors, this invention introduces a Channel Spatial Attention Module (CBAM) on top of the traditional encoder, in order to better distinguish different types of degradation. The structure is as follows: Figure 1 The left-hand IDMN module, shown in the figure, consists of 6 convolutional modules and 1 CBAM module. The specific structural parameters of the designed IDMN are shown in Table 1.

[0062] Table 1. Specific Structure of IDMN

[0063]

[0064] The input image patch has a size of H×W and 3 channels. After passing through the first convolutional layer, it yields 64-channel image features. These features are then adaptively modulated by the CBAM module, followed by 5 convolutional layers. Finally, global average pooling is used to compress the image size, resulting in a 256-dimensional vector d. This vector is then used in the DGSRN stage.

[0065] Finally, the loss is learned by contrasting the L1 loss function of the super-resolution network with that of weak supervision. Perform joint optimization to obtain the overall loss function.

[0066]

[0067] in, This is the L1 loss, as shown in formula (16):

[0068]

[0069] Among them, y i Let d represent the i-th LR image. i It is its corresponding degenerate representation vector, x i It is y i The reconstruction results, N train It is the number of training samples, T DGSRN (·) is the DGSRN network.

[0070] Since image degradation information is crucial for the reconstruction process, this invention designs two testing methods to obtain the degradation representation vector d more accurately and thus assist the reconstruction process:

[0071] Test Method 1: During testing, the input LR image is segmented into blocks, reconstructed block by block, and finally stitched together to obtain the complete image. This method improves reconstruction performance by using image segmentation to obtain multiple degradation representation vectors on the same image and then using these vectors to reconstruct multiple image blocks.

[0072] Test method 2: such as Figure 4 As shown, during testing, the input LR image is segmented into 4 regions for degradation encoding, resulting in multiple degradation representation vectors. The average of multiple vectors is taken to obtain the average degradation representation vector d, thereby increasing the accuracy of the evaluated degradation representation vector. After obtaining d, the LR image itself and d are fed into the DGSRN network for reconstruction to obtain the reconstructed image.

[0073] To verify the effectiveness of the image blind super-resolution reconstruction method based on weakly supervised contrastive learning described in this invention, a detailed comparison is conducted through experiments below.

[0074] Experimental environment: Ubuntu 20.04 operating system, one RTX 2080 Ti graphics card, code written in Python, and PyTorch 1.7 deep learning framework. This invention fairly compares the PSNR and SSIM results (higher PSNR and SSIM indicate better performance) at a reconstruction scale of 2x with blur kernel widths of 0.6, 1.2, and 1.8 and a noise level of 5. Commonly used bicubic interpolation and three deep learning-based blind image super-resolution methods were selected, specifically:

[0075] BSRGAN: A method proposed by Zhang et al., referenced in "K. Zhang, J. Liang, L. Van Gool, R. Timofte, Designing a practical degradation model for deep blind image super-resolution[C] / / Proceedings of the IEEE International Conference on ComputerVision(ICCV),2021:4791–4800."

[0076] IKC: A method proposed by Gu et al., see reference "Gu J, Lu H, Zuo W, et al. Blind super-resolution with iterative kernel correction[C] / / Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition.2019:1604-1613.". Since IKC cannot handle noisy images, the DnCNN method is used for image denoising. DnCNN is a method proposed by Zhang et al., see reference "Zhang K, Zuo W, Chen Y, et al. Beyond a gaussian denoiser: Residual learning of deep CNN for image denoising[J].IEEE transactions on image processing,2017,26(7):3142-3155."

[0077] DASR: The method proposed by Wang et al., reference "Wang L, Wang Y, Dong X, et al. Unsupervised Degradation Representation Learning for Blind Super-Resolution[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision andPattern Recognition.2021:10581-10590."

[0078] The test results are shown in Tables 2 and 3, where Table 2 shows the results for test method 1, and Table 3 shows the results for test method 2. By comparing with other methods, it can be found that the reconstruction performance of the present invention is better. Meanwhile, in... Figure 5 , Figure 6 The image also shows an objective visual effect of the invention. Figure 5 For the results of test method 1, Figure 6 The results are for test method 2. Observation shows that the present invention can recover more detailed information from the image.

[0079] Table 2 Test Method 1

[0080]

[0081] Table 3 Test Method 2

[0082]

Claims

1. A blind super-resolution image reconstruction method based on weakly supervised contrastive learning, characterized by the following steps: (1) Training phase: A low-resolution image is fed into the Implicit Degradation Modeling Network (IDMN) to generate a degradation representation vector. And perform weakly supervised comparative learning; (2) Training stage: Then the degraded representation vector is reduced to , each LR image and the corresponding degraded representation vector after dimensionality reduction are sent to the degradation information guided super-resolution network DGSRN for joint optimization, and finally the reconstructed HR image is obtained; steps (1) and (2) are training links; (3) Testing phase: Two testing methods are used to obtain the degenerate representation vector, and the image itself is combined for reconstruction; According to the method, it is characterized in that In step (1), an implicit degradation modeling network (IDMN) structure is designed. This structure introduces a channel spatial attention mechanism (CBAM) after the first convolutional layer, which can better distinguish degradation between images and thus better guide the image reconstruction process. The key feature is that step (2) involves designing a degradation information-guided feature modulation module (DGFMB); in this module, given the input features... Degenerate representation vectors with reduced dimensionality First, the input features Global average pooling is performed to obtain As shown in formula (1): wherein represents a global average pooling operation; thereafter, is merged with concatenation operation to obtain the final modulation coefficient As shown in equation (2): wherein denotes a concatenation operation, denotes 2 fully connected layers, denotes a sigmoid activation function; finally, the output of DGFMB is as shown in equation (3): wherein represents a multiplication operation.

2. The method of claim 1, wherein Step (1) employs a contrastive learning loss that incorporates positive samples: wherein is the positive sample, is the corresponding reference sample, is the negative sample, is set to the constant 0.07, , denotes the number of positive and negative samples.

3. The method of claim 1, wherein In step (1), a distance-weighted contrastive learning loss was used: in, These are the weighting coefficients corresponding to different negative samples. , , These represent the reference sample, the positive sample, and the first sample, respectively. One negative sample, Indicates the number of negative samples. Set to a constant of 0.07; It can be obtained from formula (6): in, , For reference sample and the first The width of the blur kernel for each negative sample. , For reference sample and the first The intensity of additive Gaussian noise for each negative sample.

4. The method of claim 1, wherein In step (1), a weakly supervised contrastive learning loss was designed: in, , , They represent the first The positive sample, the first One negative sample and one reference sample It is the distance weighting coefficient. , These represent the number of positive and negative samples, respectively. Set to a constant of 0.

07. It represents the number of training samples.

5. The method according to claim 1, characterized in that... In step (3), the image is reconstructed by image slicing during the testing phase. During the test, the input LR image is sliced ​​into blocks, reconstructed block by block, and finally the complete image is obtained by stitching. This method obtains multiple degradation representation vectors on the same image through image slicing, and uses multiple degradation representation vectors to reconstruct multiple image blocks, thereby improving the reconstruction performance.

6. The method of claim 1, wherein In step (3), during the testing phase, the degradation representation vector is obtained by averaging multiple degradation representation vectors to guide image reconstruction. During testing, the input LR image is segmented into 4 regions for degradation encoding to obtain multiple degradation representation vectors. And by averaging multiple vectors, the average degenerate representation vector is obtained. This is to increase the accuracy of the evaluated degenerate representation vector; thus obtaining Then, the LR image itself and The images are fed into the DGSRN network for reconstruction, resulting in a reconstructed image.

Images