A blind image super-resolution reconstruction method based on diffusion model and attention mechanism

Abstract

The present invention discloses a blind image super-resolution reconstruction method based on a diffusion model and an attention mechanism. In the first stage of training, a low-resolution image and a high-resolution image pair are used as input to train a content-aware degradation encoder and a super-resolution reconstruction network. In the second stage of training, a low-resolution image is used as input to train a LR encoder and a denoising network of a diffusion model, and a content-aware degradation prior generated by the diffusion model replaces the content-aware degradation prior in the first stage. Then, the super-resolution reconstruction network is fine-tuned, and finally the low-resolution image is reconstructed. The present invention can use the diffusion model to generate a content-aware degradation prior to guide the reconstruction of a super-resolution reconstruction network based on an attention mechanism, and a two-way aggregation method is used to perform more robust feature extraction on the super-resolution reconstruction network, thereby better utilizing the content-aware degradation prior to produce better reconstruction results.

Description

Technical Field

[0001] The present invention belongs to the field of image super-resolution reconstruction, and in particular relates to a blind image super-resolution reconstruction method based on a diffusion model and an attention mechanism. Background Art

[0002] In the classic image super-resolution framework, the low-resolution images used for training are synthesized from high-resolution images through a specific degradation method (bicubic downsampling and a specific blur kernel). In order to cope with the realistic challenges of multiple degradation effects, the low-resolution images of the blind image super-resolution reconstruction method are synthesized from high-resolution images through an unknown degradation method (bicubic downsampling and an unknown blur kernel).

[0003] Existing blind image super-resolution reconstruction methods can be divided into two categories: kernel prediction and degradation prediction. Kernel prediction methods use explicit or implicit techniques to predict blur kernels, and then guide the non-blind super-resolution reconstruction network through a kernel stretching strategy. This method produces ideal super-resolution reconstruction results in most cases, but still faces severe performance degradation when it has to deal with complex degradations. This unavoidable limitation is subject to the unavailability of real-world kernels and the mismatch of training distribution. In addition, kernel prediction methods are limited to blur kernel-based degradations and cannot handle other types of degradation (e.g., noise).

[0004] In contrast, degradation prediction methods further encourage learning degradation representations. Unsupervised Degradation Representation Learning for Blind Super-Resolution (DASR) extracts degradation representations in an unsupervised manner and can outperform some kernel prediction methods on simple degradations, but there is still a gap between degradation prediction and state-of-the-art kernel prediction methods in more complex cases.

[0005] Although the above degradation prediction methods can handle real-world degradation to some extent, only the degradation representation is used in previous works, which makes the network agnostic to texture information. Therefore, a content-aware degradation is needed that explicitly considers both the degradation representation and the content representation, and can achieve significant improvements. Summary of the invention

[0006] Purpose of the invention: The present invention provides a blind image super-resolution reconstruction method based on a diffusion model and an attention mechanism, which can utilize the diffusion model to generate content-aware degradation priors that are helpful for low-resolution image reconstruction, thereby obtaining better reconstruction results.

[0007] Technical solution: The blind image super-resolution reconstruction method based on a diffusion model and an attention mechanism described in the present invention comprises the following steps:

[0008] (1) constructing a blind image super-resolution reconstruction network, including a content-aware degradation generation module and a super-resolution reconstruction module; the content-aware degradation generation module includes a content-aware encoder, an LR encoder and a denoising network;

[0009] (2) In the first stage of training, the low-resolution image and the high-resolution image pair are used as input to train the content-aware encoder and the super-resolution reconstruction network, generating a true content-aware degradation prior and a super-resolution reconstruction network based on the content-aware degradation prior;

[0010] (3) In the second stage of training, the low-resolution image is used as input to train the LR encoder and the denoising network of the diffusion model;

[0011] (4) A content-aware degradation prior is generated using the trained LR encoder and diffusion model to replace the true content-aware degradation prior in step (1), and then the super-resolution reconstruction network in step (1) is fine-tuned to finally reconstruct the low-resolution image.

[0012] Furthermore, the content-aware encoder in step (1) is implemented as follows:

[0013] The low-resolution and high-resolution image pairs are sent to several degradation blocks consisting of convolutional layers, linear rectification functions and convolutional layers, and then passed through a pooling layer to obtain the degradation prior; the high-resolution image is sent to several content branches consisting of convolutional layers, linear rectification functions and convolutional layers, and then passed through a pooling layer to obtain the content prior; finally, the degradation prior and content prior are concatenated and sent to a multi-layer perceptron to obtain the true content-aware degradation prior.

[0014] Furthermore, in the first stage of training in step (2), the content-aware encoder and the super-resolution reconstruction module are combined to optimize the entire blind image super-resolution reconstruction network by minimizing the L1 norm loss of the reconstructed image and the high-resolution image, thereby training a content-aware degradation encoder that can extract the true content-aware degradation prior. The entire network uses the L1 norm loss as the loss function:

[0015]

[0016] Among them, I HR For a real high-resolution image, I SR Represents the reconstructed image.

[0017] Furthermore, the super-resolution reconstruction module includes several content degradation residual modules CDRB, and for each CDRB, a content-aware degradation prior is used to perform an affine transformation on the input features; then a serial structure of a shifted window self-attention mechanism and a channel self-attention mechanism is adopted to improve the attention mechanism effect of the model; channel information is supplemented for the shifted window self-attention mechanism through deep convolution, and spatial information is supplemented for the channel self-attention mechanism through deep convolution to output more robust features.

[0018] Furthermore, in the second stage of training described in step (3), the content-aware encoder of the first stage is frozen, and the diffusion model and the super-resolution reconstruction module are jointly trained to optimize the diffusion model by minimizing the L1 norm loss of the content-aware degradation prior generated by the diffusion model and the true content-aware degradation prior, and at the same time, the L1 norm loss of the reconstructed image and the high-resolution image is minimized to fine-tune the super-resolution reconstruction network:

[0019]

[0020]

[0021]

[0022] Among them, I HR For a real high-resolution image, I SR represents the reconstructed image, Z0 is the true content-aware degradation prior, represents the generated content-aware degradation prior, α diff is the weight coefficient of the diffusion model loss.

[0023] Furthermore, the denoising network in step (1) consists of five fully connected layers.

[0024] Furthermore, the degradation block is composed of a standard 3×3 convolution, a linear rectification function and a standard 3×3 convolution, and a padding of step 1 is used for the standard 3×3 convolution in each degradation block, and then an adaptive average pooling layer is used to pool the features into a 1×1 vector.

[0025] Furthermore, the first content block in the content branch is composed of a 12×12 convolution, a linear rectification function and a standard 3×3 convolution; the stride of the 12×12 convolution is set to 12;

[0026] Furthermore, the subsequent content blocks in the content branch are composed of a standard 3×3 convolution and a linear rectification function, and finally an adaptive average pooling layer is used to pool the features into a 1×1 vector.

[0027] Beneficial effects: Compared with the prior art, the advantages and beneficial effects of the present invention are as follows: Starting from generating a more reasonable content-aware degradation prior, the present invention proposes a blind image super-resolution reconstruction method based on a diffusion model and an attention mechanism. In order to improve the efficiency of the diffusion model, it is proposed to use a fully connected layer structure to construct a denoising network; in order to make the generated content-aware degradation prior more robust, the present invention directly performs L1 norm loss on the generated content-aware degradation prior and the real content-aware degradation prior, and fine-tunes the training in conjunction with the first-stage super-resolution reconstruction network to ensure that the generated content-aware degradation prior can be recognized by the super-resolution reconstruction network; the present invention also proposes to use content-aware degradation prior to perform affine transformation on input features, use deep convolution to supplement channel information for shift window self-attention, and use deep convolution to supplement spatial information for the channel self-attention mechanism, thereby making the features more robust. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a schematic diagram of the overall network framework of the present invention;

[0029] Figure 2 Schematic diagram of the content degradation residual module structure;

[0030] Figure 3 Schematic diagrams of blind super-resolution reconstruction results in different models; among them, (a) is a schematic diagram of the blind super-resolution reconstruction results of DASR; (b) is a schematic diagram of the blind super-resolution reconstruction results of DCLS; (c) is a schematic diagram of the blind super-resolution reconstruction results of StableSR; (d) is a schematic diagram of the blind super-resolution reconstruction results of the present invention. DETAILED DESCRIPTION

[0031] The present invention will be further described in detail below in conjunction with the accompanying drawings.

[0032] The present invention proposes a blind image super-resolution reconstruction method based on a diffusion model and an attention mechanism, comprising the following steps:

[0033] Since blind image super-resolution reconstruction needs to cope with the practical challenge of multiple degradation effects, unknown degradation is used to synthesize low-resolution images during network training. First, a high-resolution image is selected and degraded using a Gaussian blur kernel with random parameters. Then, the blurred image from the previous step is further degraded using bicubic downsampling. Finally, random Gaussian noise is added to the downsampled image to obtain a low-resolution image obtained by unknown degradation as training data.

[0034] like Figure 1As shown in the figure, a blind image super-resolution reconstruction network CDFormer is constructed, including a content-aware degradation generation module and a super-resolution reconstruction module. CDFormer adopts two-stage training. For the first stage, the input image first passes through the content-aware degradation encoder to obtain a true content-aware degradation prior. The content-aware degradation prior supervises the training of the super-resolution reconstruction network in the form of affine transformation; for the second stage training, the input image first passes through the LR encoder to obtain the conditional vector as the condition of the diffusion model, and the denoising network composed of fully connected layers is used for step-by-step denoising. Finally, CDFormer uses the content-aware degradation generation module to generate a content-aware degradation prior, and then the generated content-aware degradation prior is used to supervise the super-resolution network to complete the image super-resolution reconstruction.

[0035] For the content-aware degradation encoder, the low-resolution and high-resolution image pairs are sent to several degradation blocks consisting of convolutional layers, linear rectification functions and convolutional layers, and then pass through a pooling layer to obtain the degradation prior; the high-resolution image is sent to several content blocks consisting of convolutional layers, linear rectification functions and convolutional layers, and then pass through a pooling layer to obtain the content prior; finally, the degradation prior and content prior are concatenated and sent to a multi-layer perceptron to obtain the true content-aware degradation prior.

[0036] like Figure 1 As shown in the figure, for each degradation block of the degradation branch, a standard 3×3 convolution, a linear rectification function and a standard 3×3 convolution are used. At the same time, in order to ensure that the feature can keep the same size after the standard 3×3 convolution, the standard 3×3 convolution in each degradation block is padded with a step size of 1, and then an adaptive average pooling layer is used to pool the feature into a 1×1 vector; for the first content block of the content branch, a 12×12 convolution, a linear rectification function and a standard 3×3 convolution are used. The stride of the 12×12 convolution layer is set to 12 to ensure that the 12×12 convolution layer can extract local content information. The subsequent content blocks are composed of a standard 3×3 convolution and a linear rectification function. Finally, an adaptive average pooling layer is used to pool the feature into a 1×1 vector; then the vectors obtained from the degradation branch and the content branch are concatenated in the channel dimension and sent to a multi-layer perceptron to obtain the final content-aware degradation prior.

[0037] For each content degradation residual module CDRB in the super-resolution reconstruction module, such as Figure 2 As shown, we first use the content-aware degradation prior to the input feature F iAffine transformation is performed; then the serial structure of shift window self-attention and channel self-attention mechanism is adopted to improve the attention mechanism effect of the model; channel information is supplemented for shift window self-attention mechanism through deep convolution, and spatial information is supplemented for channel self-attention mechanism through deep convolution, so as to output a more robust feature F i+1 .

[0038] In the network model proposed in this invention, another very important denoising network is composed of five fully connected layers, which converts a Z sampled from a Gaussian noise of a standard normal distribution into T After 4 denoising steps, we get The generated content-aware degradation prior The loss is done with the content-aware degradation Z0 obtained by the content-aware degradation encoder.

[0039] For the first stage, the present invention combines the content-aware degradation encoder and the super-resolution reconstruction network to optimize the entire network by minimizing the L1 norm loss of the reconstructed image and the high-resolution image, so as to train a content-aware degradation encoder that can extract the true content-aware degradation prior. The entire network uses the L1 norm loss as the loss function:

[0040]

[0041] Among them, I HR For a real high-resolution image, I SR Represents the reconstructed image.

[0042] For the second stage, the present invention freezes the content-aware degradation encoder of the first stage, jointly trains the diffusion model with the super-resolution reconstruction network, minimizes the L1 norm loss of the content-aware degradation prior generated by the diffusion model and the true content-aware degradation prior to optimize the diffusion model, and minimizes the L1 norm loss of the reconstructed image and the high-resolution image to fine-tune the super-resolution reconstruction network:

[0043]

[0044]

[0045]

[0046] Among them, I HR For a real high-resolution image, I SR represents the reconstructed image, Z0 is the true content-aware degradation prior, represents the generated content-aware degradation prior, α diff is the weight coefficient of the diffusion model loss.

[0047] The method proposed in the present invention is to use a diffusion model to generate a content-aware degradation prior, which is used as auxiliary information for a super-resolution reconstruction network, and injects the content-aware degradation prior into the super-resolution reconstruction network by means of an affine transformation. Then, the present invention uses a CDRB module to aggregate spatial features and channel features to obtain more robust features, thereby reconstructing a clearer image.

[0048] The proposed method uses DIV2K and Flickr2K as training data, and creates a dataset by synthesizing low-resolution images for training using bicubic downsampling and Gaussian blur with unknown parameters as degradation functions. For the parameter setting of the network model, the network learning rate is initially set to 2×10 -4 , and adjust the learning rate at the same interval. In addition, the network uses the Adam optimizer and the batch size (mini-batch) is set to 4. After nearly 300 epochs of training in the first stage, the training loss gradually converged. After nearly 600 epochs of training in the second stage, the training loss gradually converged. The window size of the shifted window self-attention is set to 8×32, the channel size of the shifted window self-attention and channel self-attention is set to 180, and the channel size of the content-aware degradation is set to 256.

[0049] In order to verify the effectiveness of the present invention, comparative tests were conducted on images reconstructed using StableSR using the Unet denoising network and three other network models. Among these three network models, in addition to the network model proposed in the present invention, two more popular network models, DASR and Deep Constrained Least Squares for Blind Image Super-Resolution (DCLS), were also used. All models used the same degradation parameters to synthesize training data and used the same test data for reconstruction. Figure 3 The reconstruction results shown in (a) to (d) are respectively the reconstruction results of DASR, DCLS, StableSR (Exploiting Diffusion Prior for Real-World Image Super-Resolution) and the network model proposed in the present invention. Afterwards, the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) of different methods are calculated to compare the performance of the reconstruction results of each scheme. The experimental results show that the network model proposed in the present invention has the highest PSNR and SSIM.

[0050] The above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable people familiar with the technology to understand the content of the present invention and implement it accordingly, and they cannot be used to limit the protection scope of the present invention. Any equivalent transformation or modification made according to the spirit of the present invention should be included in the protection scope of the present invention.

Claims

1. A blind image super-resolution reconstruction method based on diffusion model and attention mechanism, characterized in that: The following steps are involved: (1) constructing a blind image super-resolution reconstruction network, including a content-aware degradation generation module and a super-resolution reconstruction module; the content-aware degradation generation module includes a content-aware encoder, an LR encoder and a denoising network; (2) In the first stage of training, the low-resolution image and the high-resolution image pair are used as input to train the content-aware encoder and the super-resolution reconstruction network, generating a true content-aware degradation prior and a super-resolution reconstruction network based on the content-aware degradation prior; (3) In the second stage of training, the low-resolution image is used as input to train the LR encoder and the denoising network of the diffusion model; (4) Using the trained LR encoder and diffusion model to generate a content-aware degradation prior to replace the real content-aware degradation prior in step (1), and then fine-tuning the super-resolution reconstruction network in step (1) to finally complete the reconstruction of the low-resolution image; The content-aware encoder implementation process of step (1) is as follows: The low-resolution and high-resolution image pairs are fed into several degradation blocks consisting of convolutional layers, linear rectification functions and convolutional layers, and then passed through a pooling layer to obtain the degradation prior; the high-resolution image is fed into several content branches consisting of convolutional layers, linear rectification functions and convolutional layers, and then passed through a pooling layer to obtain the content prior; finally, the concatenation of the degradation prior and the content prior is fed into a multi-layer perceptron to obtain the true content-aware degradation prior; In the first stage of training in step (2), the content-aware encoder and the super-resolution reconstruction module are combined to optimize the entire blind image super-resolution reconstruction network by minimizing the L1 norm loss of the reconstructed image and the high-resolution image, thereby training a content-aware degradation encoder that can extract the true content-aware degradation prior. The entire network uses the L1 norm loss as the loss function: Among them, I HR For a real high-resolution image, I SR represents the reconstructed image; The super-resolution reconstruction module includes a plurality of content degradation residual modules CDRB, and for each CDRB, an affine transformation is performed on the input features using a content-aware degradation prior; then a serial structure of a shifted window self-attention mechanism and a channel self-attention mechanism is adopted to improve the attention mechanism effect of the model; channel information is supplemented for the shifted window self-attention mechanism through deep convolution, and spatial information is supplemented for the channel self-attention mechanism through deep convolution to output more robust features; In the second stage of training described in step (3), the content-aware encoder of the first stage is frozen, and the diffusion model is jointly trained with the super-resolution reconstruction module. The diffusion model is optimized by minimizing the L1 norm loss of the content-aware degradation prior generated by the diffusion model and the true content-aware degradation prior. At the same time, the L1 norm loss of the reconstructed image and the high-resolution image is minimized to fine-tune the super-resolution reconstruction network: Among them, I HR For a real high-resolution image, I SR represents the reconstructed image, Z0 is the true content-aware degradation prior, represents the generated content-aware degradation prior, α diff is the weight coefficient of the diffusion model loss.

2. According to claim 1, a blind image super-resolution reconstruction method based on diffusion model and attention mechanism is characterized in that: The denoising network in step (1) consists of five fully connected layers.

3. The blind image super-resolution reconstruction method based on diffusion model and attention mechanism according to claim 1, characterized in that: The degradation block is composed of a standard 3×3 convolution, a linear rectification function and a standard 3×3 convolution. At the same time, the standard 3×3 convolution in each degradation block is padded with a step size of 1, and then an adaptive average pooling layer is used to pool the features into a 1×1 vector.

4. The blind image super-resolution reconstruction method based on diffusion model and attention mechanism according to claim 1, characterized in that: The first content block in the content branch is composed of a 12×12 convolution, a linear rectification function and a standard 3×3 convolution.

5. The blind image super-resolution reconstruction method based on diffusion model and attention mechanism according to claim 1, characterized in that: The subsequent content blocks in the content branch are composed of a standard 3×3 convolution and a linear rectification function, and finally an adaptive average pooling layer is used to pool the features into a 1×1 vector.

6. The blind image super-resolution reconstruction method based on diffusion model and attention mechanism according to claim 4, characterized in that: The stride of the 12×12 convolution is set to 12.

Images