A real-world image super-resolution method based on stable diffusion

Abstract

The present invention discloses a real-world image super-resolution method based on stable diffusion, which relates to the field of image super-resolution technology, and is specifically implemented by constructing and training an enhanced prior-guided diffusion model; the process of implementing real-world image super-resolution by the enhanced prior-guided diffusion model is as follows: a preprocessing module generates a preprocessed image based on a low-resolution image, a segmentation module generates a corresponding segmentation mask image based on the preprocessed image, and an attention fusion module generates fused latent features based on the latent features of the preprocessed image and the segmentation mask image; a semantic cue extractor extracts high-quality semantic cue words from the preprocessed image and inputs them into a denoising backbone network; a control network performs feature modulation on the denoising backbone network based on the fused latent features; and the denoising backbone network outputs a high-resolution image. The present invention pays more attention to the high-frequency details and structural levels of the image in the real-world image super-resolution task, and ensures the authenticity and consistency of the restored image.

Description

Technical Field

[0001] The present invention relates to the technical field of image super-resolution, and in particular to a real-world image super-resolution method based on stable diffusion. Background Art

[0002] Initially, the research on image super-resolution focused on generating LR images through predetermined degradation operators (including bicubic downsampling, blurring, Gaussian noise, etc.), and using low-resolution degraded images and high-resolution real images to achieve super-resolution through end-to-end network training (such as CNN, Transformer). However, due to the goal of fidelity, there is a problem of over-smoothing of details, and the predetermined degradation operator cannot simulate the degradation of real-world images. Generative adversarial networks (GANs) and their adversarial training strategies are widely used in real image super-resolution. In order to improve the generalization ability and better fit the degradation of the real world, BSRGAN proposed to use a degradation operation of random shuffle combination, while Real-ESRGAN used a high-order process to simulate the degradation of the real world. Later, a series of methods such as SwinIR and HAT used the prior ability of transformer to reconstruct HR images and achieved gratifying results. Other studies (GANs) also found that the implicit prior method was used through adversarial training, but due to the nature of the GANs network, HR images have problems with artifacts and unrealistic textures.

[0003] GANs have achieved many results in the field of image generation, and there are rich priors in pre-trained GANs. GLEAN uses priors to achieve image super-resolution through a single forward pass. On this basis, in order to be applied to real-world scenes, FeMaSR uses matching with undistorted HR in feature space to obtain high-quality images in order to compensate for the missing details of the image. MM-RealSR proposes an unsupervised degradation estimation strategy to estimate the degree of degradation in real-world scenes, and combines a metric learning strategy to achieve excellent restoration performance through unsupervised training.

[0004] With the continuous progress of large-scale text-to-image (T2I) diffusion models, these models are trained on a large number of high-definition datasets, providing a solid foundation and rich prior knowledge for solving complex image restoration tasks. As a pioneering work, StableSR trained a time-aware encoder and used the prior information of DMs to achieve good results in real-world super-resolution. DiffBIR uses a bipolar pipeline to combine SwimIR and DM and fine-tune ControlNet to achieve blind image restoration. In order to maintain a consistent pixel-level structure, PASD introduces a pixel-aware cross-attention module to achieve real-world image super-resolution and personalized image stylization. Despite the good progress in visual quality, the potential of T2I diffusion models has not been fully utilized in the restoration of high-frequency information and structural levels of real-world images. Summary of the invention

[0005] The purpose of the present invention is to solve the problem that the existing technology is still insufficient in restoring high-frequency information and structural levels in super-resolution processing of real-world images.

[0006] The technical solution adopted by the present invention to solve the technical problem is: to provide a real-world image super-resolution method based on stable diffusion, comprising the following steps:

[0007] Create a data set and select the stable diffusion model as the pre-training model;

[0008] Constructing attention-based attention fusion module, semantic cue extractor, and control network;

[0009] The attention fusion module, the semantic cue extractor and the control network are introduced into the pre-trained model to construct an enhanced prior-guided diffusion model; the enhanced prior-guided diffusion model also includes a preprocessing module and a segmentation module, the preprocessing module generates a preprocessed image according to the input low-resolution image; the segmentation module generates a corresponding segmentation mask image according to the preprocessed image; the attention fusion module integrates the potential features of the preprocessed image and the segmentation mask image to generate fused potential features; the semantic cue extractor extracts high-quality semantic cue words from the preprocessed image and inputs them into the denoising backbone network of the stable diffusion model; the control network performs feature modulation on the denoising backbone network according to the fused potential features; the denoising backbone network outputs a high-resolution image under the feature modulation of the control network;

[0010] An enhanced prior-guided diffusion model is trained based on the dataset, and the trained enhanced prior-guided diffusion model is used to achieve real-world image super-resolution.

[0011] Preferably, the preprocessing module is an Omni-SR model, the low-resolution image is input into the Omni-SR model to improve the fidelity and obtain a preprocessed image; the segmentation module is a pretrained SAM model, and the preprocessed image is obtained by the pretrained SAM model to obtain a segmentation mask image.

[0012] Preferably, the attention fusion module utilizes channel attention to integrate the latent features of the preprocessed image and the segmentation mask image to generate a fused latent feature, comprising the following steps:

[0013] Latent features of preprocessed images and the latent features of the segmentation mask Perform element-by-element summation to obtain integrated features ;

[0014] Integration Features Input local attention and global attention respectively, broadcast and add the output of local attention and the output of global attention, and obtain the first weight result through Sigmoid activation function;

[0015] The first weighted result is input into the local attention and global attention respectively, the output of the local attention and the output of the global attention are broadcast and added, and the fused potential features are obtained through the Sigmoid activation function.

[0016] Preferably, the local attention includes two convolutional layers, two group normalization layers and a nonlinear activation function Relu, and its internal calculation process is expressed as:

[0017] ;

[0018] in, Represents the integrated features or first weighted results of the input local attention, represents the local attention operation, It is represented by the nonlinear activation function Relu, B() is the group normalization operation, and Represent the point-wise convolution operations of the two convolutional layers respectively.

[0019] Preferably, the global attention includes an average pooling layer, two convolutional layers, a group normalization layer and a nonlinear activation function Relu, and its internal calculation process is expressed as:

[0020] ;

[0021] g ( z ') = 1 H × W ∑ i = 1 H ∑ j = 1 W z ' [ :, i , j ] ;

[0022] in, represents the integrated features or the first weighted result of the input global attention, G() represents the global attention operation, represents the nonlinear activation function Relu, B() is the group normalization operation, and Represent the point-wise convolution operations of the two convolutional layers respectively, represents the output of the average pooling operation, z ' [ : , i , j ] express The characteristic area is elements; H and W represent The height and width of the

[0023] Preferably, the process of extracting high-quality semantic cue words by the semantic cue extractor and inputting them into the denoising backbone network comprises the following steps:

[0024] Process the preprocessed image through an image encoder to obtain its corresponding image embedding;

[0025] Perform label extraction on image embedding to obtain corresponding label embedding;

[0026] Traverse the tag embeddings to extract the corresponding tags, and finally obtain all the tags as high-quality semantic prompt words;

[0027] High-quality semantic cue words enter the CLIP encoder of the denoising backbone network to generate multimodal features, including image features and text features ;

[0028] The first cross attention in the CLIP encoder inputs the multimodal features into the backbone denoising network by weighted summation of weight parameters, expressed as:

[0029] ;

[0030] ;

[0031] ;

[0032] ;

[0033] Among them, Q1, K1 and V1 represent the query, key and value of the first cross attention respectively; , and They represent the weight parameters of the projection layer of the query layer, key layer, and value layer respectively, T represents transposition; d1 represents the number of channels of the multimodal feature;

[0034] Using high-quality semantic cue words as positive cue words, the noise prediction guided to the denoising backbone network without a classifier is expressed as:

[0035] ;

[0036] in, and They represent the conditional noise prediction and unconditional noise prediction performed by the denoising backbone network, respectively. and Represent positive prompt words and negative prompt words respectively, Indicates the first The potential characteristics of the step, represents the bootstrap scale parameter.

[0037] Preferably, the control network includes a conditional network and a frozen denoising backbone network, and the process of implementing feature modulation by the control network includes the following steps:

[0038] The conditional network receives the fused latent features output by the attention fusion module, and at the same time receives the initial noise or the latent noise obtained by the denoising backbone network, and outputs the skip connection features;

[0039] The conditional network is connected to the frozen denoising backbone network through zero convolution, and the modulation of the intermediate layer features and the jump connection features is achieved through the second cross attention, which is expressed as:

[0040] ;

[0041] ;

[0042] ;

[0043] ;

[0044] Among them, Q2, K2 and V2 represent the query, key and value of the second cross attention respectively; , and Represent the weight parameters of the projection layer of the query layer, key layer and value layer respectively, and T represents transpose; represents the latent features of the decoder in the denoising backbone network, is the skip connection feature output by the conditional network; d2 represents the number of channels of the intermediate layer features and the skip connection features.

[0045] Preferably, the training of the enhanced prior guided diffusion model based on the data set includes training of a semantic cue extractor, which is specifically achieved by minimizing the following training objectives:

[0046] ;

[0047] in, is the balance parameter, and Preprocess the images Representation embedding and logic embedding, and High-resolution images Representation embedding and logic embedding, is the mean square error loss, is the cross entropy loss.

[0048] Preferably, the training of the enhanced prior guided diffusion model based on the data set includes training of the control network, which is specifically achieved by minimizing the following training objectives:

[0049] ℒ =  z t , t , z r , c , ε [ ‖ ε − ε θ ( z t , z F , t , c ) ‖ 2 2 ] ;

[0050] in, represents noise, represents the output of the denoising backbone network, Indicates the first The potential characteristics of the step, is the fused latent feature output by the attention fusion module, is the prompt word, t time step, Express expectations.

[0051] Preferably, the method of using the trained enhanced prior-guided diffusion model to achieve real-world image super-resolution comprises the following steps:

[0052] The real-world low-resolution image is input into the preprocessing module to obtain a preprocessed image, and the segmentation module generates a segmentation mask image according to the preprocessed image;

[0053] The preprocessed image and the segmentation mask image are input into the attention fusion module for potential feature fusion to obtain fused potential features;

[0054] Input the preprocessed image into the semantic cue extractor to obtain high-quality semantic cue words;

[0055] The control network receives the fused latent features and the initial noise or the latent noise obtained by the denoising backbone network, and performs feature modulation on the denoising backbone network of the stable diffusion model;

[0056] The denoising backbone network denoises the initial noise and generates real-world high-resolution images under the feature modulation of the control network.

[0057] The present invention has the following beneficial effects: in the real-world image super-resolution task, compared with the diffusion model based on stable diffusion, the present invention can focus on the high-frequency details and structural levels of the image, thereby ensuring the authenticity and consistency of the restored image; by introducing the prior by using the Fusion in attention (FIA) module, using the Semantic Prompt Extractor (SPE) to provide high-quality prompt words, and combining the control of the conditional network, the pixel-level restoration of the image is achieved. When faced with complex scene restoration, not only can better texture details be achieved while ensuring authenticity and consistency, but the scene semantics can also be perceived to avoid regional confusion of the restored image.

[0058] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments, but the present invention is not limited to the embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 A method step diagram of an embodiment of the present invention;

[0060] Figure 2 is a structural diagram of an attention fusion module (Fusion in attention, FIA) according to an embodiment of the present invention;

[0061] Figure 3 4 is a structural diagram of a semantic prompt extractor (SPE) according to an embodiment of the present invention;

[0062] Figure 4 is a structural diagram of a control network (FRControlNet) according to an embodiment of the present invention;

[0063] Figure 5 It is a network framework diagram of an enhanced priori-guided diffusion model (EPDIFF) according to an embodiment of the present invention;

[0064] Figure 6 A visual comparison diagram of the enhanced prior-guided diffusion model of an embodiment of the present invention and other super-resolution methods on a synthetic data set;

[0065] Figure 7 This is a visual comparison diagram of the enhanced prior-guided diffusion model of an embodiment of the present invention and other super-resolution methods on a real-world dataset. DETAILED DESCRIPTION

[0066] See also Figure 1 FIG. 1 is a diagram showing steps of a method according to an embodiment of the present invention, comprising the following steps:

[0067] S101, prepare a data set and select a stable diffusion model as a pre-training model;

[0068] S102, constructing the attention-based attention fusion module, semantic cue extractor, and control network;

[0069] S103, introduce the attention fusion module, semantic cue extractor and control network into the pre-trained model to construct an enhanced prior guided diffusion model;

[0070] S104, training an enhanced prior guided diffusion model based on the data set, and using the trained enhanced prior guided diffusion model to achieve real-world image super-resolution.

[0071] Specifically, the enhanced prior-guided diffusion model also includes a preprocessing module and a segmentation module. The preprocessing module generates a preprocessed image based on the input low-resolution image; the segmentation module generates a corresponding segmentation mask image based on the preprocessed image; the attention fusion module integrates the potential features of the preprocessed image and the segmentation mask image to generate fused potential features; the semantic cue extractor extracts high-quality semantic cue words from the preprocessed image and inputs them into the denoising backbone network of the stable diffusion model; the control network performs feature modulation on the denoising backbone network according to the fused potential features; the denoising backbone network outputs a high-resolution image under the feature modulation of the control network;

[0072] In order to further explain the technical solution of the present invention, the present invention is described in detail below through specific embodiments.

[0073] 1. Create a data set and select a pre-trained model

[0074] In order to demonstrate the generalization of the model, training and testing are performed on multiple datasets with different data distributions. First, for the training set, natural images from DIV2K, DIV8K, Flickr2K, OST, and the first 5k images of the face image dataset FFHQ are used. For the test set, three real-world benchmark datasets, RealSR, DRealSR, and DPED-iphone, are used. In addition, the validation set in DIV2K is used as an additional test set. Images with incorrect image format, images that are too similar, and images that are blurred are deleted from the dataset. The images of the retained training and test sets are first cropped to ensure that the HR images are 512×512, and then the Real-ESRGAN degradation pipeline is used to generate the corresponding LR images with a resolution of 128×128, which is more in line with the degradation conditions in the real world.

[0075] To ensure the robustness of the model, the pre-trained Stable Diffusion is used as the generative prior. Stable Diffusion is trained on the Laion-2B-en dataset, which contains 2.32 billion images and English text. Therefore, Stable Diffusion will contain a lot of prior knowledge to ensure that the model can be applied to various complex scenarios.

[0076] Verification images under synthetic datasets and real-world datasets are verified. To ensure that the test indicators can better reflect human perception, the test indicators will include reference indicators and non-reference indicators: reference indicators include FID (↓), LPIPS (↓), PNSR (↑), SSIM (↑), FID and LPIPS are used for perceptual quality measurement, and PNSR and SSIM (evaluated on the Y channel of the YCbCr color space) are used to evaluate fidelity. For non-reference indicators, NIQE (↓), MUSIQ (↑), CLIPIQA (↑), and MANIQA (↑) are used to evaluate the quality of generated images.

[0077] 2. Construct an attention-based Fusion in Attention (FIA) module

[0078] like Figure 2 As shown in FIG. 1 , the network structure diagram of the attention fusion module (FIA) based on attention in the present invention is shown. The FIA ​​module is used to integrate all the potential feature information of the conditional image (i.e., the preprocessed image and the segmentation mask image) to improve the quality of guidance. The FIA ​​module uses channel attention to strengthen the control of potential information. Assume that the potential feature is and , firstly, the integrated feature Z is obtained by element-by-element summation, and then the integrated features are input into the local attention and global attention respectively. Among them, the attention mechanism realizes channel attention operation at multiple scales by adjusting the spatial pooling size. Point-wise convolution is selected as the attention mechanism aggregator. The local attention module contains two groups of point-wise convolution, a group normalization and a linear rectification function. The calculation of the local attention module is as follows:

[0079] ;

[0080] in, is the rectified linear function (RELU), and B is the group normalization (BN). The output of the local attention maintains the same shape as the input feature to preserve the details of the potential feature information.

[0081] The global attention performs an average pooling operation (GAP) on the input features, and its hourglass structure is calculated as follows:

[0082] ;

[0083] g ( z ') = 1 H × W ∑ i = 1 H ∑ j = 1 W z ' [ :, i , j ] ;

[0084] The process of broadcasting and adding local attention and global attention and passing through the Sigmoid activation function is called the weight process. At the same time, in order to avoid the initial integration quality affecting the fusion weight, the potential features of the input are fused by repeating the attention process, and the FIA ​​module can be expressed as:

[0085] ;

[0086] ;

[0087] in, Represents dot product.

[0088] The local attention specifically includes: two convolution layers, whose input dimension is 4, output dimension is 4, convolution kernel is 1, and stride is 1; two group normalization layers; a nonlinear activation function Relu; the global attention specifically includes: an average pooling layer; two convolution layers, whose input dimension is 4, output dimension is 4, convolution kernel is 1, and stride is 1; a group normalization layer; a nonlinear activation function Relu; the input of the FIA ​​module contains two tensor variables, both of which are variables in the latent space, and the data dimension is 1×4×64×64. Assume that one of the variables is , and the other variable is First, add the two variables through broadcasting to get the initial integration weight , whose dimensions are 1×4×64×64. Then the initial integration weights Input into the local attention, after convolution layer and group normalization, the initial feature weight 1 is obtained through ReLU. The obtained initial weight is further input into the global attention to obtain the feature weight 2. The dimensions of the two feature weights are 1×4×64×64. After adding the feature weights 1 and 2 and passing through the sigmoid activation function, the integrated weight 1 is obtained, which is expressed as:

[0089] ;

[0090] To avoid initial integration weights The randomness brought about by this is repeated to obtain the final fusion potential features, which are expressed as:

[0091] , whose dimension is 1×4×64×64. The fused latent features have the same dimension as the input latent features, which preserves the subtle details of their initial features and plays a vital role in guiding subsequent downstream tasks.

[0092] 3. Build a Semantic Prompt Extractor (SPE)

[0093] See also Figure 3 As shown in FIG. 1 , the structure diagram of the semantic hint extractor of an embodiment of the present invention is shown. The present invention adopts a pre-trained text-to-image diffusion model, which supports text as a hint during inference. However, many SD-based methods use empty text as a hint, which cannot fully utilize the potential of the diffusion model. Therefore, the Recognize Anything Model (RAM) is fine-tuned to obtain the SPE module. First, the HR image Input the frozen label model to obtain the representation embedding and logic embedding Next, the preprocessed image The input is a trainable image encoder and a hint extractor. Then, the image is preprocessed by forcing The representation embedding and logic embedding of HR images are close to The representation embedding and logic embedding enable the SPE module to effectively Extract high-quality semantic cue words from

[0094] The training objectives of the SPE module are as follows:

[0095] ;

[0096] in, is the balance parameter, and Preprocess the images Representation embedding and logic embedding of . is the mean square error (MSE) loss, is the cross entropy loss. By minimizing the loss, the SPE module can extract high-quality semantic cues from the preprocessed images.

[0097] After training, the SPE module can extract reliable semantic cues. The cue words are encoded by the CLIP text encoder, and the multimodal features generated are input into Unet through cross-attention to realize the conversion of text information into image-level features.

[0098] Specifically, the latent image features encoded by CLIP and text features After transformation at the projection layer, the query is obtained ,key Sum .in , and It is the weight parameter of the projection layer of the query layer, key layer and value layer.

[0099] ;

[0100] Use the obtained prompt word as the positive prompt word and use no classifier to guide: ,in and For conditional and unconditional predict, and For the front and back text prompts, For the The potential characteristics of the step, To guide the scale parameter. In practice, the positive prompt words are additionally added with "clean, 8K, high resolution", while the negative prompt words are set to "dot, noise, blur, low resolution, smooth". Of course, these prompt words can be added selectively. By passing the prompt words to the diffusion model, the local understanding ability of the model is enhanced, the semantic correctness of the image is guaranteed, and the SR performance is greatly improved.

[0101] Specifically, the processing of the preprocessed image by the semantic hint extractor includes the following steps:

[0102] The preprocessed image is input into the image encoder in the semantic cue extractor to obtain its corresponding image embedding.

[0103] The image embedding is input into Extract Prompts, and classification is completed according to the final linear layer to obtain the corresponding tag embedding;

[0104] By traversing the tag embedding to extract the corresponding tokens, all the tokens are finally obtained as high-quality prompt words.

[0105] Specifically, the detailed processing flow of the semantic hint extractor on the preprocessed image includes:

[0106] First, the input of the SPE module is a 512×512 image. In order to change the image into a variable suitable for the model, the image is changed into a tensor through transforms and its dimension is set to 1×3×512×512. Then the tensor is cropped and normalized to reduce parameters and calculations. Finally, the input dimension of the SPE module is 1×3×384×384.

[0107] Secondly, SPE sets num_class to 4585 to extract keywords, and generates label_embed with a dimension of 1×4585×768 through random initialization. In order to obtain the high-dimensional features of the image and transfer the image from the image space to the latent space, the image is passed through the SwinTransformer class, which is a hierarchical visual Transformer using a shifted window. In this way, image_embed is obtained with a dimension of 1×145×512. In order to use the attention mechanism for image_embed and better extract the prompt words, a 1×145 image_atts tensor variable is generated to capture the correlation between different parameters;

[0108] Secondly, after obtaining image_embed, label_embed, and image_atts, these three variables are input into BertModel. BertModel contains three components: BertEncoder, Bertintermediate, and BertOutput. BertEncoder implements cross-attention calculation on variables, and performs attention calculation on its internal correlation through the QKV matrix generated by the variables to generate an attention matrix. Bertintermediate performs nonlinear changes through the Linear layer and GeluActivation(). BertOutput classifies variables through the Linear layer, layerNorm layer, and Dropout layer to help extract prompt words. BertModel finally outputs the tagging_embed variable with a dimension of 1×4585×768. The image prompt words are obtained by further analyzing the tagging_embed variable;

[0109] Finally, the logical value is extracted from the tagging_embed variable to obtain the variable logits with a dimension of 1 × 4585. Then the variable logits is traversed and the corresponding token is extracted according to the label and stored in the tag_output variable. After the traversal is completed, tag_output is the obtained high-quality prompt word.

[0110] 4. Constructed a new control network FRControlNet

[0111] Inspired by ControlNet, the embodiment of the present invention designs a conditional network of FRControlNet. Based on the pre-trained Unet network, its encoder and intermediate layers are used as trainable copies. This replication strategy provides a good weight initialization for the network and avoids performance loss during training. The input of the FRControlNet conditional network includes the potential features output by FIA and the potential noise obtained through the denoising backbone network .in Contains each The randomness of the moment can accelerate the convergence of the model and improve the accuracy of the model. The final input is expressed as In order to avoid using random noise as gradients in the early stages of training, some parameters are introduced to the first layer of the model and initialized to zero.

[0112] The conditional network uses its output to modulate multi-scale features, and connects the conditional network with the frozen Unet denoising network through zero convolution to achieve modulation of intermediate layer features and skip connection features.

[0113] ;

[0114] is the latent feature of the decoder in the denoising network, is the skip connection feature output by the conditional network. , , .

[0115] During the training process, only the parameters of the conditional network and feature modulation are updated, and the training target is achieved by MSE Loss, which is expressed as:

[0116] ;

[0117] Specifically, the structure of the control network FRControlNet of the embodiment of the present invention is shown in Figure 4 As shown. FRControlNet contains a conditional network and a frozen Unet denoising network. The conditional network consists of four important components: time embedding layer (time_embed), input layer (input_blocks), middle layer (middle_block), zero convolution layer (zero_convs);

[0118] The time embedding layer first passes through a Linear layer, whose input feature is 320 and output feature is 1280. Then it is linearly transformed through SiLU(), and finally passes through a Linear layer, whose input feature is 1280 and output feature is 1280. The time embedding layer is used to embed time information into the model and process time features through linear transformation and activation function;

[0119] The input layer (input_blocks) contains 11 time step embedding sequential blocks (TimestepEmbedSequential), each time step embedding sequential block consists of a residual block, a spatial transformer (SpatialTransformer), and a downsampling layer (Downsample);

[0120] The residual block includes in_layer, emb_layer, out_layer, and skip_connection. The in_layer first passes through the normalization layer (Group Normalization) to normalize the input features to accelerate convergence, then passes through the activation function SiLU(), and finally passes through the convolution layer. The number of channels of the input and output are both 320, the convolution kernel size is 3×3, the step size is 1, and the padding is 1. The emb_layer first passes through the SiLU() activation function, and finally passes through the fully connected layer to reduce the dimension of the input features from 1280 to 320. The out_layer contains one more Dropout layer than the in_layer, which is used between the activation function and the convolution layer to prevent overfitting. Finally, there is a skip connection layer (skip_connection), that is, a convolution layer with an input channel number of 320, an output channel number of 640, a convolution kernel of 1, and a step size of 1. The details and semantic information in the original input data are retained. It makes it easier for data to propagate to the subsequent layers, avoiding the loss of details. The purpose of the residual block is to promote the effective propagation of gradients through residual connections, while introducing group normalization and activation functions to enhance the network's expressiveness and training stability;

[0121] The spatial transformer (SpatialTransformer) includes a norm layer, a proj_in layer, a transformer_blocks layer, and a proj_out layer. The norm layer is a group normalization layer that normalizes the input features into groups. There are 32 groups here, and each group contains 640 features. The proj_in layer is a fully connected layer that maps the dimension of the input features from 640 to 640. The transformer_blocks layer contains two MemoryEfficientCrossAttentions and a feedforward neural network (FeedForward). MemoryEfficientCrossAttention completes the calculation of the QKV matrix through a linear layer, and the linear transformation layer and dropout process the output. The feedforward neural network maps the input features to a higher dimension through a generalized gated linear unit (GEGLU). The dropout operation reduces fitting. The linear mapping layer maps the input features back to the original dimension. Finally, the final features are mapped back to 640 dimensions through the proj_out layer, which is a fully connected layer;

[0122] The downsampling layer (Downsample) is not possessed by each time-step embedding sequence block. Assuming that there are two time-step embedding sequence blocks, the third time-step embedding sequence block is the downsampling layer and does not contain residual blocks and spatial transformers. The downsampling layer appears after every two time-step embedding sequence blocks. The downsampling layer passes through a convolution layer with 320 input channels and 320 output channels. It uses a 3×3 convolution kernel, a stride of 2, and a padding of 1 for convolution operations. The downsampling layer achieves downsampling by reducing the size of the feature map. It effectively reduces the spatial dimension of the feature map to reduce the amount of computation and the number of parameters, while extracting more advanced features;

[0123] The middle layer (middle_block) includes two time-step embedding sequential blocks, and the modules included in the time-step embedding sequential blocks are consistent with the input layer. Since there is no third time-step embedding sequential block, it does not include a downsampling layer;

[0124] The zero convolution layer (zero_convs) contains 12 convolution layers. The input and output channels of the first four zero convolution layers are both 320, using a 1×1 convolution kernel with a stride of 1. The input and output channels of the three middle zero convolution layers are changed to 640. The input and output channels of the last five zero convolution layers are changed to 1280. The conditional network in FRControlNet is connected to the Unet network through the zero convolution layer to achieve feature modulation.

[0125] 5. Construct the network framework of Enhanced Priori-guided Diffusion Model (EPDiff).

[0126] See also Figure 5 The figure shows the network framework of the Enhanced Priori-guided Diffusion Model (EPDiff). The pre-trained Stable Diffusion 2.1-base is used as the generated prior. The obtained FIA module, SPE module and FRControlNet are introduced into the network. In order to improve the prior of the model, the additional Omni-SR model and Segment Anything Mode (SAM) are introduced as preprocessing modules. The above modules and models constitute the Enhanced Priori-guided Diffusion Model (EPDiff) network framework.

[0127] The processing flow of EPDiff includes the following three stages. In the first stage, in order to ensure the authenticity of the texture and obtain the structural hierarchy of the image, Omni-SR is first used to improve the fidelity of the low-quality LR image, and then the preprocessed image is passed through the pre-trained SAM model to obtain an image with a segmentation mask. In the second stage, in order to effectively inject the prior into the EPDiff model, the preprocessed image and the segmentation mask image are input into the FIA ​​module for latent feature fusion to obtain the final guided feature. At the same time, in order to give full play to the potential of the text-to-image diffusion model, the SPE module is used, which can extract accurate prompt words from the preprocessed image. In the third stage, the SD model is precisely controlled by FRControlNet to generate structurally clear and semantically correct details. A large number of experiments show that the EPDiff retains fidelity while generating more complex textures.

[0128] The overall operation process of the enhanced prior guided diffusion model EPDiff is as follows:

[0129] The model input is a low-quality LR image of 128×128. The LR image first enters the Omni-SR model for preliminary degradation removal. The input layer of Omni-SR is a convolution layer with 3 input channels, 64 output channels, and 1 convolution kernel and stride. The LR image is converted into a tensor of 1×64×128×128. In order to extract the features of the module and fill the missing pixel values, the variables are sequentially input into the 5 OSAG layers of Omni-SR. The output dimension is 1×64×128×128. Finally, the variables are reconstructed into a restored SR image with a dimension of 3×512×512 through PixelShuffle. In order to obtain the segmentation mask, the initially restored SR image is input into SAM. In order to be suitable for the SAM model, the SR image is first converted into a tensor of 1×3×1024×10241. Then, the texture details of variable 1 are extracted through SAM's image encoding layer (image_encoder), the key information of the variable is retained, and a 1×256×64×64 tensor 2 is output. In order to achieve automatic segmentation of the mask, the SAM model generates point and box embedding information with dimensions of 64×2×256 and mask embedding information with dimensions of 64×256×64×64 according to tensor 2 through the prompt encoder (prompt_encoder). The generated prompt information and tensor 2 are input into the mask decoder (mask_decoder), and the mask variable 64×3×256×256 is output. This is a relatively rough mask information. In order to ensure the high quality of the mask, the mask variable is obtained multiple times and the final mask image 3×512×512 is output through upsampling.

[0130] After obtaining the preliminary restored image and mask image, they are input into the pre-trained VAE to obtain their potential feature representations, and the dimensions are all 1×4×64×64. Then they are sent to the FIA ​​module to extract key feature information and send it to the pre-trained Stable Diffusion. At the same time, the SPE module also extracts key prompt words from the preliminary restored image and integrates them into the Stable Diffusion through the cross attention layer to improve the model's semantic understanding of the scene. Finally, FRControlNet uses its conditional network to modulate the features of the Stable Diffusion denoising network, and finally outputs a HR high-quality image.

[0131] Specifically, the method of using the trained enhanced prior-guided diffusion model to achieve real-world image super-resolution includes the following steps:

[0132] The real-world low-resolution image is input into the preprocessing module to obtain a preprocessed image, and the segmentation module generates a segmentation mask image according to the preprocessed image;

[0133] The preprocessed image and the segmentation mask image are input into the attention fusion module for potential feature fusion to obtain fused potential features;

[0134] Input the preprocessed image into the semantic cue extractor to obtain high-quality semantic cue words;

[0135] The control network receives the fused potential features and the initial noise or the potential noise obtained by the denoising backbone network, and performs feature modulation on the denoising backbone network of the stable diffusion model; specifically, the initial noise (i.e., Gaussian noise) is received in the first iteration, and the potential noise obtained by the denoising backbone network is received in subsequent iterations;

[0136] The denoising backbone network denoises the initial noise and generates real-world high-resolution images under the feature modulation of the control network.

[0137] It can be seen that compared with the diffusion model based on Stable Diffusion, the present invention can focus on the high-frequency details and structural levels of the image in the real-world image super-resolution task, thereby ensuring the authenticity and consistency of the restored image. By using the FIA ​​module to introduce the prior and the SPE module to provide high-quality prompt words, under the control of FRControlNet, when faced with complex scene restoration, the EPDiff network achieves pixel-level restoration of the image.

[0138] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A real-world image super-resolution method based on stable diffusion, characterized in that The following steps are involved: Create a data set and select the stable diffusion model as the pre-training model; Constructing attention-based attention fusion module, semantic cue extractor, and control network; An attention fusion module, a semantic cue extractor, and a control network are introduced into a pre-trained model to construct an enhanced prior-guided diffusion model; the enhanced prior-guided diffusion model also includes a pre-processing module and a segmentation module, and the pre-processing module generates a pre-processed image according to an input low-resolution image; The segmentation module generates a corresponding segmentation mask image based on the preprocessed image; The attention fusion module integrates the latent features of the preprocessed image and the segmentation mask image to generate fused latent features; The semantic cue extractor extracts high-quality semantic cue words from the preprocessed image and inputs them into the denoising backbone network of the stable diffusion model; the control network modulates the features of the denoising backbone network according to the fused latent features; The denoising backbone network outputs high-resolution images under the feature modulation of the control network; Based on the dataset, an enhanced prior-guided diffusion model is trained, and the trained enhanced prior-guided diffusion model is used to achieve real-world image super-resolution; The attention fusion module uses channel attention to integrate the latent features of the preprocessed image and the segmentation mask image to generate fused latent features, including the following steps: The latent feature X of the preprocessed image and the latent feature Y of the segmentation mask are summed element by element to obtain the integrated feature Z; The integrated feature Z is input into the local attention and global attention respectively, the output of the local attention and the output of the global attention are broadcast and added, and the first weight result is obtained through the Sigmoid activation function; The first weighted result is input into the local attention and global attention respectively, the output of the local attention and the output of the global attention are broadcast and added, and the fused potential features are obtained through the Sigmoid activation function.

2. The real-world image super-resolution method based on stable diffusion according to claim 1, characterized in that: The preprocessing module is the Omni-SR model. The low-resolution image is input into the Omni-SR model to improve the fidelity and obtain the preprocessed image. The segmentation module is the pre-trained SAM model. The preprocessed image is passed through the pre-trained SAM model to obtain the segmentation mask image.

3. The real-world image super-resolution method based on stable diffusion according to claim 1, characterized in that: Local attention consists of two convolutional layers, two group normalization layers and a nonlinear activation function Relu. Its internal calculation process is expressed as: Among them, Z′ represents the integrated feature or the first weight result of the input local attention, L() represents the local attention operation, It is represented by the nonlinear activation function Relu, B() is the group normalization operation, PWConv1() and PWConv2() represent the point-to-point convolution operations of the two convolutional layers respectively.

4. The real-world image super-resolution method based on stable diffusion according to claim 1, characterized in that: Global attention includes an average pooling layer, two convolutional layers, a group normalization layer and a nonlinear activation function Relu. Its internal calculation process is expressed as: Among them, Z′ represents the integrated feature or the first weight result of the input global attention, G() represents the global attention operation, represents the nonlinear activation function Relu, B() is the group normalization operation, PWConv1() and PWConv2() represent the point-wise convolution operations of the two convolutional layers, g(Z′) represents the output of the average pooling operation, and Z′ [:,i,j] Indicates the element whose feature region of Z' is at (i, j); H and W represent the height and width of Z' respectively.

5. The real-world image super-resolution method based on stable diffusion according to claim 1, characterized in that: The process of extracting high-quality semantic cue words by the semantic cue extractor and inputting them into the denoising backbone network includes the following steps: Process the preprocessed image through an image encoder to obtain its corresponding image embedding; Perform label extraction on image embedding to obtain corresponding label embedding; Traverse the tag embeddings to extract the corresponding tags, and finally obtain all the tags as high-quality semantic prompt words; High-quality semantic cue words enter the CLIP encoder of the denoising backbone network to generate multimodal features, including image features and text features The first cross attention in the CLIP encoder inputs the multimodal features into the backbone denoising network by weighted summation of weight parameters, expressed as: Where Q1, K1 and V1 represent the query, key and value of the first cross attention respectively; W Q1 , W K1 and W V1 They represent the weight parameters of the projection layer of the query layer, key layer, and value layer respectively, T represents transposition; d1 represents the number of channels of the multimodal feature; Using high-quality semantic cue words as positive cue words, the noise prediction guided to the denoising backbone network without a classifier is expressed as: in, and ε() represent the conditional noise prediction and unconditional noise prediction performed by the denoising backbone network, respectively, c and c neg Represent positive prompt words and negative prompt words respectively, z t represents the latent features of the t-th step in the denoising backbone network, and ω represents the guided scale parameter.

6. The real-world image super-resolution method based on stable diffusion according to claim 1, characterized in that: The control network includes a conditional network and a frozen denoising backbone network, and the process of implementing feature modulation by the control network includes the following steps: The conditional network receives the fused latent features output by the attention fusion module, and at the same time receives the initial noise or the latent noise obtained by the denoising backbone network, and outputs the skip connection features; The conditional network is connected to the frozen denoising backbone network through zero convolution, and the modulation of the intermediate layer features and the jump connection features is achieved through the second cross attention, which is expressed as: <h2 style=";text-align:left;direction:ltr">Q2=W<h2 style=";text-align:left;direction:ltr"> Q2 <h2 style=";text-align:left;direction:ltr"> ·D; K2=W K2 ·J; <h2 style=";text-align:left;direction:ltr">V2=W<h2 style=";text-align:left;direction:ltr"> V2 <h2 style=";text-align:left;direction:ltr"> ·J; Where Q2, K2, and V2 represent the query, key, and value of the second cross attention, respectively; W Q2 , W K2 and W V2 They represent the weight parameters of the projection layers of the query layer, key layer, and value layer respectively, T represents transpose; D represents the potential features of the decoder in the denoising backbone network, J represents the skip connection features output by the conditional network; d2 represents the number of channels of the intermediate layer features and the skip connection features.

7. The real-world image super-resolution method based on stable diffusion according to claim 5, characterized in that: The training of the enhanced prior guided diffusion model based on the data set includes the training of the semantic cue extractor, which is specifically achieved by minimizing the following training objectives: Among them, δ is the equilibrium parameter, f y re and l y lo are the preprocessed images Y p The representation embedding and logic embedding of f x re and l x lo They are high-resolution images X HR Representation embedding and logic embedding, is the mean square error loss, is the cross entropy loss.

8. The real-world image super-resolution method based on stable diffusion according to claim 6, characterized in that: The training of the enhanced prior guided diffusion model based on the data set includes the training of the control network, which is specifically achieved by minimizing the following training objectives: Among them, ε represents noise, ε θ (z t ,z F ,t,c) represents the output of the denoising backbone network, z t represents the potential features of the t-th step in the denoising backbone network, z F is the fused potential feature output by the attention fusion module, c is the prompt word, t is the time step, Express expectations.

9. The real-world image super-resolution method based on stable diffusion according to claim 1, characterized in that: The method of using the trained enhanced prior-guided diffusion model to achieve real-world image super-resolution includes the following steps: The real-world low-resolution image is input into the preprocessing module to obtain a preprocessed image, and the segmentation module generates a segmentation mask image according to the preprocessed image; The preprocessed image and the segmentation mask image are input into the attention fusion module for potential feature fusion to obtain the fused potential features; Input the preprocessed image into the semantic cue extractor to obtain high-quality semantic cue words; The control network receives the fused latent features and the initial noise or the latent noise obtained by the denoising backbone network, and performs feature modulation on the denoising backbone network of the stable diffusion model; The denoising backbone network denoises the initial noise and generates real-world high-resolution images under the feature modulation of the control network.

Images