Zero sample image super-resolution reconstruction method and system based on dense Unet diffusion model

Abstract

The invention discloses a zero-sample image super-resolution reconstruction method and system based on a dense Unet diffusion model. The zero-sample image super-resolution reconstruction method comprises the steps that the trained dense Unet diffusion model is adopted to carry out zero-sample image super-resolution reconstruction; the dense Unet diffusion model is obtained by introducing a dense Unet network structure in a forward diffusion process of a diffusion model without classifier guidance; according to the dense Unet network structure, on the basis of a Unet network structure, two DRB modules are additionally arranged in each layer of encoder and each layer of decoder; each DRB module comprises a time step coding layer, a label coding layer, a dense residual block and a time label direct embedding standardization layer; the dense residual block comprises four convolution layers and a parameter correction linear unit; the feature maps are densely connected; and the time label direct embedding standardization layer is used for firstly embedding a time sequence coding feature from the time step coding layer and then summing the time sequence coding feature with a label coding feature from the label coding layer so as to directly embed information into the dense residual block.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and specifically relates to a zero-sample image super-resolution reconstruction method and system based on a dense Unet diffusion model. Background Art

[0002] Image resolution is usually used as a performance evaluation of the degree of rich detail information in an image. It reflects the ability of the imaging system to actually capture and present the detailed information of an object. Compared with low-resolution images, high-resolution images usually have higher pixel density, richer texture details, and higher credibility. Image processing and visual scene applications are changing with each passing day, and their rapid development has triggered an urgent demand for high-resolution images. However, due to the limitations of actual conditions, such as the constraints of acquisition equipment and environment, network transmission media and bandwidth, image degradation model itself, and many other factors, it is a challenging task to obtain an ideal high-resolution image with sharp edges and no block blur.

[0003] Image super-resolution reconstruction technology (super-resolution, SR) refers to the process of restoring a low-resolution image to a corresponding high-resolution image through a specific algorithm. Specifically, image super-resolution reconstruction technology (super-resolution, SR) uses relevant knowledge in the fields of digital image processing, computer vision, etc., and uses specific algorithms and processing procedures to reconstruct a high-resolution image from a given low-resolution image. It aims to overcome or compensate for the limitations of the image acquisition system or the acquisition environment itself, especially to solve problems such as blurred imaging, poor quality, and insignificant region of interest (ROI).

[0004] In recent years, with the rapid development of deep learning technology, image super-resolution algorithms based on large-scale data have shown an increasingly diversified trend. However, their training process still requires a large amount of data for training. In real-life scenarios, it is difficult to obtain paired data, which makes the advantages of deep learning training methods less prominent.

[0005] Single image super-resolution based on zero-shot learning can perform super-resolution reconstruction without paired low-resolution-high-resolution image pairs, that is, reconstructing a high-resolution image only from the low-resolution image itself. When data is difficult to obtain, zero-shot super-resolution has certain advantages, while also reducing the cost of data collection and annotation. Moreover, zero-shot super-resolution reconstruction has a wider range of applicability, making it more convenient and efficient for practical applications of super-resolution reconstruction in multiple fields. Currently, the commonly used zero-shot image super-resolution reconstruction is based on the diffusion model, which can show excellent performance under the condition of unknown degradation function, but the existing diffusion model is not efficient in utilizing feature information. Summary of the invention

[0006] Purpose of the invention: In order to solve the problems of low efficiency in feature information utilization and low image reconstruction quality in the existing zero-sample image super-resolution reconstruction method based on the diffusion model, the present invention proposes a zero-sample image super-resolution reconstruction method and system based on a dense Unet diffusion model.

[0007] Technical solution: A zero-shot image super-resolution reconstruction method based on a dense Unet diffusion model, including:

[0008] Build a dense Unet diffusion model;

[0009] Train a dense Unet diffusion model;

[0010] Zero-shot image super-resolution reconstruction method using a trained dense Unet diffusion model;

[0011] Among them, the dense Unet diffusion model is built according to the following steps:

[0012] In the forward diffusion process of the diffusion model without classifier guidance, a dense Unet network structure is introduced to achieve noise prediction;

[0013] The dense Unet network structure is based on the Unet network structure, and two DRB modules are added to each encoder layer and each decoder layer;

[0014] Each of the DRB modules includes a time step encoding layer, a label encoding layer, a dense residual block and a time label direct embedding normalization layer;

[0015] The dense residual block includes four convolutional layers and a parameter-corrected linear unit; each feature map is densely connected;

[0016] The time label direct embedding normalization layer is used to first embed the temporal coding features from the time step coding layer and then sum them with the label coding features from the label coding layer, thereby directly embedding the information into the dense residual block.

[0017] Furthermore, the connection bridge in the dense Unet network structure is composed of two DRB modules and an attention layer.

[0018] Furthermore, the encoder also includes a downsampling layer, which is composed of a convolutional layer and an activation layer; the decoder also includes an upsampling layer, which is composed of a convolutional layer and a transposed convolutional layer; and there is a jump connection between the encoder and the decoder.

[0019] Furthermore, the time tag is directly embedded in the normalization layer and expressed as:

[0020] GNtc(x,Emb t ,Emb c )=(1+Emb t )GroupNorm(x)+Emb c

[0021] Among them, Emb t 、Emb c They represent time series encoding features and label encoding features respectively, and GroupNorm(·) represents group normalization.

[0022] Furthermore, the loss function used in the process of training the dense Unet diffusion model is composed of the weighted sum of the noise prediction loss function, the variational lower bound objective function and the perceptual loss function.

[0023] Furthermore, the noise prediction loss function is expressed as:

[0024]

[0025] In the formula, := indicates that L simple The definition of , E is the mathematical expectation, t represents the time step, T = 1000 is the total time step, x0 ~ q (x0) is the data distribution of the input image, represents random noise, ∈ θ is the noise predicted by the network, x t represents the noise image at time t.

[0026] Furthermore, the variational lower bound objective function is expressed as:

[0027] L vlb :=L0+L1+...L t-1 +...+L T

[0028] L0:=-logp θ (x0|x1)

[0029] L t-1 :=D KL (q(x t-1 |x t ,x0)||p θ (x t-1 |x t ))

[0030] L T :=D KL (q(x T |x0)||p(x T ))

[0031] In the formula, p θ (x0|x1) represents the distribution of x0 when x1 is known, D KL is the KL divergence between two Gaussian distributions, q(x t-1 |x t ,x0) represents the known x0 and the noise image x t The data distribution in the forward denoising process, x0 represents the input image.

[0032] Furthermore, the perceptual loss function is expressed as:

[0033] L P =L perception (x pred ,x0)+L tv

[0034] in:

[0035]

[0036] In the formula, x0 represents the original image, x pred represents the image predicted by the model, C, W and H represent the number of feature channels, the width and height of the feature map respectively, f(x) represents the feature map extracted from x, m and n represent the rows and columns of the image respectively, represents the pixel value of the image at (m,n), and b=2 is the weight parameter of the total variational loss.

[0037] According to the following formula, the original image x0 estimated at the current time step is sampled and used To express:

[0038]

[0039] Where V T By the image degradation operator A = U∑V T After singular value decomposition, we get where α t :=1-β t , β t ∈(0,1) is the preset variance schedule, which determines the amount of noise added at time t, x t is the noise image at time step t, c represents the label, is the noise predicted by the model.

[0040] The present invention also discloses a zero-sample image super-resolution reconstruction system based on a dense Unet diffusion model, including a pre-trained dense Unet diffusion model;

[0041] Among them, the dense Unet diffusion model is constructed by introducing a dense Unet network structure in the forward diffusion process of the diffusion model without classifier guidance, and realizing noise prediction by the dense Unet network structure;

[0042] The dense Unet network structure is based on the Unet network structure, and two DRB modules are added to each encoder layer and each decoder layer;

[0043] Each of the DRB modules includes a time step encoding layer, a label encoding layer, a dense residual block and a time label direct embedding normalization layer;

[0044] The dense residual block includes four convolutional layers and a parameter-corrected linear unit; each feature map is densely connected;

[0045] The time label direct embedding normalization layer is used to first embed the temporal coding features from the time step coding layer and then sum them with the label coding features from the label coding layer, thereby directly embedding the information into the dense residual block.

[0046] Furthermore, the loss function used in the process of training the dense Unet diffusion model is composed of the weighted sum of the noise prediction loss function, the variational lower bound objective function and the perceptual loss function.

[0047] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0048] (1) The gradual deepening of the Unet structure of the diffusion model is prone to cause gradient vanishing or exploding, resulting in partial loss of information and hindering the model's comprehensive learning of data features. To address this problem, the method of the present invention embeds a dense residual block structure in the model architecture, which promotes the direct transmission of gradients through dense connections, effectively reducing the risk of gradient vanishing or exploding; in the dense residual block, each layer is connected to the subsequent layer, so that the network can capture richer and more complex feature patterns, while significantly reducing the parameter scale of the model;

[0049] (2) To address the problem of embedding attribute labels in the model, the present invention adopts a direct time and label embedding standardization method to integrate time and label information in the dense residual block. The model can capture the characteristics of data labels more directly, improving the sensitivity and adaptability of the model to input features.

[0050] (3) In order to improve the perceptual quality of the reconstructed image, the present invention adds a perceptual loss function in the training process. This loss function uses the VGG network to extract the deep features of the image and combines it with the total variational loss to improve the image quality and visual perception while maintaining the image content and structure. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 The dense residual block structure and upsampling structure diagram disclosed in the present invention; Figure 1 (a) in the figure is a dense residual block structure, f is the number of feature maps, Figure 1 (b) in the figure is an upsampling structure;

[0052] Figure 2 This is a dense Unet network structure diagram disclosed in the present invention;

[0053] Figure 3 A flowchart of a zero-shot image super-resolution reconstruction method based on a dense Unet diffusion model proposed by the present invention;

[0054] Figure 4 Comparison chart of reconstruction results. DETAILED DESCRIPTION

[0055] The technical solution of the present invention is now further described in conjunction with the accompanying drawings and embodiments.

[0056] This embodiment discloses a zero-shot image super-resolution reconstruction method based on a dense Unet diffusion model, which reconstructs a high-quality image by iteratively refining the sampling process of the diffusion model. The method mainly includes the following steps:

[0057] Step 1: Use the datasets: Celeba-HQ 256×256 and CelebA. The training set uses Celeba-HQ, which is a high-resolution version of the CelabA dataset. In the test phase, in order to ensure the fairness and accuracy of the evaluation and avoid the model encountering images that are too similar to the training data during the test, this embodiment uses non-repeated images in CelebA and Celeba-HQ to form a test set. The data labels include 40 attributes such as 5_o_Clock_Shadow (beard), Arched_Eyebrows (willow leaf eyebrows) and Attractive (attractive), and the numbers 1 and 0 represent the presence and absence of the attribute in the current image respectively.

[0058] Step 2: Follow the following process to build Figure 3 The dense Unet diffusion model shown:

[0059] In the forward diffusion process of the diffusion model without classifier guidance, the following is introduced: Figure 2 The dense Unet network structure shown in FIG. 1 implements noise prediction by using the dense Unet network structure. The dense Unet network structure is based on the Unet network structure, and two DRB modules are added to each encoder layer and each decoder layer. Figure 2 , all encoding and decoding layers at the same level use jump connections. Each DRB module includes a time step encoding layer, a label encoding layer, a dense residual block, and a time label direct embedding normalization layer. This embodiment replaces the original residual block structure in the Unet model with a DRB module, and optimizes the normalization layer according to a diffusion model without classifier guidance, that is, integrating the time step and label encoding information in a more intuitive way.

[0060] The structure of the dense residual block used in this embodiment is as follows: Figure 1 As shown in (a) of Fig. 1, it includes four convolutional layers and parameter-corrected linear units. In order to make full use of the information of the feature map, each feature map is densely connected. The number of feature maps input to the dense residual block is f, and the number of feature maps output by each convolutional layer is Figure 1 As shown in (a) in the figure, the number of feature maps output by the last convolutional layer is f, in order to achieve local residual weighted connections. Through dense residual learning, the information in the network can be effectively utilized, so that better feature representation can be learned.

[0061] The time tag direct embedding normalization layer used in this embodiment directly embeds the input time series encoding features and label encoding features as core information into the DRB module. This method improves the sensitivity and adaptability of the model to the input data features for the diffusion model guided by the implicit classifier. The time tag direct embedding normalization layer can be expressed by the following formula:

[0062] GNtc(x,Emb t ,Emb c )=(1+Emb t )GroupNorm(x)+Emb c

[0063] Among them, Emb t 、Emb c They represent time series encoding features and label encoding features respectively, and GroupNorm(·) represents group normalization.

[0064] The downsampling layer of the encoder part of the dense Unet network structure adopted in this embodiment consists of a convolution layer and an activation layer, where the size of the convolution kernel is 2×2 and the step size is 2. The feature map is downsampled to twice. After each downsampling layer, the number of feature maps is doubled to compensate for the loss of information.

[0065] The decoder part of the dense Unet network structure used in this embodiment is the inverse process of downsampling, corresponding to the encoder part. The upsampling layer consists of a convolution layer and a transposed convolution layer, such as Figure 1 As shown in (b) of the figure, the encoder and decoder are connected by skip connections to retain important information.

[0066] The bottom layer of the dense Unet network structure used in this embodiment includes two DRB modules and an attention layer.

[0067] In order to improve the visual quality of the image, this embodiment introduces perceptual loss in the diffusion model, which uses the VGG network to extract image depth features, which helps to enhance the clarity of the image and make it more visually realistic.

[0068] The variational lower bound (VLB) is used in the traditional diffusion model to optimize the negative log-likelihood for training:

[0069]

[0070] In the formula, p θ (x0|x1) represents the distribution of x0 when x1 is known, D KL is the KL divergence between two Gaussian distributions, q(x t-1 |x t ,x0) represents the known x0 and the noise image x t The data distribution in the forward denoising process, x0 represents the input image.

[0071] The simplified training objective is:

[0072]

[0073] In the formula, := indicates that L simple The definition of , E is the mathematical expectation, t represents the time step, T = 1000 is the total time step, x0 ~ q (x0) is the data distribution of the input image, represents random noise, ∈ θ is the noise predicted by the network, x t represents the noise image at time t.

[0074] The training objective can be viewed as L vlb A reweighted form of , where the training objective is set to a fixed value: As the number of time steps of the diffusion model increases, although the mean of the model determines the distribution better than the variance, choosing a more appropriate variance can improve the log likelihood to a certain extent, so the variance can be learned through the model. Specifically, the model outputs a vector v, where each component of v corresponds to a dimension of the data, and it is converted into variance by the following formula:

[0075]

[0076] Where β t ∈(0,1) is the preset variance schedule, which determines the amount of noise added at time t. α t :=1-β t The above formula allows the model to learn the dynamic changes of variance and improve the performance of log-likelihood, so the loss of the diffusion model can be expressed as a weighted sum:

[0077] L simple +γL vlb

[0078] Among them, λ = 0.001 is a hyperparameter used to weigh L simple and L vlb The role between.

[0079] To address the perception problem of sampled images, this embodiment introduces perceptual loss into the diffusion model loss function, including total variational loss. Specifically, at each time step of the diffusion model, the predicted image x is calculated. pred The perceptual loss between the original image x0 and the total variational loss is used as the final perceptual objective function L P :

[0080] L P =L perception (x pred ,x0)+L tv

[0081]

[0082] Where C, W, and H represent the number of feature channels, the width and height of the feature map, respectively. f(x) represents the feature map extracted from x. m and n represent the rows and columns of the image, respectively. represents the pixel value of the image at (m,n), and b=2 is the weight parameter of the total variational loss. perception represents the perceptual loss, which uses the intermediate layer features extracted from the pre-trained VGG neural network to calculate the difference between the generated image and the real image. tv It is the total variation loss, which encourages the generated image to have a smoother appearance by measuring the spatial gradient of the image.

[0083] Therefore, the training objective of the diffusion model in this embodiment is:

[0084] L=L simple +γL vlb +L P

[0085] Step 3: The deep learning framework used in this embodiment model is Pytorch, and the training is performed on RTX 3090 24G. The AdamW optimizer is used, with an initial learning rate of 1e-5, a momentum parameter of (0.9, 0.999), and an eps parameter of 1e-8. The input of the model is the images of the training set and the corresponding data labels. The batch size is set to 4, where the attribute labels are expressed in parameters p. uncond = 0.1 probability to set it to To implement the diffusion model strategy guided by implicit classifier.

[0086] Step 4: Use the trained diffusion model to perform super-resolution reconstruction on the test set images.

[0087] like Figure 3 As shown, the image super-resolution reconstruction task can be described as: y = Ax0, y represents the degraded image, is the degradation operator, x0 is the original image, and the degradation operator is subjected to singular value decomposition: A=U∑V T , then the image restored from the degraded image can be expressed as: In the sampling process of the diffusion model, the input of the DR-Unet model includes the random noise image x at time step t=T,...,1,t. t , degraded image y, corresponding label c and an empty set label The output of the DR-Unet model is the predicted noise:

[0088]

[0089] Where ω=1.

[0090] According to the following formula, we can sample the estimated x0 of the current time step and use To express:

[0091]

[0092] Iterate to find the x for the next time step t＝1 Until x0:

[0093]

[0094] Among them, σ t-1 is the noise level at the next time step, σ y is the noise level in the image, s iis a singular value, expressed as s1>s2>…>s M-1 >s M , and define s i =0,i=M+1,...,N,∈ is random noise, and η is set to 0.9.

[0095] This embodiment introduces a dense residual structure and perceptual loss into the diffusion model, enhances the model feature learning capability through the dense structure, improves the quality of image generation, and the perceptual loss has a certain enriching effect on the reconstructed details.

[0096] The method of this embodiment is compared with other image super-resolution reconstruction methods. It can be seen that the accuracy of the PSNR index is significantly improved under 4-fold and 8-fold super-resolution reconstruction. At a super-resolution ratio of 16 times, both PSNR and SSIM achieve excellent results, as shown in the following table:

[0097] Table 1 Reconstruction results of each model

[0098]

Claims

1. A zero-shot image super-resolution reconstruction method based on a dense Unet diffusion model, characterized by: include: Build a dense Unet diffusion model; Train a dense Unet diffusion model; Zero-shot image super-resolution reconstruction method using a trained dense Unet diffusion model; Among them, the dense Unet diffusion model is built according to the following steps: In the forward diffusion process of the diffusion model without classifier guidance, a dense Unet network structure is introduced to achieve noise prediction; The dense Unet network structure is based on the Unet network structure, and two DRB modules are added to each encoder layer and each decoder layer; Each of the DRB modules includes a time step encoding layer, a label encoding layer, a dense residual block and a time label direct embedding normalization layer; The dense residual block includes four convolutional layers and a parameter-corrected linear unit; each feature map is densely connected; The time label direct embedding normalization layer is used to first embed the temporal coding features from the time step coding layer and then sum them with the label coding features from the label coding layer, thereby directly embedding the information into the dense residual block.

2. The method for zero-shot image super-resolution reconstruction based on a dense Unet diffusion model according to claim 1, characterized in that: The connection bridge in the dense Unet network structure consists of two DRB modules and an attention layer.

3. The zero-shot image super-resolution reconstruction method based on the dense Unet diffusion model according to claim 1, characterized in that: The encoder also includes a downsampling layer, which is composed of a convolution layer and an activation layer; the decoder also includes an upsampling layer, which is composed of a convolution layer and a transposed convolution layer; and there is a jump connection between the encoder and the decoder.

4. The method for zero-shot image super-resolution reconstruction based on a dense Unet diffusion model according to claim 1, characterized in that: The time tag is directly embedded in the normalization layer and expressed as: GNtc(x,Emb t ,Emb c )=(1+Emb t )GroupNorm(x)+Emb c Among them, Emb t 、Emb c They represent time series encoding features and label encoding features respectively, and GroupNorm(·) represents group normalization.

5. The zero-shot image super-resolution reconstruction method based on a dense Unet diffusion model according to claim 1, characterized in that: The loss function used in the process of training the dense Unet diffusion model is composed of the weighted sum of the noise prediction loss function, the variational lower bound objective function and the perceptual loss function.

6. The method for zero-shot image super-resolution reconstruction based on a dense Unet diffusion model according to claim 5, characterized in that: The noise prediction loss function is expressed as: In the formula, := indicates that L simple The definition of , E is the mathematical expectation, t represents the time step, T is the total time step, x0~q(x0) is the data distribution of the input image, represents random noise, ∈ θ is the noise predicted by the network, x t represents the noise image at time t.

7. The method for zero-shot image super-resolution reconstruction based on a dense Unet diffusion model according to claim 5, characterized in that: The variational lower bound objective function is expressed as: L vlb :=L0+L1+...L t-1 +...+L T L0:=-logp θ (x0|x1) L t-1 :=D KL (q(x t-1 |x t ,x0)||p θ (x) t-1 |x t )) L T :=D KL (q(x T |x0)||p(x T )) In the formula, p θ (x0|x1) represents the distribution of x0 when x1 is known, D KL is the KL divergence between two Gaussian distributions, q(x t-1 |x t ,x0) represents the known x0 and the noise image x t The data distribution in the forward denoising process, x0 represents the input image.

8. The method for zero-shot image super-resolution reconstruction based on a dense Unet diffusion model according to claim 5, characterized in that: The perceptual loss function is expressed as: L P =L perception (x) pred ,x0)+L tv in: In the formula, x0 represents the original image, x pred represents the image predicted by the model, C, W and H represent the number of feature channels, the width and height of the feature map respectively, f(x) represents the feature map extracted from x, m and n represent the rows and columns of the image respectively, represents the pixel value of the image at (m,n), and b is the weight parameter of the total variational loss; According to the following formula, the original image x0 estimated at the current time step is sampled and used To express: Where V T By the image degradation operator A = U∑V T After singular value decomposition, we get where α t :=1-β t , β t ∈(0,1) is the preset variance schedule, which determines the amount of noise added at time t, x t is the noise image at time step t, c represents the label, is the noise predicted by the model.

9. A zero-shot image super-resolution reconstruction system based on a dense Unet diffusion model, characterized by: Includes pre-trained dense Unet diffusion model; Among them, the dense Unet diffusion model is constructed by introducing a dense Unet network structure in the forward diffusion process of the diffusion model without classifier guidance, and realizing noise prediction by the dense Unet network structure; The dense Unet network structure is based on the Unet network structure, and two DRB modules are added to each encoder layer and each decoder layer; Each of the DRB modules includes a time step encoding layer, a label encoding layer, a dense residual block and a time label direct embedding normalization layer; The dense residual block includes four convolutional layers and a parameter-corrected linear unit; each feature map is densely connected; The time label direct embedding normalization layer is used to first embed the temporal coding features from the time step coding layer and then sum them with the label coding features from the label coding layer, thereby directly embedding the information into the dense residual block.

10. The zero-shot image super-resolution reconstruction system based on the dense Unet diffusion model according to claim 9, characterized in that: The loss function used in the process of training the dense Unet diffusion model is composed of the weighted sum of the noise prediction loss function, the variational lower bound objective function and the perceptual loss function.

Images