Image super-resolution reconstruction method based on high-frequency guidance attention diffusion model

Abstract

The invention discloses an image super-resolution reconstruction method based on a high-frequency guidance attention diffusion model, and the method comprises the steps: collecting image data to form an image data set, and dividing the image data set into a training set and a test set; a Transform is adopted to replace a UNet network in the diffusion model to serve as a backbone network, and the diffusion model with the Transform introduced is obtained; the Transform comprises an encoder and a decoder, the encoder comprises a plurality of encoder layers, and the decoder comprises a plurality of decoder layers; discrete wavelet transform is introduced into each decoder layer, high-frequency features are extracted to serve as query vectors, a double-branch encoder-decoder layer is constructed, feature information of double branches is fused by adopting a multi-head cross attention mechanism in the double-branch encoder-decoder layer, and the feature information of the double branches is fused by adopting a multi-head cross attention mechanism in the double-branch encoder-decoder layer. The high-frequency features extracted by the decoder layer are used for assisting reconstruction of the low-frequency features; a self-adaptive normalization layer is introduced into each double-branch encoder-decoder layer, and affine transformation parameters are adjusted in a self-adaptive mode; and carrying out image super-resolution reconstruction on the image through the trained diffusion model in which the Transform is introduced.

Description

Technical Field

[0001] This invention relates to the field of image reconstruction technology, and in particular to an image super-resolution reconstruction method based on a high-frequency guided attention diffusion model. Background Technology

[0002] Image super-resolution reconstruction is a widely used and long-standing research field. It has evolved from early rule-driven to current data-driven approaches. As early as the late 20th century, researchers proposed methods that relied on rules and prior knowledge to infer details from existing information for image super-resolution reconstruction. Typical methods included bilinear interpolation, bicubic interpolation, sparse representation, and dictionary learning. With the advent of deep learning, image super-resolution reconstruction technology has become primarily based on deep learning, using neural networks to learn from massive amounts of low-resolution and high-resolution images and build models that map details to the data. For example, the SRCNN method uses shallow CNNs to directly map low-resolution images to high-resolution spaces. The FSRCNN method uses low-resolution space processing to achieve real-time super-resolution and lightweight processing, allowing direct input of the original low-resolution image and scaling up the size through deconvolutional layers. The VDSR method uses residual networks, employing very deep convolutional networks to solve the gradient vanishing problem during deep network training, thus improving super-resolution performance. The SRGAN method employs an adversarial training + perceptual loss network structure, introducing a discriminator to guide the generator in producing realistic textures. It optimizes texture realism based on VGG feature space differences, achieving a significant improvement over traditional methods in subjective visual effects for the first time. RCAN uses a channel attention mechanism to dynamically adjust feature channel weights, enhancing important high-frequency information, and employs multi-level residual block nesting to alleviate gradient vanishing. The SwinIR method uses a Transformer as its backbone network structure, combining sliding window attention with local and cross-window interactions for hierarchical feature extraction. The fusion of CNN and Transformer ushered in a new era dominated by Transformer. The SR3 method uses a DDPM diffusion model neural network structure, predicting the output of realistic high-resolution images through forward noise-adding diffusion and backward denoising diffusion processes. However, current image super-resolution reconstruction processes suffer from problems such as loss of detail features, poor image perceptual effects, and severe distortion in the generated super-resolution images. Summary of the Invention

[0003] Purpose of the invention: In order to overcome the shortcomings of the existing technology, the present invention provides an image super-resolution reconstruction method based on a high-frequency guided attention diffusion model. The method uses Transformer to replace the traditional UNet network in the diffusion model. In the dual-branch encoder-decoder structure of Transformer, a multi-head cross-attention mechanism is used to fuse the feature information of the two branches, thereby improving the image perception effect and enhancing the detail features of the image.

[0004] Technical Solution: To achieve the above objectives, this invention provides an image super-resolution reconstruction method based on a high-frequency guided attention diffusion model. The method involves collecting image data to form an image dataset, which is then divided into a training set and a test set. A Transformer is used instead of the UNet network in the diffusion model as the backbone network, resulting in a diffusion model incorporating the Transformer. The Transformer includes an encoder and a decoder, with the encoder containing several encoder layers and the decoder containing several decoder layers. Discrete wavelet transform is introduced in each decoder layer to extract high-frequency features as query vectors. Each encoder layer and its corresponding decoder layer are combined to construct a dual-branch encoder-decoder layer. A multi-head cross-attention mechanism in the dual-branch encoder-decoder layer is used to fuse the feature information from both branches, using the high-frequency features extracted by the decoder layer to assist in the reconstruction of low-frequency features. An adaptive normalization layer is introduced in each dual-branch encoder-decoder layer to adaptively adjust the affine transformation parameters. The diffusion model incorporating the Transformer is trained using the training and test sets, and the inference performance of the model is tested. The trained diffusion model incorporating the Transformer is then used to perform image super-resolution reconstruction.

[0005] Furthermore, the dual-branch encoder-decoder layer includes an encoder layer branch, a decoder layer discrete wavelet transform branch, and a multi-head cross-attention module; the encoder layer branch includes a first input layer, a first adaptive normalization layer, and a multi-head self-attention mechanism layer; the first input layer inputs the processed low-resolution image, the output of the first input layer is input to the first adaptive normalization layer for processing, and the output of the first adaptive normalization layer is input to the multi-head self-attention mechanism layer for processing; the output of the multi-head self-attention mechanism layer is fused with the output of the input layer to obtain a first feature image as the output of the encoder layer branch, and the encoder layer branch outputs the extracted low-frequency features.

[0006] Furthermore, the discrete wavelet transform branch of the decoder layer includes a second input layer, a discrete wavelet transform layer, a fusion layer, and a convolutional layer. The second input layer receives the processed real high-definition image, and the output of the second input layer is input to the discrete wavelet transform layer for processing to obtain high-frequency features in multiple directions. The high-frequency features in multiple directions output by the discrete wavelet transform layer are input to the fusion layer for fusion, and the output of the fusion layer is input to the convolutional layer for convolution. The output of the convolutional layer is used as the output of the discrete wavelet transform branch of the decoder layer, and the discrete wavelet transform branch of the decoder layer outputs the fusion result of the high-frequency features.

[0007] Furthermore, the multi-head cross-attention module includes a second adaptive normalization layer, a multi-head cross-attention mechanism layer, a third adaptive normalization layer, and a point-by-point feedforward network layer. The second adaptive normalization layer takes the output of the encoder layer branch as input, and the output of the second adaptive normalization layer is input to the multi-head cross-attention mechanism layer for processing. The output of the discrete wavelet transform branch of the decoder layer is also input to the multi-head cross-attention mechanism layer. The multi-head cross-attention mechanism layer fuses the output of the second adaptive normalization layer with the output of the discrete wavelet transform branch of the decoder layer for feature information. The output of the multi-head cross-attention mechanism layer is fused with the first feature image output by the encoder layer branch to obtain a second feature image. The second feature image is input to the third adaptive normalization layer for processing, and the output of the third adaptive normalization layer is input to the point-by-point feedforward network layer for processing. The output of the point-by-point feedforward network layer and the second feature image are fused to obtain the reconstructed image output by each encoder-decoder layer.

[0008] Furthermore, in the dual-branch encoder-decoder layer of the Transformer, a multi-head cross-attention mechanism is employed to fuse the dual-branch feature information of the encoder layer branch and the discrete wavelet transform branch of the decoder layer. Both the low-resolution image and the real high-resolution image are used as input, with the real high-resolution image serving as the label to guide the reconstruction of the low-resolution image. The calculation process of the multi-head cross-attention mechanism is shown below:

[0009]

[0010] In the formula, F O Q is the output of the multi-head cross-attention mechanism. h For the query matrix, K and V are the key matrix and value matrix, respectively, T is the transpose of the matrix, and d k is the dimension of the key vector.

[0011] Furthermore, the query matrix Q hIn the reconstruction of low-resolution images using real high-definition images as labels, only high-frequency features are used for guidance. Therefore, in the discrete wavelet transform branch of the decoder layer, the discrete wavelet transform layer performs discrete wavelet transform on the real high-definition image to obtain its high-frequency features. The fusion layer adds the high-frequency features in the horizontal, vertical, and diagonal directions to obtain the output Q of the fusion layer. h The calculation process is as follows:

[0012]

[0013]

[0014] In the formula, i = {H, V, D}, where H represents the horizontal direction, V represents the vertical direction, and D represents the diagonal direction; f(x,y) represents the discrete form of the real high-definition image; M and N are the length and width of the image f(x,y); and ψ(x,y) represents the translation basis function.

[0015] Furthermore, the adaptive normalization operation of the adaptive normalization layer involves introducing an adaptive mechanism to adjust the affine transformation parameters γ and β. For external conditions or context information c, the scaling and offset parameters of the adaptive normalization layer are represented as γ(c) and β(c), and the calculation process is as follows:

[0016]

[0017] Since the scaling parameter γ(c) and the offset parameter β(c) are calculated based on external conditions or context information c, they are implemented through a fully connected layer. The calculation process is as follows:

[0018]

[0019]

[0020] In the formula, fγ and fβ are both parameterized functions, obtained through a learning network.

[0021] Furthermore, the diffusion model incorporating Transformer is trained using a training set. Each dual-branch encoder-decoder layer, during training, includes the following steps:

[0022] Step 1: Obtain the training set from the image dataset. The training set includes low-resolution images and real high-resolution images.

[0023] Step 2: Input both the low-resolution image and the actual high-resolution image into the dual-branch encoder-decoder layer;

[0024] Step 3: The encoder layer branch processes the low-resolution image sequentially through the first input layer, the first adaptive normalization layer, and the multi-head self-attention mechanism layer to extract the low-frequency feature information of the low-resolution image.

[0025] Step 4: The discrete wavelet transform branch of the decoder layer processes the real high-definition image sequentially through the second input layer, discrete wavelet transform layer, fusion layer and convolutional layer to extract the high-frequency feature information of the real high-definition image;

[0026] Step 5: The multi-head cross-attention module fuses the low-frequency feature information of the low-resolution image with the high-frequency feature information of the real high-resolution image, and uses the high-frequency feature information of the real high-resolution image to guide the reconstruction of the low-resolution image.

[0027] Step 6: Output the reconstructed image of this dual-branch encoder-decoder layer.

[0028] Beneficial Effects: This invention provides an image super-resolution reconstruction method based on a high-frequency guided attention diffusion model. It replaces the traditional UNet network in the diffusion model with a Transformer. The dual-branch structure in the Transformer fuses high-frequency feature information from the discrete wavelet transform branch of the decoder layer with feature information from the low-resolution image through a multi-head cross-attention mechanism, achieving high-frequency feature-guided low-resolution image reconstruction. By guiding the reconstruction of the low-resolution image with high-frequency feature information, a super-resolution image is generated, improving image perception and enhancing image detail. Simultaneously, adaptive batch normalization is employed to accelerate model convergence and enhance numerical stability. Attached Figure Description

[0029] Figure 1 The flowchart shows an image super-resolution reconstruction method based on a high-frequency guided attention diffusion model.

[0030] Figure 2 This is a demonstration diagram of the dual-branch fusion model in the dual-branch encoder-decoder layer;

[0031] Figure 3 This is a structural diagram of a dual-branch encoder-decoder layer;

[0032] Figure 4 This is a comparison chart showing the effects of different methods and the present invention on the CelebA dataset. Detailed Implementation

[0033] The invention will now be further described with reference to the accompanying drawings.

[0034] like Figure 1As shown, an image super-resolution reconstruction method based on a high-frequency guided attention diffusion model is presented. Image data is collected to form an image dataset, which is then divided into a training set and a test set. In the diffusion model, the traditional approach uses UNet as the neural network. However, UNet performs worse than Transformer in terms of feature global dependency and model expansion performance. Therefore, Transformer is used instead of UNet as the backbone network in the diffusion model, resulting in a diffusion model incorporating Transformer. The Transformer includes an encoder and a decoder. The encoder contains several encoder layers, and the decoder contains several decoder layers. Each decoder layer introduces... Discrete wavelet transform is used to extract high-frequency features as query vectors. Each encoder layer and its corresponding decoder layer are combined to construct a dual-branch encoder-decoder layer. A multi-head cross-attention mechanism in the dual-branch encoder-decoder layer is used to fuse the feature information from both branches. The high-frequency features extracted by the decoder assist in the reconstruction of low-frequency features, thus using high-frequency feature information to assist in the reconstruction of low-resolution images. An adaptive normalization layer is introduced in each dual-branch encoder-decoder layer to adaptively adjust the affine transformation parameters. The diffusion model incorporating Transformer is trained using training and testing sets, and the inference performance of the detection model is evaluated. The trained diffusion model incorporating Transformer is then used to perform super-resolution image reconstruction. The diffusion model incorporating Transformer is the high-frequency guided attention diffusion model. Transformer is a deep learning architecture based on a self-attention mechanism, primarily used in Natural Language Processing (NLP) and Computer Vision (CV) fields, capable of efficiently processing sequential data and capturing long-range dependencies.

[0035] The acquired image dataset is subjected to a forward diffusion process in a diffusion model on all image data within the dataset. This forward diffusion process results in image data that has undergone image diffusion, which is then divided into a training set and a test set. Therefore, both the training and test sets are image data generated through the forward diffusion process of the diffusion model. Consequently, both the high-definition realistic images and low-resolution images input to the dual-branch encoder-decoder layer are generated by the forward diffusion process of the diffusion model. Since a Transformer is used instead of the UNet network in the diffusion model as the backbone network, and the UNet network is located in the reverse diffusion process of the diffusion model, the Transformer, which replaces the UNet network, is also located in the reverse diffusion process of the diffusion model. This means that the reverse diffusion process is performed using several dual-branch encoder-decoder layers within the Transformer.

[0036] like Figure 2The diagram illustrates a dual-branch fusion model in a dual-branch encoder-decoder layer. It employs an encoder-decoder structure, with one branch acting as the encoder and the other as the decoder. First, shallow features are extracted from the low-resolution image. These shallow features, i.e., latent space features, are split into patch sequences using an overlapping block operation. These patches serve as the processed low-resolution image and are input into the Transformer module. Next, the high-resolution real image is temporally encoded so that the temporal encoding corresponds to the features of the low-resolution image. Similarly, the features of the high-resolution real image are split into patch sequences, which are then used as the processed real high-resolution image and input into the Transformer module. The Transformer module reconstructs the low-resolution image and the processed real high-resolution image to obtain a reconstructed image. The reconstructed image output by the Transformer module undergoes adaptive normalization and fully connected layer operations sequentially to finally obtain the reconstructed super-resolution image. The Transformer module consists of several stacked dual-branch encoder-decoder layers.

[0037] like Figure 3 As shown, the dual-branch encoder-decoder layer includes an encoder layer branch, a decoder layer discrete wavelet transform branch, and a multi-head cross-attention module. The encoder layer branch includes a first input layer, a first adaptive normalization layer, and a multi-head self-attention mechanism layer. The first input layer receives the processed low-resolution image as input, and the output of the first input layer is fed into the first adaptive normalization layer for further processing. The output of the first adaptive normalization layer is then fed into the multi-head self-attention mechanism layer for further processing. The output of the multi-head self-attention mechanism layer is fused with the output of the input layer to obtain a first feature image, which is then used as the output of the encoder layer branch. The encoder layer branch outputs the extracted low-frequency features. The first input layer assigns input tokens to the input low-resolution image.

[0038] The decoder layer's discrete wavelet transform branch includes a second input layer, a discrete wavelet transform layer, a fusion layer, and a convolutional layer. The second input layer takes the processed real high-definition image as input, and its output is fed into the discrete wavelet transform layer for further processing to obtain high-frequency features in multiple directions. These high-frequency features from the discrete wavelet transform layer are then fed into the fusion layer for fusion. The output of the fusion layer is fed into the convolutional layer for convolution, and the output of the convolutional layer serves as the output of the decoder layer's discrete wavelet transform branch. The decoder layer's discrete wavelet transform branch outputs the fused high-frequency features. The second input layer performs conditional processing on the real high-definition image; the high-frequency feature information extracted by the decoder layer's discrete wavelet transform branch is the high-frequency feature extracted by the decoder layer.

[0039] The multi-head cross-attention module includes a second adaptive normalization layer, a multi-head cross-attention mechanism layer, a third adaptive normalization layer, and a point-by-point feedforward network layer. The second adaptive normalization layer receives the output of the encoder layer branch, and its output is processed by the multi-head cross-attention mechanism layer. The output of the decoder layer's discrete wavelet transform branch is also input to the multi-head cross-attention mechanism layer. The multi-head cross-attention mechanism layer fuses the output of the second adaptive normalization layer with the output of the decoder layer's discrete wavelet transform branch. The output of the multi-head cross-attention mechanism layer is then fused with the first feature image output from the encoder layer branch to obtain a second feature image. This second feature image is processed by the third adaptive normalization layer, and its output is processed by the point-by-point feedforward network layer. The output of the point-by-point feedforward network layer is then fused with the second feature image to obtain the reconstructed image output by each encoder-decoder layer. By fusing high-frequency feature information from the decoder layer's discrete wavelet transform branch with the feature information of the low-resolution image through the multi-head cross-attention mechanism, high-frequency features guide low-frequency feature reconstruction, thus achieving low-resolution image reconstruction.

[0040] The adaptive normalization layer AdaLN introduces an adaptive mechanism on top of the traditional Layer Normalization (LN) layer, enhancing the model's adaptability to different input conditions by dynamically adjusting the normalization parameters. The multi-head self-attention mechanism layer captures complex dependencies between different positions within a sequence through parallelization. The multi-head cross-attention mechanism layer is used to achieve information interaction and fusion between different sequences. The pointwise feedforward network layer processes the embedding at each position independently through nonlinear transformation, enhancing the model's expressive power.

[0041] In the dual-branch encoder-decoder layer of the Transformer, a multi-head cross-attention mechanism is used to fuse the dual-branch feature information of the encoder layer branch and the discrete wavelet transform branch of the decoder layer. Both low-resolution images and real high-resolution images are used as inputs, with the real high-resolution images serving as labels to guide the reconstruction of the low-resolution images. The calculation process of the multi-head cross-attention mechanism is shown below:

[0042]

[0043] In the formula, F OThe output of the multi-head cross-attention mechanism is represented by Q, the query matrix, which is typically derived from the decoder output and composed of several query vectors; K and V are the key and value matrices, respectively, usually derived from the encoder output; T is the transpose of the matrix; and dk is the dimension of the key vectors, serving a scaling function to prevent excessive parameter changes that could affect model training. The query matrix Q... h The discrete wavelet transform obtained through the discrete wavelet transform branch of the decoder layer is as follows:

[0044]

[0045] In the formula, I HR The input is a real high-definition image, and DWT is the operation of the discrete wavelet transform branch of the decoder layer.

[0046] The query matrix Q h In the reconstruction of low-resolution images using real high-definition images as labels, only high-frequency features are used for guidance. Therefore, in the discrete wavelet transform branch of the decoder layer, the discrete wavelet transform layer performs discrete wavelet transform on the real high-definition image to obtain its high-frequency features. The fusion layer adds the high-frequency features in the horizontal, vertical, and diagonal directions to obtain the output Q of the fusion layer. h Specifically, the high-frequency features in the horizontal, vertical, and diagonal directions are added and fused in the fusion layer to obtain Q. h The calculation process is as follows:

[0047]

[0048]

[0049] In the formula, i = {H, V, D}, where H represents the horizontal direction, V represents the vertical direction, and D represents the diagonal direction; f(x,y) represents the discrete form of the real high-definition image; M and N are the length and width of the image f(x,y); ψ(x,y) represents the translation basis function; Wi represents the high-frequency features obtained from the discrete wavelet transform layer in the horizontal, vertical, or diagonal directions; m and n represent the corresponding frequency domain image coordinates; x and y represent the initial image coordinates; where Q h This is the result obtained by adding the high-frequency features together.

[0050] The adaptive batch normalization operation is applied to the features of each layer of the model to accelerate model convergence and enhance numerical stability. The adaptive normalization operation of the adaptive normalization layer introduces an adaptive mechanism to adjust the affine transformation parameters γ and β. For external conditions or context information c, the scaling and offset parameters of the adaptive normalization layer are represented as γ(c) and β(c), and the calculation process is as follows:

[0051]

[0052] In the formula, x represents the input, y represents the output, μ represents the mean, and σ represents the standard deviation;

[0053] Since the scaling parameter γ(c) and the offset parameter β(c) are calculated based on external conditions or context information c, they are implemented through a fully connected layer. The calculation process is as follows:

[0054]

[0055]

[0056] In the formula, fγ and fβ are both parameterized functions, obtained through a learning network; and c is a part of the input data.

[0057] The diffusion model incorporating Transformer is trained using a training set. Each dual-branch encoder-decoder layer, during training, includes the following steps:

[0058] Step 1: Obtain the training set from the image dataset. The training set includes low-resolution images and real high-resolution images.

[0059] Step 2: Input both the low-resolution image and the real high-resolution image into the dual-branch encoder-decoder layer; the high-resolution real image and the low-resolution image input into the dual-branch encoder-decoder layer are both generated by the forward diffusion process of the diffusion model;

[0060] Step 3: The encoder layer branch processes the low-resolution image sequentially through the first input layer, the first adaptive normalization layer, and the multi-head self-attention mechanism layer to extract the low-frequency feature information of the low-resolution image.

[0061] Step 4: The discrete wavelet transform branch of the decoder layer processes the real high-definition image sequentially through the second input layer, discrete wavelet transform layer, fusion layer and convolutional layer to extract the high-frequency feature information of the real high-definition image;

[0062] Step 5: The multi-head cross-attention module fuses the low-frequency feature information of the low-resolution image with the high-frequency feature information of the real high-resolution image, and uses the high-frequency feature information of the real high-resolution image to guide the reconstruction of the low-resolution image.

[0063] Step 6: Output the reconstructed image of this dual-branch encoder-decoder layer.

[0064] In the diffusion model incorporating Transformer, the output query matrix Q from the discrete wavelet transform branch of the decoder layer is used. hThe high-frequency features are used to reconstruct the low-frequency features, along with the output key matrix K and value matrix V from the encoder layer branches. This high-frequency feature information is then used to reconstruct the low-resolution image, resulting in a super-resolution image. After training the Transformer-introduced diffusion model on the training set, the trained model can be used for super-resolution image reconstruction. Alternatively, the low-resolution image can be input into the trained Transformer-introduced diffusion model for processing to obtain the reconstructed super-resolution image.

[0065] Example

[0066] The image datasets used are CelebA and FFHQ. To evaluate model performance, three metrics were selected: Peak Signal-to-Noise Ratio (PSNR), a measure of the ratio of the maximum possible power to the noise power between two images; a higher PSNR indicates better model performance. Structural Similarity (SSIM) considers the similarity between the generated and original images based on three dimensions: brightness, contrast, and structural variation; a higher SSIM indicates a superior model. Fréchet Inception Distance (FID) measures the difference between the generated and real image distributions; a lower FID indicates that the generated image is closer to the real image distribution.

[0067] like Figure 4 As shown, the graphs compare the performance of different methods and the method of this invention on the CelebA dataset. The different methods are the Bicubic algorithm, the SR3 algorithm, the SRDiff algorithm, and the ResDiff algorithm. Figure 4 The comparison involves the Bicubic algorithm, SR3 algorithm, SRDiff algorithm, and ResDiff algorithm, as well as the image processed by this invention on the CelebA dataset; the Reference is a real high-resolution image. Through qualitative comparative analysis, it can be found that, compared with other methods, although the comparison methods have successfully learned satisfactory boundary information, significant color differences still exist. This invention, however, achieves satisfactory results in both boundary information fidelity and color reproduction, thus making its method superior to other algorithms.

[0068] To further verify the model's performance, this invention was compared quantitatively with SRGAN, ESRGAN, SRDiff, SR3, and ResDiff algorithms on the CelebA dataset, and the results were compared using PSNR, SSIM, and FID evaluation metrics. Table 1 below shows the quantitative comparison of this invention with other algorithms on the CelebA dataset.

[0069] Table 1. Quantization Comparison of the Invention with Other Algorithms on the CelebA Dataset

[0070]

[0071] As shown in Table 1, Ground Truth is the gold standard, which represents the optimal values ​​of PSNR, SSIM, and FID under ideal conditions. It can be seen that when reconstructing images from 32 to 128 and from 256 to 1024 on the CelebA dataset, this invention demonstrates better performance in terms of PSNR, SSIM, and FID compared to other algorithms.

[0072] To further verify the model's performance, this invention was compared with the SRDiff, SR3, and ResDiff algorithms on the FFHQ dataset using quantitative comparisons, and the results were compared in terms of PSNR, SSIM, and FID evaluation metrics. Table 2 below shows the quantitative comparison of this invention with other algorithms on the FFHQ dataset.

[0073] Table 2. Quantization Comparison of the Invention with Other Algorithms on the FFHQ Dataset

[0074]

[0075] As shown in Table 2, Ground Truth is the gold standard, which represents the optimal values ​​of PSNR, SSIM, and FID under ideal conditions. It can be seen that when reconstructing images from 20 to 160 and from 64 to 256 on the FFHQ dataset, this invention demonstrates better performance in terms of PSNR, SSIM, and FID compared to other algorithms.

[0076] The above description is merely a preferred embodiment of the present invention. Those skilled in the art can make several modifications and optimizations based on the above disclosure without departing from the basic principles described above. These modifications and optimizations should be considered within the scope of protection as understood by the present invention.

Claims

1. An image super-resolution reconstruction method based on a high-frequency guided attention diffusion model, characterized in that: Image data is collected to form an image data set, and the image data set is divided into a training set and a test set; a Transformer is used to replace a UNet network in a diffusion model as a backbone network to obtain a diffusion model with the Transformer; the Transformer includes an encoder and a decoder, the encoder includes a plurality of encoder layers, and the decoder includes a plurality of decoder layers; a discrete wavelet transform is introduced into each decoder layer to extract high-frequency features as query vectors; each encoder layer and the corresponding decoder layer are combined to construct a double-branch encoder-decoder layer, and a multi-head cross attention mechanism in the double-branch encoder-decoder layer is used to fuse feature information of the double branches, so that the high-frequency features extracted by the decoder layer assist the reconstruction of low-frequency features; An adaptive normalization layer is introduced into each double-branch encoder-decoder layer to adaptively adjust affine transformation parameters; the training set and the test set are used to train the diffusion model with the Transformer and to detect the inference effect of the trained model; and the diffusion model with the Transformer after training is used to perform image super-resolution reconstruction on images.

2. The image super-resolution reconstruction method based on the high-frequency attention guidance diffusion model according to claim 1, characterized in that: The double-branch encoder-decoder layer includes an encoder layer branch, a decoder layer discrete wavelet transform branch, and a multi-head cross attention module; The encoder layer branch includes a first input layer, a first adaptive normalization layer, and a multi-head self-attention mechanism layer; the first input layer inputs a processed low-resolution image, the output of the first input layer is input to the first adaptive normalization layer for processing, and the output of the first adaptive normalization layer is input to the multi-head self-attention mechanism layer for processing; the output of the multi-head self-attention mechanism layer is fused with the output of the input layer to obtain a first feature image as the output of the encoder layer branch, and the low-frequency features extracted by the encoder layer branch are output.

3. The image super-resolution reconstruction method based on the high-frequency attention diffusion model according to claim 2, characterized in that: The decoder layer discrete wavelet transform branch includes a second input layer, a discrete wavelet transform layer, a fusion layer, and a convolution layer; the second input layer inputs a processed real high-definition image, the output of the second input layer is input to the discrete wavelet transform layer for processing to obtain high-frequency features in multiple directions; The high-frequency features in multiple directions output by the discrete wavelet transform layer are input to the fusion layer for fusion, the output of the fusion layer is input to the convolution layer for convolution, and the output of the convolution layer is the output of the decoder layer discrete wavelet transform branch, and the fusion result of the high-frequency features output by the decoder layer discrete wavelet transform branch.

4. The image super-resolution reconstruction method based on the high-frequency attention diffusion model according to claim 3, characterized in that: The multi-head cross attention module comprises a second adaptive normalization layer, a multi-head cross attention mechanism layer, a third adaptive normalization layer and a point-by-point feedforward network layer; the second adaptive normalization layer inputs the output of the encoder layer branch, the output of the second adaptive normalization layer is input to the multi-head cross attention mechanism layer for processing, and the output of the decoder layer discrete wavelet transform branch is input to the multi-head cross attention mechanism layer; the multi-head cross attention mechanism layer fuses the output of the second adaptive normalization layer and the output of the decoder layer discrete wavelet transform branch in feature information; the output of the multi-head cross attention mechanism layer is fused with the first feature image output by the encoder layer branch to obtain a second feature image, the second feature image is input to the third adaptive normalization layer for processing, the output of the third adaptive normalization layer is input to the point-by-point feedforward network layer for processing, and the output of the point-by-point feedforward network layer is fused with the second feature image to obtain a reconstructed image output by each encoder-decoder layer.

5. The image super-resolution reconstruction method based on the high-frequency attention diffusion model according to claim 4, characterized in that: In the double-branch encoder-decoder layer of the Transformer, the multi-head cross attention mechanism is used to fuse the double-branch feature information of the encoder layer branch and the decoder layer discrete wavelet transform branch, the low-resolution image and the real high-definition image are both used as input, and the real high-definition image is used as a label to guide the reconstruction of the low-resolution image; the calculation process of the multi-head cross attention mechanism is as follows: In the formula, F O is the output of the multi-head cross-attention mechanism, Q h is the query matrix, K and V are the key matrix and the value matrix respectively, T is the transpose of the matrix, d k is the dimension of the key vector.

6. The image super-resolution reconstruction method based on the high-frequency attention guidance diffusion model according to claim 5, characterized in that: The query matrix Q h In the reconstruction of the low-resolution image guided by the real high-definition image as the label, only high-frequency features are used for guidance; therefore, in the discrete wavelet transform branch of the decoder layer, the discrete wavelet transform layer performs discrete wavelet transform on the real high-definition image to obtain high-frequency features of the real high-definition image. The fusion layer adds high frequency features in horizontal, vertical and diagonal directions to obtain the output Q of the fusion layer h The calculation process is as follows: In the formula, i={H, V, D}, H represents the horizontal direction, V represents the vertical direction, and D represents the diagonal direction; f(x, y) represents the discrete form of the real high-definition image; M and N are the length and width of the image f(x, y); and represents the translation basis function.

7. The image super-resolution reconstruction method based on the high-frequency attention diffusion model according to claim 4, characterized in that: The adaptive normalization operation of the adaptive normalization layer is to introduce an adaptive mechanism to adjust the affine transformation parameters γ and β; for external conditions or context information c, the scaling parameter and the offset parameter of the adaptive normalization layer are represented as γ(c) and β(c), and the calculation process is as follows: Since the scaling parameter γ(c) and the offset parameter β(c) are calculated according to the external conditions or the context information c, the scaling parameter γ(c) and the offset parameter β(c) are realized through a fully connected layer, and the calculation process is as follows: In the formula, fγ and fβ are both parameterized functions, which are obtained through a learning network.

8. The image super-resolution reconstruction method based on the high-frequency attention diffusion model according to claim 4, characterized in that: The diffusion model introducing the Transformer is trained using a training set, and each double-branch encoder-decoder layer includes the following steps in the training process: Step one, obtaining a training set in an image data set, the training set comprising a low-resolution image and a real high-definition image; Step two, inputting the low-resolution image and the real high-definition image into the double-branch encoder-decoder layer; Step three, the encoder layer branch processes the low-resolution image through a first input layer, a first adaptive normalization layer and a multi-head self-attention mechanism layer in sequence to extract low-frequency feature information of the low-resolution image; Step four, the decoder layer discrete wavelet transform branch sequentially passes through the second input layer, the discrete wavelet transform layer, the fusion layer and the convolution layer processing to the real high-definition image, and the high-frequency feature information of the real high-definition image is extracted; Step five, the multi-head cross attention module fuses the low-frequency feature information of the low-resolution image and the high-frequency feature information of the real high-definition image, and uses the high-frequency feature information of the real high-definition image to guide the reconstruction of the low-resolution image; Step six, the double-branch encoder-decoder layer outputs the reconstructed image of the double-branch encoder-decoder layer.

Images