Method for enhancing super-resolution of single image through double-domain learning network, electronic equipment and storage medium

Abstract

The invention discloses a method for enhancing the super-resolution of a single image through a double-domain learning network, electronic equipment and a storage medium, solves the problem of insufficient reconstruction details and textures in the prior art, and can integrate frequency domain and spatial domain information to improve the reconstruction quality from a low-resolution image to a high-resolution image. The method comprises the following steps: 1, inputting a low-resolution image into a double-domain learning network comprising a plurality of double-domain learning blocks, and generating two super-resolution image outputs through processing of the plurality of double-domain learning blocks: one is frequency domain output, and the other is spatial domain output; step 2, introducing a complex convolution discriminator and cross-modal distillation loss to enhance the performance of the double-domain learning network, specifically, the complex convolution discriminator is used for processing data for converting an image from an HSV color space into a complex form; the cross-modal distillation loss serves as a loss function, and effective fusion of frequency domain and space domain output is guided in the model training process.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method, electronic device and storage medium for enhancing single image super-resolution through a dual-domain learning network. Background Art

[0002] In the digital age, image super-resolution (SR) technology aims to restore high-resolution (HR) images from low-resolution (LR) images. This technology is crucial for a variety of application scenarios including medical imaging, satellite imaging, and surveillance systems.

[0003] Traditional interpolation methods (such as bicubic interpolation and Lanczos resampling) were once standard super-resolution methods. Usually, the images produced by these methods are too smooth and lack high-frequency details. With the emergence of convolutional neural networks (CNN), people can achieve more complex mappings from LR to HR images. Although convolutional neural networks have made significant progress in the field of image super-resolution, most existing technologies still focus on feature learning in the spatial domain and ignore the details in the frequency domain, which are crucial for reconstructing fine textures and patterns. Therefore, existing technologies are usually unable to fully reconstruct high-frequency details and textures in images. Secondly, existing methods are usually not sufficient in feature fusion and fail to effectively combine information in the spatial and frequency domains. The feature processing of a single domain limits the expressive power of the model, thereby limiting the improvement of the details and quality of super-resolution images. Summary of the invention

[0004] The purpose of the present invention is to provide a method, electronic device and storage medium for enhancing the super-resolution of a single image through a dual-domain learning network, so as to solve the problem of insufficient reconstruction of details and textures in the prior art, and to integrate frequency domain and spatial domain information to improve the reconstruction quality from low-resolution images to high-resolution images, which is suitable for image sharpening processing in various application scenarios such as satellite imaging, medical diagnosis, and security monitoring.

[0005] In order to achieve the above object, one of the solutions of the present invention is: A method for enhancing single image super-resolution via a dual-domain learning network, comprising: Step 1: Input a low-resolution image into a dual-domain learning network containing multiple dual-domain learning blocks, and generate two super-resolution image outputs through processing by multiple dual-domain learning blocks: one is a frequency domain output and the other is a spatial domain output; Each dual-domain learning block contains a frequency feature block and a spatial feature block. The frequency feature block enhances the frequency domain features of the image through fast Fourier transform and complex convolution, and the spatial feature block optimizes the spatial domain features of the image using real convolution. Afterwards, the outputs of the frequency feature block and the spatial feature block are fused through the self-attention feature calibration module, and the self-attention mechanism is used to enhance the correlation and complementarity between features. Step 2: Introduce complex convolution discriminator and cross-modal distillation loss to enhance the performance of the dual-domain learning network. Specifically, the complex convolution discriminator is used to process data that converts images from HSV color space to complex form; cross-modal distillation loss is used as the loss function to guide the effective fusion of frequency domain and spatial domain outputs during model training.

[0006] The step 1 specifically includes: For a given low-resolution image , input the dual-domain learning network, the frequency features and spatial features are extracted by the feature extraction layer of the dual-domain learning network, and input into the dual-domain learning block to achieve dual-domain feature refinement, and finally obtain the frequency output and spatial output ; Among them, the frequency features and spatial features are respectively input into the frequency feature block and the spatial feature block for processing; the structure of the frequency feature block includes a complex convolution, a CReLU activation function and an additional complex convolution. When the frequency features pass through the frequency feature block, the real-valued features are converted into complex-valued features through fast Fourier transform. The real-valued and imaginary-valued components of the complex-valued features interact with the corresponding convolution kernels to generate new complex-valued features, and then these two components are output after CReLU activation; the structure of the spatial feature block includes a real convolution, a ReLU activation function and an additional real convolution.

[0007] Preferably, in step 1, in the self-attention feature calibration module, for a given frequency domain feature and spatial domain characteristics , the self-attention layer is defined as follows: ; in, represents the real number space, , , Respectively represent the height, width and number of channels of the image, represents the convolutional layer, represents the Softmax function, express The transpose of .

[0008] Preferably, in step 2, in the complex convolution discriminator, for a given HSV color Pixels , perform the following conversion: ; ; ; in, , , Represents pixels The three components of , , Represent the hue, saturation, and value of the pixel respectively. The unit representing the imaginary part of a complex number; Thus, the real-valued components of the image are obtained , complex-valued components ,in , , They represent the specific sizes of hue, saturation and value in the real-valued components respectively. , , They represent the specific sizes of hue, saturation and value in the complex value components respectively; The complex-valued feature map is obtained by merging the real-valued component and the complex-valued component. The generated complex-valued feature map is processed by ten discriminant blocks, each of which includes a Complex convolution, followed by a complex batch normalization layer and a CReLU activation function to achieve the reverse conversion from HSV color space to the complex valued domain, and finally a projection layer for discriminative classification of the super-resolution image.

[0009] Preferably, in step 2, a high-resolution target image is defined , frequency output and spatial output , cross-modal distillation loss is defined as: ; in, represents L1 loss, represents the knowledge distillation loss, represents the hyperparameter that controls the balance between pixel-level L1 loss and knowledge distillation loss, Represents the temperature parameter of the distillation process; At the same time, the following loss function is used for supervised dual-domain learning network: ; ; ; in, Represents resistance to loss, represents the output of the discriminator, represents the perceived loss, represents the L1 norm (i.e. L1 loss), represents the image features extracted by a pre-trained network (such as a VGG network), , They represent weight parameters respectively.

[0010] The second solution of the present invention is: An electronic device comprises a processor, a memory and an application program; the application program is stored in the memory and is configured to be executed by the processor to perform the method for enhancing single image super-resolution through a dual-domain learning network.

[0011] The second solution of the present invention is: A computer-readable storage medium stores a computer program; when the computer program is executed in the computer, the computer is caused to execute the method for enhancing single image super-resolution through a dual-domain learning network.

[0012] After adopting the above technical solution, the present invention has the following technical effects: The present invention can achieve dual-domain feature optimization. By simultaneously optimizing frequency domain and spatial domain features, it can capture image details more comprehensively and improve the quality and realism of reconstructed images, which is particularly important for fields that require high-precision image processing (such as medical imaging and security monitoring). The complex convolution discriminator is introduced, and the phase information of the image is maintained by complex value processing, which is crucial for maintaining the structure and texture details of the image. It not only improves the realism of super-resolution images, but also opens up a new path for future research on complex domain image processing inventions. The cross-modal distillation loss is introduced to guide the effective fusion of frequency domain and spatial domain features. By learning the complementarity of the two features, the overall learning ability of the network can be enhanced, which not only improves the quality of super-resolution images, but also enhances the generalization ability of the model for different image types. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 A schematic diagram of a specific embodiment of the present invention; Figure 2 This is a detailed internal architecture diagram of a dual-domain learning module according to a specific embodiment of the present invention; Figure 3 It is a complex-valued convolution discriminator architecture diagram of a specific embodiment of the present invention; Figure 4 A schematic diagram of a model loss design for a specific embodiment of the present invention; Figure 5 A visual comparison diagram of super-resolution results generated by a specific embodiment of the present invention and the prior art; Figure 6 A visual comparison diagram of the cross-modal loss results of different temperature parameters in the specific embodiment of the present invention and the prior art Figure 7 It is a detailed comparison diagram of the model performance of a specific embodiment of the present invention and the prior art. DETAILED DESCRIPTION

[0014] In order to further explain the technical solution of the present invention, the present invention is described in detail below through specific embodiments.

[0015] refer to Figure 1-6 As shown, the present invention discloses a method for enhancing single image super-resolution through a dual-domain learning network, comprising: Step 1: See Figure 1 , a low-resolution image is input into a dual-domain learning network (DDLN) containing multiple dual-domain learning blocks (DDLBs), and two super-resolution image outputs are generated through multiple dual-domain learning blocks (DDLBs): one is the frequency domain output and the other is the spatial domain output. In the dual-domain learning network, this dual-domain parallel processing overcomes the limitation of traditional super-resolution technology that only focuses on one domain. By fusing frequency domain and spatial domain information, high-frequency details and textures are more comprehensively restored, improving the detail richness and realism of the image.

[0016] See also Figure 2 Each dual-domain learning block (DDLB) contains a frequency feature block (FRB) and a spatial feature block (SPB). The frequency feature block enhances the frequency domain features of the image through fast Fourier transform (FFT) and complex convolution, while the spatial feature block optimizes the spatial domain features of the image using real convolution. Afterwards, the outputs of the frequency feature block and the spatial feature block are fused through the self-attention feature calibration (SAFC) module, and the self-attention mechanism is used to enhance the correlation and complementarity between features. This design enables DDLN to not only strengthen the feature expression of each domain separately, but also further improve the overall quality of super-resolution images through feature fusion.

[0017] Step 2: See Figure 3 , introduces complex convolution discriminator (CVCD) and cross-modal distillation loss (CMDL) to enhance the performance of dual-domain learning network, specifically: The complex convolution discriminator is used to process the data that converts the image from the HSV color space to complex form. This innovative processing method not only improves the realism of the super-resolution image, but also preserves the phase information of the image, which is crucial to image quality.

[0018] Cross-modal distillation loss is used as the loss function to guide the effective fusion of frequency domain and spatial domain outputs during model training, ensuring the complementarity of the two features and maximizing the use of information from different domains.

[0019] The design of this dual strategy not only improves the quality of generated images, but also enhances the model's generalization ability for various image types.

[0020] Through the above scheme, the present invention can achieve dual-domain feature optimization. By optimizing the frequency domain and spatial domain features at the same time, it can capture image details more comprehensively and improve the quality and realism of the reconstructed image, which is particularly important for fields that require high-precision image processing (such as medical imaging and security monitoring); the complex convolution discriminator is introduced, and the phase information of the image is maintained by complex value processing, which is crucial for maintaining the structure and texture details of the image. It not only improves the realism of the super-resolution image, but also opens up a new path for future research on complex domain image processing inventions; see Figure 4 , the introduction of cross-modal distillation loss is used to guide the effective fusion of frequency domain and spatial domain features. By learning the complementarity of the two features, the overall learning ability of the network can be enhanced, which not only improves the quality of super-resolution images, but also enhances the generalization ability of the model for different image types.

[0021] Specific embodiments of the present invention are shown below.

[0022] The above step 1 is specifically: For a given low-resolution image , input the dual-domain learning network, the frequency features and spatial features are extracted by the feature extraction layer of the dual-domain learning network, and input into the dual-domain learning block to achieve dual-domain feature refinement, and finally obtain the frequency output and spatial output ; Among them, the frequency features and spatial features are input into the frequency feature block and the spatial feature block for processing respectively; the structure of the frequency feature block includes a complex convolution, a CReLU activation function and an additional complex convolution. When the frequency features pass through the frequency feature block, the real-valued features are converted into complex-valued features through fast Fourier transform (FFT). The real-valued and imaginary-valued components of the complex-valued features interact with the corresponding convolution kernels to generate new complex-valued features, and then these two components are output after CReLU activation; the structure of the spatial feature block includes a real convolution, a ReLU activation function and an additional real convolution. This design ensures the comprehensive enhancement of image features in the frequency domain and the spatial domain.

[0023] In step 1 above, the structure of the self-attention feature calibration (SAFC) module can be found in Figure 2 , for a given frequency domain feature and spatial domain characteristics , the self-attention layer is defined as follows: ; in, represents the real number space, , , Respectively represent the height, width and number of channels of the image, represents the convolutional layer, represents the Softmax function, express The transpose of .

[0024] In step 2 above, the structure of the complex convolution discriminator can be found in Figure 2 , for a given HSV color Pixels , perform the following conversion: ; ; ; in, , , Represents pixels The three components of , , Respectively represent the hue, saturation, and value of the pixel. The unit representing the imaginary part of a complex number; Thus, the real-valued components of the image are obtained , complex-valued components ,in , , They represent the specific sizes of hue, saturation and value in the real-valued components respectively. , , They respectively represent the specific sizes of hue, saturation and value in the complex value component; The complex-valued feature map is obtained by merging the real-valued component and the complex-valued component. The generated complex-valued feature map is processed by ten discriminant blocks (DB), each of which includes a Complex convolution, followed by a complex batch normalization layer and a CReLU activation function to achieve the reverse conversion from HSV color space to the complex valued domain, and finally a projection layer for discriminative classification of the super-resolution image.

[0025] In step 2 above, see Figure 4, the cross-modal distillation loss is crucial to the performance of the calibrated model by enforcing a reciprocal teaching relationship between the outputs in the frequency and spatial domains. The spectral density plot on the right side of the figure illustrates the distribution matching function of the cross-modal distillation loss, which ensures that during the training phase, the two domains not only learn from the high-resolution ground truth, but also learn from each other, resulting in a coordinated enhancement of functionality. Defining a high-resolution target image , frequency output and spatial output , cross-modal distillation loss is defined as: ; in, represents L1 loss, represents the knowledge distillation loss, represents the hyperparameter that controls the balance between pixel-level L1 loss and knowledge distillation loss, Represents the temperature parameter of the distillation process; At the same time, the following loss function is used for supervised dual-domain learning network: ; ; ; in, Represents resistance to loss, represents the output of the discriminator, represents the perceived loss, represents the L1 norm (i.e. L1 loss), represents the image features extracted by a pre-trained network (such as a VGG network), , They represent weight parameters respectively.

[0026] The technical effects of the present invention are described below through some experimental results.

[0027] See also Figure 5 , the figure shows a visual comparison of different methods, which shows the excellent performance of the ESR-DDLN in reconstructing fine details and textures: in the first example, ESR-DDLN shows a significant improvement in the clarity of the pattern in the highlighted area, while the textures produced by other methods (such as RCAN and SwinIR) are relatively soft; in the second example, the ability of ESR-DDLN is further illustrated. Compared with the output results of VDSR and EDSR, the high-frequency area in the output of ESR-DDLN is more clearly reconstructed, with less blur and artifacts. These qualitative results show the advantages of ESR-DDLN in terms of numerical indicators and visual quality.

[0028] See also Figure 6 , shows a visual comparison of the output results of ESR-DDLN using the CMDL loss function under different temperature settings. It can be seen that: the higher the temperature, the worse the fusion effect between the frequency domain and spatial domain features, resulting in smoother output and PSNR values ​​closer to the original spatial domain; on the contrary, the lower the temperature, the higher the degree of fusion of the two outputs. At the optimal temperature, the balance between visual perception quality and PSNR can be achieved most effectively.

[0029] See also Figure 7 , the performance difference between the present invention and the existing methods was quantitatively analyzed, and ESR-DDLN showed a significant advantage on the Urban100 dataset with a scaling factor of x4. The leading model in this category, SwinIR, has a PSNR of 26.80dB and a SSIM of 0.7914. In comparison, ESR-DDLN has a PSNR improvement of 0.20dB and a SSIM improvement of 0.0019, which highlights the superior ability of ESR-DDLN to reconstruct complex texture images at higher magnification levels. For the x2 scale, the PSNR and SSIM of ESR-DDLN on the Set14 dataset are 0.09dB and 0.0011dB higher than those of the RCAN and RCAN models, respectively, indicating that ESR-DDLN has a modest but noteworthy improvement in preserving image details.

[0030] The present invention also discloses an electronic device, comprising a processor, a memory and an application program; the application program is stored in the memory and is configured so that the processor executes the above method.

[0031] The present invention also discloses a computer-readable storage medium, which stores a computer program; when the computer program is executed in a computer, the computer executes the above method. The storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0032] The above embodiments and drawings do not limit the product form and style of the present invention. Any appropriate changes or modifications made by ordinary technicians in the relevant technical field should be deemed to be within the patent scope of the present invention.

Claims

1. A method for enhancing single image super-resolution via a dual-domain learning network, characterized in that include: Step 1: Input a low-resolution image into a dual-domain learning network containing multiple dual-domain learning blocks, and generate two super-resolution image outputs through processing by multiple dual-domain learning blocks: one is a frequency domain output and the other is a spatial domain output; Each dual-domain learning block contains a frequency feature block and a spatial feature block. The frequency feature block enhances the frequency domain features of the image through fast Fourier transform and complex convolution, and the spatial feature block optimizes the spatial domain features of the image using real convolution. Afterwards, the outputs of the frequency feature block and the spatial feature block are fused through the self-attention feature calibration module, and the self-attention mechanism is used to enhance the correlation and complementarity between features. Step 2: Introduce complex convolution discriminator and cross-modal distillation loss to enhance the performance of the dual-domain learning network. Specifically, the complex convolution discriminator is used to process data that converts images from HSV color space to complex form; cross-modal distillation loss is used as the loss function to guide the effective fusion of frequency domain and spatial domain outputs during model training.

2. The method for enhancing single image super-resolution by dual-domain learning network as claimed in claim 1, characterized in that The step 1 specifically includes: For a given low-resolution image , input the dual-domain learning network, the frequency features and spatial features are extracted by the feature extraction layer of the dual-domain learning network, and input into the dual-domain learning block to achieve dual-domain feature refinement, and finally obtain the frequency output and spatial output ; Among them, the frequency features and spatial features are respectively input into the frequency feature block and the spatial feature block for processing; the structure of the frequency feature block includes a complex convolution, a CReLU activation function and an additional complex convolution. When the frequency features pass through the frequency feature block, the real-valued features are converted into complex-valued features through fast Fourier transform. The real-valued and imaginary-valued components of the complex-valued features interact with the corresponding convolution kernels to generate new complex-valued features, and then these two components are output after CReLU activation; the structure of the spatial feature block includes a real convolution, a ReLU activation function and an additional real convolution.

3. The method for enhancing single image super-resolution by dual-domain learning network as claimed in claim 2, characterized in that: In step 1, in the self-attention feature calibration module, for a given frequency domain feature and spatial domain characteristics , the self-attention layer is defined as follows: ； in, represents the real number space, , , Respectively represent the height, width and number of channels of the image, represents the convolutional layer, represents the Softmax function, express The transpose of .

4. The method for enhancing single image super-resolution by dual-domain learning network as claimed in claim 3, characterized in that: In step 2, in the complex convolution discriminator, for a given HSV color Pixels , perform the following conversion: ； ； ； in, , , Represents pixels The three components of , , Represent the hue, saturation, and value of the pixel respectively. The unit representing the imaginary part of a complex number; Thus, the real-valued components of the image are obtained , complex-valued components ,in , , They represent the specific sizes of hue, saturation and value in the real-valued components respectively. , , They represent the specific sizes of hue, saturation and value in the complex value components respectively; The complex-valued feature map is obtained by merging the real-valued component and the complex-valued component. The generated complex-valued feature map is processed by ten discriminant blocks, each of which includes a Complex convolution, followed by a complex batch normalization layer and a CReLU activation function to achieve the reverse conversion from HSV color space to the complex valued domain, and finally a projection layer for discriminative classification of the super-resolution image.

5. The method for enhancing single image super-resolution by dual-domain learning network as claimed in claim 4, characterized in that: In step 2, a high-resolution target image is defined , frequency output and spatial output , cross-modal distillation loss is defined as: ； in, represents L1 loss, represents the knowledge distillation loss, represents the hyperparameter that controls the balance between pixel-level L1 loss and knowledge distillation loss, Represents the temperature parameter of the distillation process; At the same time, the following loss function is used for supervised dual-domain learning network: ； ； ； in, Represents resistance to loss, represents the output of the discriminator, represents the perceived loss, represents the L1 norm, represents the image features extracted by the pre-trained network, , They represent weight parameters respectively.

6. An electronic device, characterized in that The invention comprises a processor, a memory and an application program; the application program is stored in the memory and is configured to be executed by the processor to perform the method for enhancing single image super-resolution by a dual-domain learning network as described in any one of claims 1 to 5.

7. A computer-readable storage medium, characterized in that The storage medium stores a computer program; when the computer program is executed in the computer, the computer executes the method for enhancing single image super-resolution through a dual-domain learning network as described in any one of claims 1 to 5.

Images