Single-image super-resolution method and device combining feature coding and diffusion model

Abstract

The invention provides a single-image super-resolution method and device combining feature coding and a diffusion model, and relates to the technical field of computer images, and the method comprises the steps: carrying out the up-sampling of a low-resolution image, and obtaining a first super-resolution image used for representing the low-frequency details of the low-resolution image; a second super-resolution image used for representing high-frequency details of the low-resolution image is obtained through a diffusion model, and the denoising process of the diffusion model is guided by key features extracted from the low-resolution image; a super-resolution image of the low-resolution image is generated in combination with the first super-resolution image and the second super-resolution image. According to the method, the random iteration mechanism of diffusion model denoising is combined with the feature coding technology, the key features extracted from the LR image are utilized to guide the denoising process of the diffusion model, and the efficiency and quality of the SISR high-frequency detail recovery process are improved.

Description

Technical Field

[0001] The present invention relates to the field of computer image technology, and in particular to a single image super-resolution method and device combining feature coding and diffusion model. Background Art

[0002] Single Image Super Resolution (SISR) is a difficult task in computer vision, which aims to restore high-resolution images from low-resolution images. It is widely used in medical imaging, satellite imaging, surveillance systems, etc. During the image degradation process, high-frequency components are lost, and a single low-resolution image often corresponds to multiple unknown high-resolution images, making the task ill-posed.

[0003] With the development of deep learning, convolutional neural networks have significantly improved the effect of single image super-resolution. Early SR (Super Resolution) models were mainly oriented towards MSE (Mean Squared Error) / PSNR (Peak Signal-to-Noise Ratio), but the generated results were not visually realistic and lacked high-frequency details.

[0004] After the Generative Adversarial Network (GAN) was proposed, the main SISR methods based on generative models are all driven by GAN. However, GAN-based methods are difficult to train and are prone to mode collapse, resulting in a lack of diversity. The diffusion module is trained by optimizing a variant of the variational lower bound, avoiding mode collapse and generating different HR (High Resolution) images for the same LR input, showing good prospects in SISR tasks.

[0005] However, the high computational burden of diffusion limits their applications. Therefore, existing single image super-resolution methods still find it difficult to balance the performance and efficiency of image reconstruction. Summary of the invention

[0006] The present invention provides a single image super-resolution method and device combining feature coding and diffusion model, so as to solve the defect that SISR in the prior art is difficult to better balance the performance and efficiency of image reconstruction, and realize a single image super-resolution method that takes both performance and efficiency into consideration.

[0007] The present invention provides a single image super-resolution method combining feature coding and diffusion model, comprising: Upsampling the low-resolution image to obtain a first super-resolution image for characterizing low-frequency details of the low-resolution image; Acquire a second super-resolution image for characterizing high-frequency details of the low-resolution image through a diffusion model, wherein a denoising process of the diffusion model is guided by key features extracted from the low-resolution image; The first super-resolution image and the second super-resolution image are combined to generate a super-resolution image of the low-resolution image.

[0008] According to a single image super-resolution method combining feature encoding and diffusion model provided by the present invention, the step of obtaining a second super-resolution image for characterizing high-frequency details of the low-resolution image through the diffusion model specifically includes: Performing feature extraction on the low-resolution image to obtain key features of the low-resolution image; In the denoising process of the diffusion model, the key features are connected to the initial downsampling layer of the decoding network to guide iterative denoising; The second super-resolution image output after denoising by the diffusion model is obtained.

[0009] According to a single image super-resolution method combining feature encoding and diffusion model provided by the present invention, the step of connecting the key features to the initial downsampling layer of the decoding network for guiding iterative denoising specifically includes: Performing dimensionality reduction on the extracted key features based on distribution-aware feature subset fusion; The reduced key features are connected to the initial downsampling layer of the decoding network.

[0010] According to a single image super-resolution method combining feature encoding and diffusion model provided by the present invention, the step of extracting features from the low-resolution image to obtain key features of the low-resolution image specifically includes: Using a pre-trained feature encoding module to extract features from the low-resolution image, to obtain key features output by the feature encoding module; The feature encoding module is composed of a plurality of continuously connected dense blocks with residual structures, and each of the dense blocks is provided with an attention module for achieving stable training.

[0011] According to a single image super-resolution method combining feature encoding and diffusion model provided by the present invention, the attention module is defined as: ; In the formula, x represents the input of the attention module, W is the weight matrix,avgpool represents global average pooling; represents element-wise multiplication; ; In the formula, c is the number of feature channels, A feature vector corresponding to the first half of the input channels of the attention module; A feature vector corresponding to the second half of the input channel of the attention module; is the magnification factor; Represents the feature vector obtained by uniformly sampling the input of the attention module.

[0012] According to a single image super-resolution method combining feature encoding and diffusion model provided by the present invention, the step of obtaining a second super-resolution image for characterizing high-frequency details of the low-resolution image through the diffusion model specifically includes: During the diffusion process, the diffusion model gradually introduces noise into the residual image to obtain a normally distributed noise image, wherein the residual image represents the difference between the high-resolution image and the low-resolution image.

[0013] The present invention also provides a single image super-resolution device combining feature coding and diffusion model, comprising: A low-frequency image acquisition module, used for upsampling the low-resolution image to obtain a first super-resolution image for characterizing low-frequency details of the low-resolution image; A high-frequency image acquisition module, configured to acquire a second super-resolution image for characterizing high-frequency details of the low-resolution image through a diffusion model, wherein a denoising process of the diffusion model is guided by key features extracted from the low-resolution image; An image reconstruction module is used to generate a super-resolution image of the low-resolution image by combining the first super-resolution image and the second super-resolution image.

[0014] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a single image super-resolution method combining feature coding and diffusion model as described above is implemented.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a single image super-resolution method combining feature encoding and a diffusion model as described in any one of the above.

[0016] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the single image super-resolution method combining feature encoding and diffusion model as described above is implemented.

[0017] The single image super-resolution method and device combining feature coding and diffusion model provided by the present invention combine the random iteration mechanism of diffusion model denoising with feature coding technology, use the key features extracted from LR images to guide the denoising process of the diffusion model, thereby improving the efficiency and quality of the high-frequency detail recovery process of SISR to achieve high-quality SR reconstruction, reducing the computational burden of the SISR model, and reducing the size of the SISR model. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 It is a flow chart of a single image super-resolution method combining feature coding and diffusion model provided by the present invention; Figure 2 It is an architecture diagram of SRCAD in a single image super-resolution method combining feature coding and diffusion model provided by the present invention; Figure 3 It is a structural schematic diagram of a feature encoding module in a single image super-resolution method combining feature encoding and a diffusion model provided by the present invention; Figure 4 It is a schematic diagram of a dimensional staggered product structure in a single image super-resolution method combining feature coding and a diffusion model provided by the present invention; Figure 5 It is a schematic diagram for comparing reconstruction results in the case of 4 times super resolution in the single image super resolution method combining feature coding and diffusion model provided by the present invention; Figure 6 It is a schematic diagram for comparing reconstruction results in the case of 8 times super resolution in a single image super resolution method combining feature coding and diffusion model provided by the present invention; Figure 7 It is a structural schematic diagram of a single image super-resolution device of feature coding and diffusion model provided by the present invention; Figure 8 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] Combine the following Figures 1 to 6 The present invention introduces a single image super-resolution method combining feature encoding and diffusion model, such as Figure 1 As shown, including: Step 101, upsampling the low-resolution image to obtain a first super-resolution image for representing low-frequency details of the low-resolution image; Optionally, in this embodiment, a pre-built SRCAD (Single Image Super-Resolution with Channel Attention and Diffusion) framework is used to implement a single image super-resolution method combining feature encoding and diffusion model. In other embodiments, other feasible frameworks can also be used to implement the single image super-resolution method provided in this application.

[0022] like Figure 2 As shown in FIG. 1 , the SRCAD framework consists of two branches. The lower branch of SRCAD upsamples the low-resolution image (LowResolution, LR) by a multiple of 4 or 8 to obtain a first super-resolution image with low-frequency blur, that is, the low-frequency details of the low-resolution image to be reconstructed are represented by the first super-resolution image.

[0023] Step 102, obtaining a second super-resolution image for characterizing high-frequency details of the low-resolution image through a diffusion model, wherein a denoising process of the diffusion model is guided by key features extracted from the low-resolution image; The upper branch of SRCAD is used to obtain a second super-resolution image by combining the diffusion model and the key features extracted. The second super-resolution image is used to characterize the high-frequency details of the low-resolution image to be reconstructed.

[0024] like Figure 2 As shown in Figure 2, the application of diffusion model in single image super-resolution usually includes two processes: forward diffusion and backward denoising.

[0025] The forward diffusion process is to gradually add Gaussian noise to the image input to the diffusion model until the input image becomes a pure noise image. The reverse denoising process is a trainable process, the goal is to train a neural network, starting from a pure noise image, and gradually denoising until a real image similar to the input image is obtained.

[0026] The number of steps of adding noise in the forward diffusion process corresponds to the number of denoising steps in the reverse denoising process. N Sub-Gaussian noise, corresponding to the reverse denoising process N The final real image is obtained through prediction.

[0027] It is understandable that in the diffusion process, the more steps of adding Gaussian noise, that is, the less Gaussian noise is added each time, the less time and calculation the neural network needs for each prediction during the denoising process. At the same time, it is easier to obtain more accurate prediction results in each prediction.

[0028] Therefore, in the process of obtaining the second super-resolution image representing high-frequency details through the conventional diffusion model, a method of adding Gaussian noise in small amounts and multiple times is tended to be chosen to obtain a more accurate prediction result. However, this method requires more prediction times for the diffusion model in the denoising process, which makes the overall computational burden of the diffusion model heavier.

[0029] On this basis, in order to overcome the problem that the diffusion model needs to add noise multiple times and the computational burden caused by prediction in order to obtain more accurate prediction results, in this embodiment, the denoising process of the diffusion model is guided by the key features extracted from the low-resolution image to be reconstructed, and the deep features of the low-resolution image to be reconstructed are used as conditions to guide the iterative denoising, so that in each prediction process, the noise added in the diffusion process can be more accurately predicted and identified, thereby reducing the dependence on synthetic degradation and improving the quality.

[0030] That is to say, the denoising process guided by key features can achieve accurate prediction of more information in each prediction. Correspondingly, more noise can be added each time during the diffusion process, thereby reducing the number of steps required for the diffusion and denoising processes and obtaining a predicted image with more realistic details as the second super-resolution image.

[0031] Step 103: Generate a super-resolution image of the low-resolution image by combining the first super-resolution image and the second super-resolution image.

[0032] After obtaining a first super-resolution image representing low-frequency details and a second super-resolution image representing high-frequency details in the above manner, the first super-resolution image is combined with the second super-resolution image to obtain an SR (Super-Resolution) image of the low-resolution image, thereby achieving high-quality SR reconstruction.

[0033] The present invention combines the random iteration mechanism of diffusion model denoising with feature encoding technology, and uses the key features extracted from LR images to guide the denoising process of the diffusion model, thereby improving the efficiency and quality of the high-frequency detail recovery process of SISR to achieve high-quality SR reconstruction, and reducing the computational burden of the SISR model and the size of the SISR model.

[0034] In the single image super-resolution method combining feature encoding and diffusion model of the present invention, the step of obtaining a second super-resolution image for characterizing high-frequency details of the low-resolution image through the diffusion model specifically includes: Performing feature extraction on the low-resolution image to obtain key features of the low-resolution image; Optionally, a feature encoding module for extracting features from low-resolution images is provided in the upper branch of SRCAD.

[0035] In this embodiment, there is no restriction on the type of feature encoding module, as long as the key features of the LR image can be extracted.

[0036] In the denoising process of the diffusion model, the key features are connected to the initial downsampling layer of the decoding network to guide iterative denoising; The second super-resolution image output after denoising by the diffusion model is obtained.

[0037] The decoding network is a neural network used in the denoising process. Optionally, UNet (a semantic segmentation model) is used as the decoding network in this embodiment.

[0038] The key features are connected to the initial downsampling layer of UNet, that is, connected to the encoder of the UNet model, to guide the iterative denoising of the diffusion model in the denoising process.

[0039] Thus, a second super-resolution image output by the diffusion model is obtained after being guided by the key features. Under the guidance of the features, fewer diffusion steps are required and the restored details are more realistic.

[0040] In the single image super-resolution method combining feature encoding and diffusion model of the present invention, the step of connecting the key features to the initial downsampling layer of the decoding network for guiding iterative denoising specifically includes: Performing dimensionality reduction on the extracted key features based on distribution-aware feature subset fusion; The reduced key features are connected to the initial downsampling layer of the decoding network.

[0041] Further, such as Figure 2 As shown, a DFSF (Distributed Feature Subset Fusion) module is also provided directly between the feature encoding module and the decoding network, which is used to compress and reduce the dimension of the key features directly extracted by the feature encoding module to reduce redundancy.

[0042] The key features after DFSF dimension reduction are connected to the initial downsampling layer of the decoding network, thereby introducing feature encoding to guide denoising.

[0043] In the single image super-resolution method combining feature encoding and diffusion model of the present invention, the step of extracting features from the low-resolution image to obtain key features of the low-resolution image specifically includes: Using a pre-trained feature encoding module to extract features from the low-resolution image, to obtain key features output by the feature encoding module; The feature encoding module is composed of a plurality of continuously connected dense blocks with residual structures, and each of the dense blocks is provided with an attention module for achieving stable training.

[0044] In order to extract key features at a lower computational cost, a pre-trained feature encoding module is provided in this embodiment.

[0045] The structure of the feature encoding module is as follows Figure 3 As shown in the figure, it is composed of multiple continuously connected dense blocks with residual structures. The convolutional layers in each dense block are skipped and connected. The shallow features and deep features are gradually integrated through the skip connection, which improves the reconstruction accuracy of the diffusion model during reverse denoising.

[0046] Furthermore, since the features between adjacent channels extracted by the encoder are similar, it is easy to cause overfitting and increase the amount of calculation. Therefore, continue to refer to Figure 3 In the feature encoding module provided in this embodiment, an attention module is provided in the dense block to introduce an attention mechanism to avoid overfitting as much as possible, thereby achieving stable training.

[0047] In the single image super-resolution method combining feature encoding and diffusion model of the present invention, the attention module is defined as: ; In the formula, x represents the input of the attention module, W is the weight matrix, avgpool represents global average pooling; represents element-wise multiplication; ; In the formula, c is the number of feature channels, A feature vector corresponding to the first half of the input channels of the attention module; A feature vector corresponding to the second half of the input channel of the attention module; is the magnification factor; Represents the feature vector obtained by uniformly sampling the input of the attention module.

[0048] In this embodiment, a The attention mechanism of dimensional interleaved product channel attention is used to achieve stable training.

[0049] Among them, the dimension interleaved product structure is as follows Figure 4 As shown, C is the number of feature channels.

[0050] The Dimensional Interleaved Product (DIP) has two branches, defined as: ; like Figure 4 As shown, in the lower branches, The nonlinearity between channels is introduced by element-by-element multiplication of the first half and the second half of the feature. The upper branch uniformly samples the feature vector , and further zoom in times.

[0051] Additional flexibility is introduced, allowing the strength of nonlinear transformations to be adjusted dynamically without adding additional parameters. It should be noted that DIP introduces nonlinearity in a simple and effective way, such as channel interleaving and element-wise product, which improves the modeling capability of complex textures and structures.

[0052] Furthermore, in DIP, the attention mechanism is modified as , which is a simplified channel adaptive weight learning strategy: ; Compared with the traditional attention mechanism, the The intermediate transformation and activation function are omitted, which greatly reduces the amount of calculation.

[0053] In the single image super-resolution method combining feature encoding and diffusion model of the present invention, the step of obtaining a second super-resolution image for characterizing high-frequency details of the low-resolution image through the diffusion model specifically includes: During the diffusion process, the diffusion model gradually introduces noise into the residual image to obtain a normally distributed noise image, wherein the residual image represents the difference between the high-resolution image and the low-resolution image.

[0054] Since high-frequency details mainly reflect the difference between HR (High Resolution) images and LR images, Figure 2 As shown, SRCAD uses the residual image r 0 replaces the original input x 0 as input, thus simplifying the model.

[0055] During the forward diffusion process, SRCAD gradually moves to the residual image r 0, thus obtaining a normally distributed noise image x T In the reverse denoising process, UNet (a convolutional neural network-based architecture) is used as the denoising network, and the features extracted by the encoder are compressed and connected to the initial downsampling layer of the decoding network through DFSF to guide the iterative denoising.

[0056] In addition, in order to verify the single image super-resolution method provided by the present invention, the method is compared with the SOTA SR (State of the arts, i.e. other single image super-resolution models) method on the BSD100, Urban100 and DIV2K datasets at 4 times super-resolution. The results are shown in Table 1 and Table 2. Figure 5 As shown: Table 1 ; Table 1 shows the comparison results between the SRCAD model of the present invention and other SR models on the three datasets of BSD100, Urban100 and DIV2K. Specifically, three evaluation indicators, namely peak signal-to-noise ratio (PSNR), low-resolution peak signal-to-noise ratio (LR-PSNR) and structural similarity index (SSIM), are selected.

[0057] In the perception-oriented comparison, the subscripts (1) and (2) in the table are the top two results of the indicator in the perception-oriented methods. It can be found that the SRCAD framework is the best in all three indicators of BSD100 and DIV2K, and also performs well on the Urban100 dataset. At the same time, the number of parameters required by SRCAD is much smaller than that of the suboptimal SROOE model.

[0058] In addition, in the PSNR-oriented results, although the RRDB method also has a high evaluation index, it is still lower than Figure 5 It can be found that the imaging of the RRDB method is too smooth and therefore blurry. It can be seen that SRCAD is better than the RRDB method in imaging performance.

[0059] At 8x super resolution, the results are compared with the SOTA SR method on the CelebA dataset. The results are shown in Table 2 and Figure 6 As shown: Table 2 .

[0060] Table 2 shows the comparison results of the SRCAD model of the present invention and other SR models on the CelebA dataset, specifically selecting three evaluation indicators: peak signal-to-noise ratio (PSNR), low-resolution peak signal-to-noise ratio (LR-PSNR) and structural similarity index (SSIM).

[0061] In the comparison of perception-oriented results, the subscripts (1) and (2) in the table are the top two results of the indicator in the perception-oriented methods. It can be seen that the SRCAD framework has the best indicator evaluation on the CelebA dataset, and the number of parameters required is much smaller than the second-best SRFlow model.

[0062] In addition, in the PSNR-oriented results, although the RRDB method also has a high indicator evaluation, Figure 6 It can be found that the imaging of the RRDB method is too smooth and therefore blurry. It can be seen that SRCAD is still better than the RRDB method in imaging performance.

[0063] As can be seen from the above table, the present invention solves the ill-posedness of the SR task by combining efficient feature encoding and diffusion model, and can generate diverse SR images with rich details and high quality. DIP-CA The feature encoding module and DFSF significantly reduce the computational complexity and the number of parameters required for the model.

[0064] Compared with the GAN (Generative Adversarial Network) model, SRCAD is more stable during training and avoids problems such as mode collapse.

[0065] The single image super-resolution device combining feature coding and diffusion model provided by the present invention is described below. The single image super-resolution device combining feature coding and diffusion model described below and the single image super-resolution method combining feature coding and diffusion model described above can be referenced to each other.

[0066] like Figure 7 As shown, the single image super-resolution device combining feature coding and diffusion model includes a low-frequency image acquisition module 701, a high-frequency image acquisition module 702 and an image reconstruction module 703: A low-frequency image acquisition module 701 is used to upsample the low-resolution image to obtain a first super-resolution image for representing low-frequency details of the low-resolution image; Optionally, in this embodiment, a pre-built SRCAD (Single Image Super-Resolution with Channel Attention and Diffusion) framework is used to implement a single image super-resolution method combining feature encoding and diffusion model. In other embodiments, other feasible frameworks can also be used to implement the single image super-resolution method provided in this application.

[0067] like Figure 2 As shown in FIG. 1 , the SRCAD framework consists of two branches. The lower branch of SRCAD upsamples the low-resolution image (LowResolution, LR) by a multiple of 4 or 8 to obtain a first super-resolution image with low-frequency blur, that is, the low-frequency details of the low-resolution image to be reconstructed are represented by the first super-resolution image.

[0068] A high-frequency image acquisition module 702 is used to acquire a second super-resolution image for characterizing high-frequency details of the low-resolution image through a diffusion model, wherein a denoising process of the diffusion model is guided by key features extracted from the low-resolution image; The upper branch of SRCAD is used to obtain a second super-resolution image by combining the diffusion model and the key features extracted. The second super-resolution image is used to characterize the high-frequency details of the low-resolution image to be reconstructed.

[0069] like Figure 2 As shown in Figure 2, the application of diffusion model in single image super-resolution usually includes two processes: forward diffusion and backward denoising.

[0070] The forward diffusion process is to gradually add Gaussian noise to the image input to the diffusion model until the input image becomes a pure noise image. The reverse denoising process is a trainable process, the goal is to train a neural network, starting from a pure noise image, and gradually denoising until a real image similar to the input image is obtained.

[0071] The number of steps of adding noise in the forward diffusion process corresponds to the number of denoising steps in the reverse denoising process. N Sub-Gaussian noise, corresponding to the reverse denoising process N The final real image is obtained through prediction.

[0072] It is understandable that in the diffusion process, the more steps of adding Gaussian noise, that is, the less Gaussian noise is added each time, the less time and calculation the neural network needs for each prediction during the denoising process. At the same time, it is easier to obtain more accurate prediction results in each prediction.

[0073] Therefore, in the process of obtaining the second super-resolution image representing high-frequency details through the conventional diffusion model, a method of adding Gaussian noise in small amounts and multiple times is tended to be chosen to obtain a more accurate prediction result. However, this method requires more prediction times for the diffusion model in the denoising process, which makes the overall computational burden of the diffusion model larger.

[0074] On this basis, in order to overcome the problem that the diffusion model needs to add noise multiple times and predict the computational burden of navigation in order to obtain more accurate prediction results, in this embodiment, the denoising process of the diffusion model is guided by the key features extracted from the low-resolution image to be reconstructed, and the deep features of the low-resolution image to be reconstructed are used as conditions to guide the iterative denoising, so that in each prediction process, the noise added in the diffusion process can be more accurately predicted and identified, thereby reducing the reliance on synthetic degradation and improving the quality.

[0075] That is to say, the denoising process guided by key features can achieve accurate prediction of more information in each prediction. Correspondingly, more noise can be added each time during the diffusion process, thereby reducing the number of steps required for the diffusion and denoising processes and obtaining a predicted image with more realistic details as the second super-resolution image.

[0076] The image reconstruction module 703 is used to generate a super-resolution image of the low-resolution image by combining the first super-resolution image and the second super-resolution image.

[0077] After obtaining a first super-resolution image representing low-frequency details and a second super-resolution image representing high-frequency details in the above manner, the first super-resolution image is combined with the second super-resolution image to obtain an SR (Super-Resolution) image of the low-resolution image, thereby achieving high-quality SR reconstruction.

[0078] The present invention combines the random iteration mechanism of diffusion model denoising with feature encoding technology, and uses the key features extracted from LR images to guide the denoising process of the diffusion model, thereby improving the efficiency and quality of the high-frequency detail recovery process of SISR to achieve high-quality SR reconstruction, and reducing the computational burden of the SISR model and the size of the SISR model.

[0079] Figure 8 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 8 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication interface 820 and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute a single image super-resolution method combining feature encoding and a diffusion model, the method comprising: upsampling a low-resolution image to obtain a first super-resolution image for characterizing low-frequency details of the low-resolution image; obtaining a second super-resolution image for characterizing high-frequency details of the low-resolution image through a diffusion model, wherein the denoising process of the diffusion model is guided by key features extracted from the low-resolution image; and generating a super-resolution image of the low-resolution image by combining the first super-resolution image and the second super-resolution image.

[0080] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0081] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the single image super-resolution method combining feature encoding and diffusion model provided by the above methods, the method including: upsampling a low-resolution image to obtain a first super-resolution image for characterizing the low-frequency details of the low-resolution image; obtaining a second super-resolution image for characterizing the high-frequency details of the low-resolution image through a diffusion model, wherein the denoising process of the diffusion model is guided by key features extracted from the low-resolution image; and generating a super-resolution image of the low-resolution image by combining the first super-resolution image and the second super-resolution image.

[0082] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the single image super-resolution method combining feature encoding and a diffusion model provided by the above-mentioned methods, the method comprising: upsampling a low-resolution image to obtain a first super-resolution image for characterizing the low-frequency details of the low-resolution image; obtaining a second super-resolution image for characterizing the high-frequency details of the low-resolution image through a diffusion model, wherein the denoising process of the diffusion model is guided by key features extracted from the low-resolution image; and generating a super-resolution image of the low-resolution image by combining the first super-resolution image and the second super-resolution image.

[0083] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0084] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0085] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A single image super-resolution method combining feature encoding and diffusion model, characterized in that: include: Upsampling the low-resolution image to obtain a first super-resolution image for characterizing low-frequency details of the low-resolution image; Acquire a second super-resolution image for characterizing high-frequency details of the low-resolution image through a diffusion model, wherein a denoising process of the diffusion model is guided by key features extracted from the low-resolution image; A super-resolution image of the low-resolution image is generated by combining the first super-resolution image and the second super-resolution image.

2. The single image super-resolution method combining feature encoding and diffusion model according to claim 1, characterized in that: The step of acquiring a second super-resolution image for characterizing high-frequency details of the low-resolution image through a diffusion model specifically includes: Performing feature extraction on the low-resolution image to obtain key features of the low-resolution image; In the denoising process of the diffusion model, the key features are connected to the initial downsampling layer of the decoding network to guide iterative denoising; The second super-resolution image output after denoising by the diffusion model is obtained.

3. The single image super-resolution method combining feature encoding and diffusion model according to claim 1, characterized in that: The step of connecting the key features to the initial downsampling layer of the decoding network for guiding iterative denoising specifically comprises: Performing dimensionality reduction on the extracted key features based on distribution-aware feature subset fusion; The reduced key features are connected to the initial downsampling layer of the decoding network.

4. The single image super-resolution method combining feature encoding and diffusion model according to claim 2, characterized in that: The step of extracting features from the low-resolution image to obtain key features of the low-resolution image specifically includes: Using a pre-trained feature encoding module to extract features from the low-resolution image, to obtain key features output by the feature encoding module; The feature encoding module is composed of a plurality of continuously connected dense blocks with residual structures, and each of the dense blocks is provided with an attention module for achieving stable training.

5. The single image super-resolution method combining feature encoding and diffusion model according to claim 4, characterized in that: The attention module is defined as: ； In the formula, x represents the input of the attention module, W is the weight matrix, avgpool represents global average pooling; represents element-wise multiplication; ； In the formula, c is the number of feature channels, A feature vector corresponding to the first half of the input channels of the attention module; A feature vector corresponding to the second half of the input channel of the attention module; is the magnification factor; Represents the feature vector obtained by uniformly sampling the input of the attention module.

6. The single image super-resolution method combining feature coding and diffusion model according to any one of claims 1 to 5, characterized in that: The step of acquiring a second super-resolution image for characterizing high-frequency details of the low-resolution image through a diffusion model specifically includes: During the diffusion process, the diffusion model gradually introduces noise into the residual image to obtain a normally distributed noise image, wherein the residual image represents the difference between the high-resolution image and the low-resolution image.

7. A single image super-resolution device combining feature encoding and diffusion model, characterized in that: include: A low-frequency image acquisition module, used for upsampling the low-resolution image to obtain a first super-resolution image for characterizing low-frequency details of the low-resolution image; A high-frequency image acquisition module, configured to acquire a second super-resolution image for characterizing high-frequency details of the low-resolution image through a diffusion model, wherein a denoising process of the diffusion model is guided by key features extracted from the low-resolution image; An image reconstruction module is used to generate a super-resolution image of the low-resolution image by combining the first super-resolution image and the second super-resolution image.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the single image super-resolution method combining feature encoding and diffusion model as described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the single image super-resolution method combining feature encoding and diffusion model as claimed in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the single image super-resolution method combining feature encoding and diffusion model as claimed in any one of claims 1 to 6 is implemented.

Images