An arbitrary upscaling super-resolution enhancement system and method based on a diffusion model

Abstract

The application provides an arbitrary-magnification super-resolution enhancement system and method based on a diffusion model, comprising: an input module that inputs a low-resolution image and a target super-resolution enhancement magnification; an up-sampling unit that up-converts the low-resolution image to a target resolution through bicubic interpolation to obtain an intermediate target-resolution image; an image encoder that encodes the intermediate target-resolution image to obtain an implicit space input feature; an auxiliary feature module that extracts text description and semantic features from the intermediate target-resolution image; an implicit space module that denoises the implicit space input feature based on the target super-resolution enhancement magnification, the text description and the semantic features to obtain an intermediate feature; and an image decoder that decodes the intermediate feature to obtain a final image that meets the target super-resolution enhancement magnification. The application injects super-resolution magnification perception capability into the generation process of the diffusion model, so that the diffusion model can adaptively process image features according to the super-resolution enhancement magnification.

Description

Technical Field

[0001] This invention relates to a deep learning-based super-resolution enhancement method, and particularly to an arbitrary-magnification super-resolution enhancement system and method based on a diffusion model. Background Technology

[0002] In the 5G era, increased network bandwidth has driven the development of high-definition video streaming, virtual reality (VR), and augmented reality (AR), with users demanding increasingly clear and detailed visual experiences. As technology advances and application scenarios expand, especially with the widespread adoption of ultra-high-definition display devices such as 4K and 8K, traditional low-resolution images can no longer meet the growing demands for visual quality. Super-resolution technology has become key to overcoming this bottleneck. Super-resolution technology can effectively enhance low-resolution images to high resolution, improving visual details, reducing data transmission burden, and ensuring a smooth viewing experience. This provides technical support for major internet platforms, gaming companies, and film and television production companies.

[0003] Traditional super-resolution enhancement methods are usually only applicable to fixed magnification (e.g., 2×, 3×, 4×). Different magnifications require different super-resolution enhancement models. However, arbitrary magnification super-resolution can use a single model to flexibly generate high-resolution images of different magnifications as needed. It also supports fractional super-resolution, which greatly improves the flexibility and practicality of super-resolution enhancement technology applications.

[0004] Most current arbitrary-magnification super-resolution enhancement methods employ regression-based optimization approaches. These methods typically exhibit pattern averaging behavior in high-magnification scenarios, leading to blur artifacts and significantly reducing the visual quality for human observers. In recent years, researchers have improved optimization patterns by introducing generative models such as GANs and Diffusion, enhancing image realism. However, this has also introduced new problems, namely, inconsistencies between the generated results and the original image, or supplementary pixels that do not conform to objective logic. Therefore, achieving high fidelity and a high degree of conformity to natural image distribution have become two major challenges for current arbitrary-magnification super-resolution enhancement methods. Summary of the Invention

[0005] To address the shortcomings of existing technologies, the present invention aims to provide an arbitrary magnification super-resolution enhancement system and method based on a diffusion model.

[0006] According to one aspect of the present invention, an arbitrary-magnification super-resolution enhancement system based on a diffusion model is provided, comprising:

[0007] The input module is given a low-resolution image of arbitrary size and an arbitrary target super-resolution enhancement ratio;

[0008] The upsampling unit upsamples the low-resolution image to the target resolution using bicubic interpolation to obtain an intermediate target resolution image.

[0009] An image encoder encodes the intermediate target resolution image to obtain latent space input features;

[0010] The dual semantic guidance module collects the text description and semantic features of the intermediate target resolution image;

[0011] The latent space module denoises the latent space input features based on the target super-resolution enhancement ratio, the text description, and the semantic features to obtain intermediate features;

[0012] An image decoder decodes the intermediate features to obtain the final image that matches the target super-resolution enhancement ratio.

[0013] Preferably, the latent space module adopts a dual-branch network, including a low-resolution image branch and a diffusion model main branch;

[0014] The low-resolution image branch combines the text description, the semantic features, and the target super-resolution enhancement ratio to extract multi-scale features of the latent space input features;

[0015] The diffusion model's main branch combines the text description, semantic features, target super-resolution enhancement ratio, and random noise to extract latent space input image features, and fuses them with the multi-scale features to obtain the fused intermediate features.

[0016] Preferably, the low-resolution image branch comprises multiple groups, each group sequentially comprising a convolutional layer, a text attention layer, and a feature attention layer; the convolutional kernels of the convolutional layers in each group have different sizes; wherein, the convolutional layer injects the target super-resolution enhancement factor into the latent space input features or the output of the previous group; the text attention layer injects the text description into the latent space input features or the output of the previous group; the feature attention layer injects the semantic features into the latent space input features or the output of the previous group; through the outputs of multiple groups, multi-scale features of the latent space input features are obtained;

[0017] The main branch of the diffusion model includes a symmetric diffusion encoder and a diffusion decoder. The diffusion encoder and diffusion decoder each include multiple subgroups, each subgroup sequentially including a convolutional layer, a text attention layer, and a feature attention layer. The convolutional kernels of the convolutional layers in each subgroup have different sizes. The diffusion encoder takes noise, the text description, the semantic features, and the target super-resolution enhancement factor as input, and gradually reduces the width and height of the noise features while increasing the number of channels. The decoder takes the text description, the semantic features, the target super-resolution enhancement factor, and the multi-scale features as input, and gradually restores the original width and height of the noise features while reducing the number of channels to obtain the fused intermediate features.

[0018] Preferably, it also includes a global rate modulation unit, which modulates the time step features of the main branch of the diffusion model based on the target super-resolution enhancement rate modulation;

[0019] The global rate modulation unit includes:

[0020] The first encoder performs position encoding on the target super-resolution enhancement ratio;

[0021] The first linear layer performs linear processing on the position encoding to obtain a linear processing result;

[0022] The first activation layer activates the linear processing result to obtain super-resolution magnification features.

[0023] The update unit adds the super-resolution magnification feature to the time step feature of the main branch of the diffusion model to update the original time step feature.

[0024] Preferably, it further includes a local scaling unit, which modulates the multi-scale low-resolution image features extracted by the conditional branch of the low-resolution image based on the target super-resolution enhancement scaling.

[0025] The local rate modulation unit includes:

[0026] The second encoder performs position encoding on the target super-resolution enhancement ratio;

[0027] The second linear layer performs linear processing on the position encoding to obtain a linear processing result.

[0028] The second activation layer activates the linear processing result to obtain super-resolution magnification features.

[0029] The third linear layer performs linear processing on the super-resolution magnification features to obtain scale parameters and bias parameters.

[0030] The modulation unit multiplies the multi-scale low-resolution image features with the corresponding scale parameters, and then adds them to the corresponding bias parameters to obtain the modulated features.

[0031] Preferably, it includes a first training module, which trains the main branch network of the diffusion model;

[0032] The first training module includes:

[0033] The data submodule, given a high-resolution image, obtains the latent space representation z0 after passing through the main branch network of the diffusion model;

[0034] The noise-adding submodule progressively adds noise to the latent space representation z0 to obtain the noisy latent space representation z. t ;

[0035] The training submodule, combined with low-resolution image I lr Target super-resolution enhancement factor s, diffusion model time step t, text description c, semantic features p, training diffusion model main branch network ∈ θ To predict the latent space representation z added to the noisy space. t The noise in the training submodule; the loss function used by the training submodule is:

[0036]

[0037] in Indicates the calculation of prediction noise ∈ θ The L2 loss of the real noise ∈ is used as the diffusion loss, where N represents a uniformly distributed random noise distribution, and ε ~ N represents sampling a noise sample from the noise distribution N.

[0038] Preferably, it includes a second training module that trains the image encoder; the loss function of the second training module is a reconstruction loss function, which reconstructs structural detail information by constraining the intermediate features of the image encoder, and its specific expression is:

[0039]

[0040] in This represents the nth multi-scale feature extracted by the image encoder, and toRGB indicates that the feature is transformed into an RGB image using convolution. This represents the resolution of the nth feature when the real high-resolution image is downsampled.

[0041] According to a second aspect of the present invention, an arbitrary-magnification super-resolution enhancement method based on a diffusion model is provided, comprising:

[0042] Given a low-resolution image of arbitrary size and an arbitrary target super-resolution enhancement ratio;

[0043] The low-resolution image is upscaled to the target resolution using bicubic interpolation to obtain an intermediate target resolution image;

[0044] The intermediate target resolution image is encoded to obtain latent space input features;

[0045] Collect textual descriptions and semantic features of the intermediate target resolution image;

[0046] Based on the target super-resolution enhancement ratio, the text description, and the semantic features, the latent space input features are denoised to obtain intermediate features;

[0047] Decode the intermediate features to obtain the final image that matches the target super-resolution enhancement ratio.

[0048] According to a third aspect of the present invention, a terminal is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, can be used to run the system or to execute the method described herein.

[0049] According to a fourth aspect of the invention, a computer-readable storage medium has a computer program stored thereon, which, when executed by a processor, can be used to run the system or perform the method described herein.

[0050] Compared with the prior art, the embodiments of the present invention have at least one of the following beneficial effects:

[0051] This invention proposes an arbitrary-magnification super-resolution enhancement system and method based on a diffusion model, given a low-resolution image I. LR Given a desired super-resolution enhancement factor s (or the resolution of the target image), a unified latent space module is used to enhance a given low-resolution image to a high-resolution image I at the desired resolution. HR It expands image resolution, enhances visual details, and generates true high-resolution images that conform to the natural image distribution while fully preserving the original image information.

[0052] The present invention proposes an arbitrary magnification super-resolution enhancement system and method based on a diffusion model, which supports super-resolution enhancement at arbitrary resolution magnification. By super-resolution magnification modulation, it injects super-resolution magnification perception capability into the generation process of the diffusion model, enabling the diffusion model to adaptively process image features according to the super-resolution enhancement magnification.

[0053] The present invention proposes an arbitrary magnification super-resolution enhancement system and method based on a diffusion model. Through dual semantic guidance, the semantic preservation capability during the generation process is enhanced, so that the image generated by the diffusion model has high fidelity and high conformity to the distribution of real images.

[0054] The present invention proposes an arbitrary magnification super-resolution enhancement system and method based on a diffusion model, which can still generate realistic and high-fidelity images even when the resolution of the low-resolution image is as low as 32×32 and the target super-resolution enhancement magnification exceeds 20×, thus outperforming other arbitrary magnification super-resolution enhancement methods. Attached Figure Description

[0055] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0056] Figure 1 This is a structural diagram of an arbitrary magnification super-resolution enhancement system based on a diffusion model according to an embodiment of the present invention.

[0057] Figure 2 This is a schematic diagram of the training process of an arbitrary magnification super-resolution enhancement system and method based on a diffusion model according to an embodiment of the present invention.

[0058] Figure 3 This is a comparison of the diffusion-based arbitrary-magnification super-resolution enhancement system in this embodiment of the invention with other baseline models;

[0059] Figure 4 This is a visualization result of arbitrary magnification enhancement according to a specific embodiment of the present invention, and a comparison result with the baseline method;

[0060] Figure 5 This is a visualization of a specific embodiment of the present invention compared with more baseline methods. Detailed Implementation

[0061] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention. These all fall within the scope of protection of the present invention.

[0062] In one embodiment of the present invention, such as Figure 1 As shown, an arbitrary-magnification super-resolution enhancement system based on a diffusion model includes:

[0063] The input module is given a low-resolution image of arbitrary size and an arbitrary target super-resolution enhancement ratio;

[0064] The upsampling unit upsamples the low-resolution image to the target resolution using bicubic interpolation, resulting in an intermediate target resolution image.

[0065] An image encoder encodes an intermediate target resolution image to obtain latent space input features;

[0066] The dual semantic guidance module collects textual descriptions and semantic features of intermediate target resolution images;

[0067] The latent space module denoises the latent space input features based on the target super-resolution enhancement magnification, text description, and semantic features to obtain intermediate features;

[0068] The image decoder decodes intermediate features to obtain the final image that meets the target super-resolution enhancement ratio.

[0069] In the above embodiments, by providing a low-resolution image I LR Given the expected super-resolution enhancement factor s (or the resolution of the target image), a unified latent space module is used to upscale the specified low-resolution image to the expected high-resolution image I. HR This process expands the image resolution and enhances visual details. In this process, the information of the original image is fully preserved, and a true high-resolution image conforming to the natural image distribution is generated. The intermediate target resolution image is a coarse image, merely an extension of the low-resolution image I through interpolation. LR Adjusting to the target resolution results in a blurry, low-quality image with little texture detail.

[0070] In a preferred embodiment of the present invention, an upsampling unit and an image encoder are designed. The image encoder maps the features of the low-resolution image to the latent space module. Before passing through the image encoder, the low-resolution image is first upsampled to the target resolution through bicubic interpolation. Then, the latent space representation of the low-resolution image is obtained through the image encoder implemented by a convolutional neural network, which serves as the input to the latent space diffusion model (low-resolution branch).

[0071] To improve the high fidelity of the final generated image and ensure it conforms to the distribution characteristics of natural images, in a preferred embodiment of the present invention, the latent space module adopts a dual-branch design: one branch is a low-resolution image branch, and the other is a diffusion model main branch. The low-resolution image branch mainly targets the latent space input features mapped by the image encoder, further extracting multi-scale features from these latent space input features. The diffusion model main branch mainly relies on text description, semantic features, and the target super-resolution enhancement ratio to iteratively denoise the input random noise. During the iterative denoising process, image features are generated and fused with multi-scale features to obtain fused intermediate features.

[0072] To avoid inconsistencies between the intermediate feature decoding results of the latent space module and the original image, or to prevent supplementary pixels from deviating from objective logic, a preferred embodiment of the present invention includes a dual semantic guidance module. This module guides the latent space module in generating a high-resolution image that conforms to the content, structure, and texture of the original image, thereby enhancing the fidelity of the low-resolution input image. Specifically, the dual semantic guidance module includes a text guidance unit and a semantic guidance unit. The text guidance unit first generates a text description for the low-resolution image and then uses the generated text description to guide the generation of the high-resolution image. The semantic guidance unit extracts the semantic features of the low-resolution image and calculates cross-attention with the features of the diffusion model, ensuring that the semantic information of the low-resolution image is fully perceived during the generation of the high-resolution image.

[0073] In one specific embodiment, the low-resolution image branch includes multiple groups, each group sequentially including a convolutional layer, a text attention layer, and a feature attention layer; the convolutional kernels of the convolutional layers in each group have different sizes; wherein, the convolutional layer injects the target super-resolution enhancement factor into the latent space input features or the output of the previous group; the text attention layer injects text descriptions into the latent space input features or the output of the previous group; the feature attention layer injects semantic features into the latent space input features or the output of the previous group; through the outputs of multiple groups, multi-scale features of the latent space input features are obtained;

[0074] The main branch of the diffusion model includes a symmetric diffusion encoder and a diffusion decoder. Both the diffusion encoder and decoder consist of multiple subgroups, each containing a convolutional layer, a text attention layer, and a feature attention layer in sequence. The convolutional kernels in each subgroup have different sizes. The diffusion encoder takes noise, text description, semantic features, and target super-resolution enhancement factor as input, gradually reducing the width and height of the image features while increasing the number of channels. The decoder takes text description, semantic features, target super-resolution enhancement factor, and multi-scale features as input, gradually restoring the original width and height of the image features while reducing the number of channels, resulting in a fused intermediate feature. The fusion process involves fusing the multi-scale features output by the decoder's local scaling unit with the corresponding scale in the decoder of the main branch, ultimately outputting a feature with the same dimensions as the encoder's input features.

[0075] In the above embodiments, the abstract semantic information of the text description and the spatial semantic information of the semantic features complement each other, improving the fidelity of high-resolution images to low-resolution input images.

[0076] Having solved the aforementioned fidelity issue, it is necessary to consider the adaptation to the super-resolution enhancement ratio s. In order to enable the entire latent space module to adaptively process image features according to the super-resolution enhancement ratio, a preferred embodiment designs a global modulation unit and a local modulation unit.

[0077] The global rate modulation unit utilizes the time-step features of the rate modulation diffusion model enhanced by super-resolution.

[0078] The global rate modulation unit includes an encoder, linear layers, activation layers, and an update layer. First, the encoder performs position encoding on the super-resolution rate *s*. Then, two linear layers activated by the SiLU activation layer are used to obtain the super-resolution rate features. Finally, the update layer adds these features to the time-step features *E*. t The original time step features are updated to achieve global information modulation. Specifically, the formula for modulating the time step features based on the super-resolution magnification is as follows:

[0079] E p =Sinusoidal(s)

[0080] E s =Linear(SiLU(Linear(E) p )))

[0081] E = E t +E s

[0082] Where s represents the super-resolution scaling factor, Sinusoidal represents sinusoidal coding, Linear represents the linear layer, and E t SiLU represents the original time step and the activation function.

[0083] The local rate modulation unit extracts multi-scale low-resolution image features F for the conditional branch of the low-resolution image, modulates the multi-scale features F by super-resolution rate modulation, and then adds the modulated features to the corresponding resolution features of the Unet decoder of the diffusion model.

[0084] The local rate modulation unit is a channel-level super-resolution enhancement rate-adaptive feature transformation unit. It first obtains a rate feature for local feature modulation using a method similar to global rate modulation.

[0085] E ls =Linear(SiLU(Linear(E) p )))

[0086] Then, scale features α and bias features β are obtained through linear layers, and these two parameters are used to modulate low-resolution features across multiple scales. The formula for modulating low-resolution features through super-resolution magnification is as follows:

[0087] α,β=Linear α (E ls ), Linear β (E ls )

[0088] F = α⊙F + β

[0089] Linear α and Linear β This indicates a linear layer.

[0090] The super-resolution magnification modulation unit designed in the above embodiments injects super-resolution magnification perception capability into the generation process of the diffusion model, enabling the diffusion model to adaptively process image features according to the super-resolution enhancement magnification, and adaptively adjust the generation capability according to the target enhancement magnification: when the magnification is high, the generation capability is fully utilized to expand the missing pixels, and when the magnification is low, more attention is paid to the fidelity capability.

[0091] The above embodiments utilize the target super-resolution modulation high-resolution image generation process to adaptively adjust the generation capability of the generation model, thereby achieving super-resolution enhancement that supports arbitrary resolution magnification.

[0092] To improve the accuracy of the entire diffusion-based arbitrary-magnification super-resolution enhancement system, training is required. In a preferred embodiment, a training module is used, such as... Figure 2 As shown, the training module mainly includes:

[0093] The data pair construction submodule constructs training data pairs for arbitrary-magnification super-resolution enhancement. A 512×512 image is randomly cropped from a high-quality image as the high-resolution image. Then, the high-resolution image is downsampled by an arbitrary floating-point number between 1 and 16 to become the low-resolution image. To support high-quality high-resolution image output for any target resolution (including resolutions lower than 512×512), the resolution of the data pairs is randomly transformed during subsequent training. Text descriptions are generated based on the high-resolution images for subsequent training. At this point, paired high-resolution-low-resolution images and corresponding text descriptions are obtained, which serve as training data for subsequent training. Data augmentation methods can be used during training.

[0094] System framework construction sub-module: Constructing an arbitrary-magnification super-resolution enhancement system based on a diffusion model.

[0095] An architecture based on a latent space diffusion model is used. A VAE codec maps low-resolution images to a latent space module, where the diffusion model is applied. The VAE codec uses pre-trained weights and does not update parameters throughout training. The latent space module contains two branches: a main branch for the diffusion model and a branch for the low-resolution image. The low-resolution image conditional branch first upsamples the downsampled low-resolution image to the target resolution, then encodes the low-resolution image into the latent space module using an image encoder. Multi-scale features are then extracted and incorporated into the main branch of the diffusion model.

[0096] For the main branch of the entire diffusion model, the super-resolution rate modulation module (including global rate modulation unit and local rate modulation unit) and the dual semantic information guidance module (including text guidance unit and semantic guidance unit) mentioned in the above embodiments are used to enhance fidelity and the quality of the generated image, obtaining a high-resolution image that is closer to the distribution of the real image and has better visual effects without damaging the information of the original image. Among them, the semantic features in the semantic guidance unit are obtained through the latent space features of a target recognition network.

[0097] The aforementioned sub-modules apply the diffusion model to the latent space module, and the VAE codec required for encoding images into the latent space module uses a pre-trained model. The latent space diffusion model utilizes super-resolution rate modulation and dual semantic information guidance designed in this embodiment of the invention.

[0098] Loss function submodule: The latent space module is trained using diffusion loss and reconstruction loss.

[0099] Given a high-resolution image, it is first mapped to a latent space module using a VAE encoder to obtain latent space features z0. Then, noise is progressively added to z0 to obtain a noisy latent space representation z. t Using low-resolution image I LR The super-resolution enhancement factor s, the diffusion time step t, the text description c, and the semantic features p are used to train the diffusion network ε. θ To predict the latent space representation z added to the noisy space. t The noise in the data. The loss function is:

[0100]

[0101] in Indicates the calculation of prediction noise ∈ θ The L2 loss of the real noise ∈ is used as the diffusion loss, where N represents a uniformly distributed random noise distribution, and ε ~ N represents sampling a noise sample from the noise distribution N.

[0102] A low-resolution image encoder is jointly trained while the diffusion model is being trained. The multi-scale features extracted by the encoder are converted into RGB images of the corresponding resolution through convolution, and then the L1 loss is calculated by comparing them with the RGB images of the corresponding resolution.

[0103]

[0104] Where F e n This represents the multi-scale features of a low-resolution image encoder, where n represents a dimension. toRGB indicates converting the features into an RGB image of the corresponding resolution.

[0105] The complete training loss is a weighted sum of the diffusion loss and the reconstruction loss:

[0106] L total =L diff +α rec L rec

[0107] Where α rec Hyperparameters used to balance diffusion loss and reconstruction loss.

[0108] The aforementioned loss function submodule jointly optimizes the image encoder and latent space diffusion model for mapping low-resolution images to the latent space module.

[0109] Application Module: During inference, given an input low-resolution image and the target super-resolution enhancement ratio, the low-resolution image is first upscaled to the target resolution using bicubic interpolation. Then, an image encoder encodes the low-resolution image into the latent space module and feeds it into the low-resolution image branch network. For high-resolution coarse images, their textual and semantic features are extracted and fed into the low-resolution image branch and the main branch as conditions.

[0110] The main branch network is randomly initialized with noise, and the sampling method of DDPM is used for iterative denoising. The number of steps is set to 20. After obtaining the denoised latent space representation, the final high-resolution output image is obtained through the pre-trained VAE decoder.

[0111] The inference process of the above application module is implemented after all training is completed, and can enhance the input low-resolution image to a high-resolution image of arbitrary resolution according to the expected super-resolution magnification.

[0112] Based on the same inventive concept, other embodiments of the present invention also provide an arbitrary magnification super-resolution enhancement method based on a diffusion model, the steps of which are as follows:

[0113] Step 1: Given a low-resolution image of arbitrary size and an arbitrary target super-resolution enhancement ratio;

[0114] Step 2: Upgrade the low-resolution image to the target resolution using bicubic interpolation to obtain a coarse target resolution image;

[0115] Step 3: Encode the coarse target resolution image to obtain the latent space input features;

[0116] Step 4: Collect textual descriptions and semantic features of the coarse target resolution image;

[0117] Step 5: Based on the target super-resolution enhancement ratio, text description, and semantic features, denoise the latent space input features to obtain intermediate features;

[0118] Step 6: Decode the intermediate features to obtain the final image that matches the target super-resolution enhancement ratio.

[0119] The steps in the above examples of the present invention can refer to the implementation techniques of each unit / module of the diffusion-based arbitrary magnification super-resolution enhancement system in the above embodiments, and will not be repeated here.

[0120] To verify the feasibility and effectiveness of the diffusion model-based arbitrary-magnification super-resolution enhancement system and method described in the above embodiments, a specific embodiment of the present invention selects the validation set of the DIV2K dataset as the test image dataset. Arbitrary-magnification super-resolution enhancement methods LIIF, CiaoSR, and IDM, as well as real-world blind super-resolution enhancement methods StableSR, PASD, and SeeSR, are used as baseline models.

[0121] PSNR, LPIPS, FID, and CLIPIQA were used as evaluation metrics. PSNR measures pixel similarity, LPIPS evaluates perceptual similarity, FID calculates the distance between the generated image distribution and the real distribution, and CLIPIQA is used to evaluate image quality.

[0122] Figure 3 The results of comparing the arbitrary-magnification super-resolution enhancement system and method based on the diffusion model of this invention with other baseline models are shown. 4×, 8×, and 16× represent results within the training distribution, while 18.2× represents results outside the training distribution. CIQA is an abbreviation for CLIPIQA. Figure 3 The best and second-best results are marked with bold and underline, respectively. From Figure 3It can be seen that the system / method of this invention significantly outperforms other methods at high super-resolution magnification. The two regression-based methods, LIIF and CiaoSR, achieved the highest PSNR values, but their LPIPS, FID, and CLIPIQA were far inferior to the method of this invention, indicating that they cannot generate high-quality images that conform to the real image distribution and human visual perception. Most diffusion-based blind super-resolution methods perform well at 4× super-resolution, but as the super-resolution magnification gradually increases, the image quality generated by IDM, StableSR, and PASD significantly deteriorates, reflected in a marked decrease in LPIPS, FID, and CLIPIQA metrics.

[0123] Figure 4 The visualization demonstrates the system and method of this invention's embodiments, showcasing their ability to achieve arbitrary-magnification super-resolution enhancement. The top and bottom images respectively display two arbitrary super-resolution scenes. The top image shows the ability to achieve arbitrary-magnification super-resolution enhancement with a fixed low-resolution image. The bottom image shows the ability to upsample different low-resolution images to a fixed-resolution high-resolution image, with the top row showing the interpolation results and the bottom row showing the results of the system and method of this invention's embodiments. The top-right image visualizes the comparison between the system and method of this invention's embodiments and IDM; clearly, it significantly outperforms IDM in terms of the realism and naturalness of structure and texture. The bottom-right image shows the comparison between the system and method of this invention's embodiments and SeeSR; although SeeSR can generate rich details, it is significantly superior to SeeSR in terms of semantic plausibility and image fidelity. Figure 5 The results show comparisons with other baseline models in 16× super-resolution enhancement. CiaoSR and StableSR fail to generate results that conform to the natural image distribution in high-magnification super-resolution enhancement scenarios. SeeSR generates images with high realism but low fidelity. The system and method of this invention can generate high-quality, high-fidelity images.

[0124] Those skilled in the art will understand that embodiments of the present invention can be provided as methods or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0125] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0126] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0127] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0128] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the invention.

[0129] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. An arbitrary-magnification super-resolution enhancement system based on a diffusion model, characterized in that, include: The input module is given a low-resolution image of arbitrary size and an arbitrary target super-resolution enhancement ratio; The upsampling unit upsamples the low-resolution image to the target resolution using bicubic interpolation to obtain an intermediate target resolution image. An image encoder encodes the intermediate target resolution image to obtain latent space input features; The dual semantic guidance module collects the text description and semantic features of the intermediate target resolution image; The latent space module denoises the latent space input features based on the target super-resolution enhancement ratio, the text description, and the semantic features to obtain intermediate features; An image decoder decodes the intermediate features to obtain the final image that matches the target super-resolution enhancement ratio; The latent space module employs a dual-branch network, including a low-resolution image branch and a diffusion model main branch; The low-resolution image branch combines the text description, the semantic features, and the target super-resolution enhancement ratio to extract multi-scale features of the latent space input features; The diffusion model's main branch combines the text description, semantic features, target super-resolution enhancement ratio, and random noise to extract latent space input feature image features, and fuses them with the multi-scale features to obtain the fused intermediate features. The low-resolution image branch comprises multiple groups, each group sequentially including a convolutional layer, a text attention layer, and a feature attention layer; the convolutional kernels of the convolutional layers in each group have different sizes; wherein, the convolutional layer injects the target super-resolution enhancement factor into the latent space input features or the output of the previous group; the text attention layer injects the text description into the latent space input features or the output of the previous group; the feature attention layer injects the semantic features into the latent space input features or the output of the previous group; through the outputs of multiple groups, the multi-scale features of the latent space input features are obtained; The main branch of the diffusion model includes a symmetric diffusion encoder and a diffusion decoder. The diffusion encoder and diffusion decoder each include multiple subgroups, each subgroup sequentially including a convolutional layer, a text attention layer, and a feature attention layer. The convolutional kernels of the convolutional layers in each subgroup have different sizes. The diffusion encoder takes noise, the text description, the semantic features, and the target super-resolution enhancement factor as input, and gradually reduces the width and height of the noise features while increasing the number of channels. The decoder takes the text description, the semantic features, the target super-resolution enhancement factor, and the multi-scale features as input, and gradually restores the original width and height of the noise features while reducing the number of channels to obtain the fused intermediate features.

2. The arbitrary-magnification super-resolution enhancement system based on a diffusion model according to claim 1, characterized in that, It also includes a global rate modulation unit, which enhances the time step features of the main branch of the diffusion model based on the target super-resolution enhancement rate modulation. The global rate modulation unit includes: The first encoder performs position encoding on the target super-resolution enhancement ratio; The first linear layer performs linear processing on the position encoding to obtain a linear processing result; The first activation layer activates the linear processing result to obtain super-resolution magnification features. The update unit adds the super-resolution magnification feature to the time step feature of the main branch of the diffusion model to update the original time step feature.

3. The arbitrary-magnification super-resolution enhancement system based on a diffusion model according to claim 1, characterized in that, It also includes a local scaling unit, which modulates the multi-scale low-resolution image features extracted by the conditional branch of the low-resolution image based on the target super-resolution enhancement scaling. The local rate modulation unit includes: The second encoder performs position encoding on the target super-resolution enhancement ratio; The second linear layer performs linear processing on the position encoding to obtain a linear processing result. The second activation layer activates the linear processing result to obtain super-resolution magnification features. The third linear layer performs linear processing on the super-resolution magnification features to obtain scale parameters and bias parameters. The modulation unit multiplies the multi-scale low-resolution image features with the corresponding scale parameters, and then adds them to the corresponding bias parameters to obtain the modulated features.

4. The arbitrary-magnification super-resolution enhancement system based on a diffusion model according to claim 1, characterized in that, Includes a first training module, which trains the main branch network of the diffusion model; The first training module includes: The data submodule, given a high-resolution image, obtains the latent space representation after passing through the main branch network of the diffusion model. ; The noise-adding submodule represents the latent space. By gradually adding noise, a noisy latent space representation is obtained. ; Training submodule, combined with low-resolution images Target super-resolution enhancement factor s diffusion model time step t Text description c semantic features p Training the main branch network of the diffusion model To predict the latent space representation added to the noisy space. The noise in the training submodule; the loss function used by the training submodule is: ； in Indicates the calculation of prediction noise and real noise The L2 loss is used as the diffusion loss, where N represents a uniformly distributed random noise distribution, and ε~N represents sampling a noise sample from the noise distribution N.

5. The arbitrary-magnification super-resolution enhancement system based on a diffusion model according to claim 4, characterized in that, The system includes a second training module that trains the image encoder. The loss function of this second training module is a reconstruction loss function, which reconstructs structural details by constraining the intermediate features of the image encoder. Its specific expression is as follows: ； in This represents the nth feature extracted from the multi-scale features by the image encoder. This indicates that convolution is used to transform the features into an RGB image. This represents the resolution of the nth feature when the real high-resolution image is downsampled.

6. A method for arbitrary-magnification super-resolution enhancement based on a diffusion model, employing an arbitrary-magnification super-resolution enhancement system based on a diffusion model according to any one of claims 1-5, characterized in that, include: Given a low-resolution image of arbitrary size and an arbitrary target super-resolution enhancement ratio; The low-resolution image is upscaled to the target resolution using bicubic interpolation to obtain an intermediate target resolution image; The intermediate target resolution image is encoded to obtain latent space input features; Collect textual descriptions and semantic features of the intermediate target resolution image; Based on the target super-resolution enhancement ratio, the text description, and the semantic features, the latent space input features are denoised to obtain intermediate features; Decode the intermediate features to obtain the final image that matches the target super-resolution enhancement ratio.

7. A terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it can be used to run the system of any one of claims 1-5, or to execute the method of claim 6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program can be used to run the system of any one of claims 1-5, or to perform the method of claim 6.

Images