Super-resolution image generation method and device, electronic equipment and storage medium

Abstract

The application provides a super-resolution image generation method and device, electronic equipment and storage medium, and relates to the technical field of image processing, wherein the method comprises: inputting latent features of a low-resolution image into a feature extraction module to obtain multi-scale low-resolution spatial features; inputting features corresponding to random noise and the multi-scale low-resolution spatial features into a first feature modulation layer to obtain features outputted by iteration after denoising, and inputting the features outputted by iteration after denoising and features outputted by corresponding layers in a first encoder into a second feature modulation layer to obtain an initial high-resolution image; and inputting the latent features and the initial high-resolution image into a refiner to obtain a super-resolution image. The application designs a spatial-prior feature extraction module, embeds the multi-scale low-resolution spatial features into a denoising process through the first feature modulation layer to constrain the spatial consistency of the initial high-resolution image and the low-resolution image, and improves the quality and fidelity of the generated super-resolution image.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method, apparatus, electronic device, and storage medium for generating super-resolution images. Background Technology

[0002] With the rapid development of display technology, people's demand for high-resolution and high-definition images is increasing. Real-scene image super-resolution technology aims to restore low-quality, low-resolution images into high-quality, high-detail high-resolution images, and has extremely high application value in fields such as medical imaging, satellite remote sensing, security monitoring, and restoration of old images.

[0003] In related technologies, low-resolution images are typically input into a diffusion model. The diffusion model upsamples the low-resolution image and guides it through cross-attention, ultimately outputting a super-resolution image.

[0004] However, in the aforementioned related technologies, the low-resolution image is only upsampled using a diffusion model and then guided by cross-attention, resulting in the generated super-resolution image being spatially inconsistent with the original low-resolution image, thereby reducing the quality and fidelity of the generated super-resolution image. Summary of the Invention

[0005] This invention provides a method, apparatus, electronic device, and storage medium for generating super-resolution images, in order to address the shortcomings of existing technologies that reduce the quality and fidelity of generated super-resolution images.

[0006] This invention provides a method for generating super-resolution images, comprising the following steps.

[0007] The low-resolution image is input into the first encoder of the image super-resolution model to obtain the latent features output by the first encoder. The latent features are input into the feature extraction module of the image super-resolution model to obtain the multi-scale low-resolution spatial features output by the feature extraction module. The random noise is input into the second encoder of the image super-resolution model to obtain the features corresponding to the random noise output by the second encoder; The features corresponding to the random noise and the low-resolution spatial features of the multi-scale are input into the first feature modulation layer inserted in the first decoder of the image super-resolution model to obtain the denoised features output by the first feature modulation layer after iteration. The denoised features and the features output by the corresponding layer in the first encoder are input into the second feature modulation layer inserted in the second decoder of the image super-resolution model to obtain the initial high-resolution image output by the second feature modulation layer. The latent features and the initial high-resolution image are input into the thinner of the image super-resolution model to obtain the super-resolution image output by the thinner.

[0008] According to a super-resolution image generation method provided by the present invention, the step of inputting a low-resolution image into the first encoder of an image super-resolution model to obtain the latent features output by the first encoder includes: Obtain the original low-resolution image; The original low-resolution image is upsampled to obtain the upsampled image; The upsampled image is input as the low-resolution image into the first encoder to obtain the latent features output by the first encoder.

[0009] According to a super-resolution image generation method provided by the present invention, the feature extraction module includes a first convolutional layer, a feature enhancement module, a first downsampling module, and a processing module. The feature enhancement module includes a first residual network and a first self-attention layer. The processing module includes a second residual network, a second self-attention layer, and a channel shifting operation module. The latent features are input into the feature extraction module of the image super-resolution model to obtain multi-scale low-resolution spatial features output by the feature extraction module, including: The latent features are input into the first convolutional layer to obtain the convolutional features output by the first convolutional layer. The convolutional features are sequentially input into the first residual network and the first self-attention layer of the feature enhancement module to obtain the first latent feature. The first potential feature is input into the first downsampling module to obtain the first feature output by the first downsampling module; The first feature is sequentially input into the second residual network, the second self-attention layer, and the channel displacement operation module of the processing module to obtain the second potential feature; Based on the first latent feature and the second latent feature, the multi-scale low-resolution spatial features are determined.

[0010] According to a super-resolution image generation method provided by the present invention, the feature extraction module further includes a second downsampling module and a depth feature extraction module, wherein the depth feature extraction module includes a third residual network and a translation operation module; The determination of the multi-scale low-resolution spatial features based on the first latent feature and the second latent feature includes: The second potential feature is input into the second downsampling module to obtain the second feature output by the second downsampling module; The second feature is sequentially input into the third residual network and the displacement operation module of the depth feature extraction module to obtain the third latent feature; Based on the first latent feature, the second latent feature, and the third latent feature, the multi-scale low-resolution spatial features are determined.

[0011] According to the super-resolution image generation method provided by the present invention, the first feature modulation layer includes a convolutional network and a normalization module; The step of inputting the features corresponding to the random noise and the multi-scale low-resolution spatial features into the first feature modulation layer inserted in the first decoder of the image super-resolution model to obtain the denoised features output by the first feature modulation layer after iterative processing includes: The multi-scale low-resolution spatial features are input into the convolutional network to obtain the affine parameter scaling factor and offset output by the convolutional network. The features corresponding to the random noise are input into the normalization module to obtain the mean and standard deviation of the features corresponding to the random noise output by the normalization module. Based on the features corresponding to the affine parameter scaling factor, the offset, the mean, the standard deviation, and the random noise, the fused features are obtained; The fused features are iterated to obtain the denoised features.

[0012] According to a super-resolution image generation method provided by the present invention, the step of inputting the latent features and the initial high-resolution image into the thinner of the image super-resolution model to obtain the super-resolution image output by the thinner includes: The latent features and the initial high-resolution image are concatenated along the channel dimension to obtain a joint feature representation; The joint feature representation is input into the fourth residual network to obtain the residual features output by the fourth residual network; The super-resolution image is obtained by controlling the contribution ratio of the latent features and the initial high-resolution image in the residual features based on the trade-off parameter.

[0013] The present invention also provides a super-resolution image generation apparatus, comprising: The first encoding unit is used to input the low-resolution image into the first encoder of the image super-resolution model to obtain the latent features output by the first encoder. An extraction unit is used to input the latent features into the feature extraction module of the image super-resolution model to obtain multi-scale low-resolution spatial features output by the feature extraction module; The second encoding unit is used to input random noise into the second encoder of the image super-resolution model to obtain the features corresponding to the random noise output by the second encoder; The modulation unit is used to input the features corresponding to the random noise and the low-resolution spatial features of the multi-scale into the first feature modulation layer inserted in the first decoder of the image super-resolution model to obtain the denoised features output by the first feature modulation layer after iteration, and input the denoised features and the features output by the corresponding layer in the first encoder into the second feature modulation layer inserted in the second decoder of the image super-resolution model to obtain the initial high-resolution image output by the second feature modulation layer. A thinning unit is used to input the latent features and the initial high-resolution image into the thinner of the image super-resolution model to obtain the super-resolution image output by the thinner.

[0014] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the super-resolution image generation method as described above.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the super-resolution image generation method as described above.

[0016] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the super-resolution image generation method as described above.

[0017] The super-resolution image generation method, apparatus, electronic device, and storage medium provided by this invention involve inputting a low-resolution image into a first encoder of an image super-resolution model to obtain latent features; inputting the latent features into a feature extraction module of the image super-resolution model to obtain multi-scale low-resolution spatial features; inputting random noise into a second encoder of the image super-resolution model to obtain features corresponding to the random noise; inputting the features corresponding to the random noise and the multi-scale low-resolution spatial features into a first feature modulation layer inserted into a first decoder of the image super-resolution model to obtain denoised features after iterative output; inputting the denoised features and the features output from the corresponding layer in the first encoder into a second feature modulation layer inserted into a second decoder of the image super-resolution model to obtain an initial high-resolution image; and finally inputting the latent features and the initial high-resolution image into a thinner of the image super-resolution model to obtain a super-resolution image output by the thinner. As can be seen, the present invention designs a spatial prior feature extraction module to extract multi-scale low-resolution spatial features from the latent features of low-resolution images, and embeds the multi-scale low-resolution spatial features into the denoising process through the first feature modulation layer to constrain the spatial consistency between the initial high-resolution image and the low-resolution image. Furthermore, the initial high-resolution image is thinned by a thinner to obtain a super-resolution image, thereby improving the quality and fidelity of the generated super-resolution image. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 This is one of the flowcharts illustrating the super-resolution image generation method provided in this embodiment of the invention.

[0020] Figure 2 This is the second flowchart illustrating the super-resolution image generation method provided in this embodiment of the invention.

[0021] Figure 3 This is the third flowchart of the super-resolution image generation method provided in this embodiment of the invention.

[0022] Figure 4 This is the fourth flowchart of the super-resolution image generation method provided in this embodiment of the invention.

[0023] Figure 5 This is the fifth flowchart illustrating the super-resolution image generation method provided in this embodiment of the invention.

[0024] Figure 6This is a schematic diagram of the comparative experimental visualization results provided in the embodiments of the present invention.

[0025] Figure 7 This is a schematic diagram of the structure of the super-resolution image generation device provided in an embodiment of the present invention.

[0026] Figure 8 This is a schematic diagram of the physical structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0028] The following is combined Figures 1-6 The present invention describes a super-resolution image generation method. The execution subject of this super-resolution image generation method can be an electronic device such as a computer, tablet computer, terminal, or server, or it can be a super-resolution image generation device installed in such an electronic device. This super-resolution image generation device can be implemented through software, hardware, or a combination of both.

[0029] Figure 1 This is one of the flowcharts illustrating the super-resolution image generation method provided in this embodiment of the invention, such as... Figure 1 As shown, the super-resolution image generation method includes the following steps: Step 101: Input the low-resolution image into the first encoder of the image super-resolution model to obtain the latent features output by the first encoder.

[0030] The first encoder can be a variational autoencoder (VAE).

[0031] For example, a low-resolution image is input into a pre-trained VAE encoder. Through feature compression and mapping by the VAE encoder, the latent features corresponding to the low-resolution image are output, thus completing the feature encoding process of the low-resolution image.

[0032] Step 102: Input the latent features into the feature extraction module of the image super-resolution model to obtain the multi-scale low-resolution spatial features output by the feature extraction module.

[0033] For example, latent features are input into a preset feature extraction module. Through the multi-scale convolution and feature fusion structure of the feature extraction module, spatial structure information (e.g., edges, texture details, etc.) of low-resolution images at different scales is captured, and finally multi-scale low-resolution spatial features are output as prior information.

[0034] Step 103: Input random noise into the second encoder of the image super-resolution model to obtain the features corresponding to the random noise output by the second encoder.

[0035] The second encoder can be a U-Net encoder.

[0036] For example, according to the noise generation rules of the diffusion model, random noise that conforms to a normal distribution is generated. This random noise is then input into a pre-trained U-Net encoder. Through the hierarchical feature extraction of the U-Net encoder, the features corresponding to the random noise are obtained, providing a basis for subsequent feature fusion.

[0037] Step 104: Input the features corresponding to the random noise and the low-resolution spatial features of the multi-scale into the first feature modulation layer inserted in the first decoder of the image super-resolution model to obtain the denoised features output by the first feature modulation layer after iterative processing. Then, input the denoised features and the features output by the corresponding layer in the first encoder into the second feature modulation layer inserted in the second decoder of the image super-resolution model to obtain the initial high-resolution image output by the second feature modulation layer.

[0038] The first decoder can be a U-Net decoder, and the second decoder can be a VAE decoder.

[0039] For example, the first feature modulation layer is inserted into each level of the U-Net decoder, and the second feature modulation layer is inserted into each level of the VAE decoder. Then, the features corresponding to random noise and the low-resolution spatial features of multiple scales are accurately injected into the spatial prior through the first feature modulation layer to generate fused features that have both spatial prior constraints and noise guidance characteristics. After iterating the fused features for a preset number of steps, the denoised features are obtained. Then, the denoised features and the features output by the corresponding layer in the VAE encoder are input into the second feature modulation layer inserted in the VAE decoder. Here, the corresponding layer refers to the layer corresponding to the current layer of the VAE decoder, and finally the initial high-resolution image is obtained.

[0040] Step 105: Input the latent features and the initial high-resolution image into the thinner of the image super-resolution model to obtain the super-resolution image output by the thinner.

[0041] For example, an initial high-resolution image is input into a preset thinner. Through local detail optimization and texture enhancement by the thinner, a super-resolution image that meets the requirements is finally output, realizing the reconstruction from a low-resolution image to a super-resolution image.

[0042] The super-resolution image generation method provided by this invention involves inputting a low-resolution image into the first encoder of an image super-resolution model to obtain latent features. These latent features are then input into the feature extraction module of the image super-resolution model to obtain multi-scale low-resolution spatial features. Random noise is input into the second encoder of the image super-resolution model to obtain features corresponding to the random noise. The features corresponding to the random noise and the multi-scale low-resolution spatial features are then input into the first feature modulation layer inserted into the first decoder of the image super-resolution model to obtain denoised features after iterative output. The denoised features and the features output from the corresponding layer in the first encoder are then input into the second feature modulation layer inserted into the second decoder of the image super-resolution model to obtain an initial high-resolution image. Finally, the latent features and the initial high-resolution image are input into the thinner of the image super-resolution model to obtain the super-resolution image output by the thinner. As can be seen, the present invention designs a spatial prior feature extraction module to extract multi-scale low-resolution spatial features from the latent features of low-resolution images, and embeds the multi-scale low-resolution spatial features into the denoising process through the first feature modulation layer to constrain the spatial consistency between the initial high-resolution image and the low-resolution image. Furthermore, the initial high-resolution image is thinned by a thinner to obtain a super-resolution image, thereby improving the quality and fidelity of the generated super-resolution image.

[0043] In one embodiment, step 101 above inputs the low-resolution image into the first encoder of the image super-resolution model to obtain the latent features output by the first encoder, which can be implemented in the following way: Acquire the original low-resolution image; upsample the original low-resolution image to obtain the upsampled image; input the upsampled image as the low-resolution image into the first encoder to obtain the latent features output by the first encoder.

[0044] For example, the acquired original low-resolution image is upsampled to obtain an upsampled image. The upsampled image is then used as a low-resolution image and input into the first encoder to obtain the latent features corresponding to the original low-resolution image output by the first encoder.

[0045] In this embodiment, the original low-resolution image is first upsampled to increase the spatial resolution of the input original low-resolution image to a higher dimension. Then, the upsampled image is input into the first encoder for feature extraction. This allows the first encoder to capture denser spatial pixel information in the early stage of feature extraction, so that the final potential features retain richer details in the spatial dimension, providing a more sufficient spatial feature foundation for subsequent super-resolution reconstruction.

[0046] In one embodiment, the feature extraction module includes a first convolutional layer, a feature enhancement module, a first downsampling module, and a processing module. The feature enhancement module includes a first residual network and a first self-attention layer. The processing module includes a second residual network, a second self-attention layer, and a channel shifting operation module. Figure 2 This is a second schematic flowchart of the super-resolution image generation method provided in this embodiment of the invention, as shown below. Figure 2 As shown, step 102 above inputs the latent features into the feature extraction module of the image super-resolution model to obtain the multi-scale low-resolution spatial features output by the feature extraction module. This can be achieved through the following steps: Step 201: Input the latent features into the first convolutional layer to obtain the convolutional features output by the first convolutional layer.

[0047] For example, the latent features are first passed through the first convolutional layer for dimensionality adjustment to obtain the convolutional features output by the first convolutional layer.

[0048] Step 202: Input the convolutional features into the first residual network and the first self-attention layer of the feature enhancement module in sequence to obtain the first latent feature.

[0049] For example, the convolutional features are input into a feature enhancement module consisting of a first residual network and a first self-attention layer. First, the first residual network ensures the effective preservation of feature details, and then the first self-attention layer enhances the modeling ability of global spatial relationships, finally outputting the first latent feature.

[0050] Step 203: Input the first potential feature into the first downsampling module to obtain the first feature output by the first downsampling module.

[0051] For example, the first latent feature is input into the first downsampling module, and the first downsampling module downsamples the first latent feature to obtain the first feature.

[0052] Step 204: Input the first feature sequentially into the second residual network, the second self-attention layer, and the channel displacement operation module of the processing module to obtain the second potential feature.

[0053] For example, the first feature obtained from downsampling is sequentially input into a processing module consisting of a second residual network, a second self-attention layer, and a channel shifting operation module. Specifically, the first feature is first input into the second residual network, the output of the second residual network is used as the input of the second self-attention layer, and then the output of the second self-attention layer is used as the input of the channel shifting operation module. This process is repeated three times: the feature output from the first channel shifting operation module is again sequentially input into the second residual network, the second self-attention layer, and the channel shifting operation module; the feature output from the second channel shifting operation module is again sequentially input into the second residual network, the second self-attention layer, and the channel shifting operation module. This grouping and shifting of feature channels enhances the local feature perception capability. Finally, the feature output from the third channel shifting operation module is used as the second latent feature.

[0054] Step 205: Based on the first latent feature and the second latent feature, determine the multi-scale low-resolution spatial features.

[0055] For example, when the first latent feature and the second latent feature are obtained, the combination of the first latent feature and the second latent feature is used as a multi-scale low-resolution spatial feature.

[0056] In this embodiment, the latent features are sequentially passed through a first convolutional layer, a feature enhancement module consisting of a first residual network and a first self-attention layer, a first downsampling module, and a processing module consisting of a second residual network, a second self-attention layer, and a channel shifting operation module, ultimately obtaining multi-scale low-resolution spatial features that contain both global spatial relationships and local detail perception capabilities.

[0057] In one embodiment, the feature extraction module further includes a second downsampling module and a depth feature extraction module, wherein the depth feature extraction module includes a third residual network and a displacement operation module; Figure 3 This is the third flowchart illustrating the super-resolution image generation method provided in this embodiment of the invention, as shown below. Figure 3 As shown, step 205 above determines the multi-scale low-resolution spatial features based on the first latent feature and the second latent feature, which can be achieved through the following steps: Step 2051: Input the second potential feature into the second downsampling module to obtain the second feature output by the second downsampling module.

[0058] For example, the second latent feature is input into the second downsampling module, and the second latent feature is downsampled by the second downsampling module to obtain the second feature.

[0059] Step 2052: Input the second feature into the third residual network and displacement operation module of the depth feature extraction module in sequence to obtain the third potential feature.

[0060] For example, the second feature obtained by downsampling is input into a deep feature extraction module consisting of a third residual network and a displacement operation module. Deep feature extraction is performed through alternating processing of the third residual network and the displacement operation module, and finally a third latent feature is obtained.

[0061] Step 2053: Based on the first latent feature, the second latent feature, and the third latent feature, determine the multi-scale low-resolution spatial features.

[0062] For example, when the first, second, and third latent features are obtained, the combination of the first, second, and third latent features is used as a multi-scale low-resolution spatial feature.

[0063] In this embodiment, the latent features are sequentially passed through a first convolutional layer, a feature enhancement module consisting of a first residual network and a first self-attention layer, a first downsampling module, a processing module consisting of a second residual network, a second self-attention layer, and a channel shifting operation module, a second downsampling module, and a deep feature extraction module consisting of a third residual network and a shifting operation module. This process ultimately yields a spatial feature set containing different resolutions, forming multi-scale spatial prior information. This provides precise constraints for super-resolution reconstruction and adapts to complex real-world scenarios. Furthermore, the innovative introduction of channel shifting operations, which group feature channels and spatially shift them along different directions, followed by a padding mechanism to maintain structural integrity, effectively enhances the perception of local features such as edges and textures, providing pixel-level spatial prior guidance for the diffusion model.

[0064] In one embodiment, the first feature modulation layer includes a convolutional network and a normalization module; Figure 4 This is the fourth flowchart illustrating the super-resolution image generation method provided in this embodiment of the invention, as shown below. Figure 4 As shown, in step 104 above, the features corresponding to the random noise and the multi-scale low-resolution spatial features are input into the first feature modulation layer inserted in the first decoder of the image super-resolution model to obtain the denoised features output by the first feature modulation layer after iteration. This can be achieved through the following steps: Step 401: Input the multi-scale low-resolution spatial features into the convolutional network to obtain the affine parameter scaling factor and offset output by the convolutional network.

[0065] For example, the first feature modulation layer first uses multi-scale low-resolution spatial features output by the feature extraction module. As input, multi-scale low-resolution spatial features Input to a convolutional network consisting of several convolutional layers The affine parameter scaling factor for regulation is calculated using a convolutional network. and offset Specifically, it can be expressed by the following formula (1): in, Indicates the different number of layers at multiple scales. This represents the parameters of the convolutional network.

[0066] Step 402: Input the features corresponding to the random noise into the normalization module to obtain the mean and standard deviation of the features corresponding to the random noise output by the normalization module.

[0067] For example, the features corresponding to random noise are normalized using a normalization module. Normalization is performed to obtain the features corresponding to random noise. mean with standard deviation .

[0068] Step 403: Based on the features corresponding to the affine parameter scaling factor, the offset, the mean, the standard deviation, and the random noise, obtain the fused features.

[0069] For example, after obtaining the affine parameter scaling factor, offset, mean, and standard deviation, the offset is then multiplied point-by-point by ⊙. Acting on the mean with standard deviation Simultaneously, the affine parameter scaling factor is superimposed through an addition operation. Finally, the fused features are generated. Specifically, it can be expressed by the following formula (2): This enables adaptive calibration of low-resolution spatial features at multiple scales to the features corresponding to random noise, thereby improving reconstruction quality while preserving visual details of the image.

[0070] Step 404: Iterate the fused features to obtain the denoised features.

[0071] For example, after obtaining the fused features, the fused features are iterated for a preset number of steps to finally obtain the denoised features.

[0072] In this embodiment, the low-resolution spatial features at multiple scales and the features corresponding to random noise are input into the first feature modulation layer composed of a convolutional network and a normalization module, and finally the denoised features are obtained. The spatial prior and latent features are fused by affine transformation, which improves the reconstruction quality while preserving the visual details of the image.

[0073] In one embodiment, Figure 5 This is the fifth flowchart illustrating the super-resolution image generation method provided in this embodiment of the invention, as shown below. Figure 5 As shown, step 105 above inputs the latent features and the initial high-resolution image into the thinner of the image super-resolution model to obtain the super-resolution image output by the thinner. This can be achieved through the following steps: Step 501: Concatenate the latent features and the initial high-resolution image along the channel dimension to obtain a joint feature representation.

[0074] For example, the latent features and the initial high-resolution image are stitched together along the channel dimension to form a joint feature representation that includes information from both.

[0075] Step 502: Input the joint feature representation into the fourth residual network to obtain the residual features output by the fourth residual network.

[0076] For example, the joint feature representation is input into a preset fourth residual network. Through the feature extraction and optimization capabilities of the fourth residual network, the joint feature representation is refined, and finally the residual features are output.

[0077] Step 503: Based on the trade-off parameter, control the contribution ratio of the residual features representing latent features and the initial high-resolution image to obtain the super-resolution image.

[0078] For example, a tradeoff parameter is introduced to control the contribution ratio of the residual features to the latent features and the initial high-resolution image, ultimately generating a refined super-resolution image.

[0079] In this embodiment, latent features and an initial high-resolution image are input into a thinner. The thinner dynamically adjusts the spatial prior extraction strategy to ensure the effectiveness of the spatial prior under different degradation scenarios. This improves the quality of the super-resolution image while further enhancing the fidelity and detail accuracy of the super-resolution image content.

[0080] In one embodiment, the image super-resolution model is trained in the following manner: A first low-resolution image sample is acquired; the first low-resolution image sample is input into the first encoder of the initial image super-resolution model to obtain latent feature samples output by the first encoder; the latent feature samples are input into the initial feature extraction module of the initial image super-resolution model to obtain multi-scale low-resolution spatial feature samples output by the initial feature extraction module; random noise samples are input into the second encoder of the initial image super-resolution model to obtain features corresponding to the random noise samples output by the second encoder; the features corresponding to the random noise samples and the multi-scale low-resolution spatial feature samples are input into the first initial feature modulation layer inserted into the first decoder of the initial image super-resolution model to obtain the denoised output of the first initial feature modulation layer after iterative processing. The denoised feature samples and the corresponding features output from the first encoder are input into the second initial feature modulation layer inserted in the second decoder of the initial image super-resolution model to obtain the initial high-resolution image sample output by the second initial feature modulation layer; the latent feature samples and the initial high-resolution image sample are input into the initial thinner of the initial image super-resolution model to obtain the first reconstructed super-resolution image output by the initial thinner; based on the super-resolution image labels corresponding to the first reconstructed super-resolution image and the first low-resolution image sample, a first loss information is constructed; based on the first loss information, the parameters of the initial feature extraction module and the parameters of the first initial feature modulation layer are iteratively adjusted to obtain the image super-resolution model.

[0081] For example, training is performed using existing publicly available real super-resolution image datasets as labels. A super-resolution framework is constructed based on a pre-trained diffusion model, retaining its ability to generate natural images. Using a first low-resolution image sample as input, all parameters of the VAE encoder, VAE decoder, U-Net encoder, and U-Net decoder in the initial image super-resolution model are fixed. The VAE encoder, VAE decoder, U-Net encoder, and U-Net decoder form the diffusion model, and parameters are updated only for the trainable initial feature extraction module and the first initial feature modulation layer. The optimization objective in this stage is to minimize the mean square error of noise prediction; that is, the first loss information is the mean square error loss. Through training in this stage, the intermediate processing stages of the initial image super-resolution model can be fine-tuned to adapt to the image super-resolution task, enabling the finally trained image super-resolution model to have better low-resolution image feature extraction and modulation capabilities.

[0082] In one embodiment, the step of iteratively adjusting the parameters of the initial feature extraction module and the parameters of the first initial feature modulation layer based on the first loss information to obtain the image super-resolution model can be implemented in the following way: Based on the first loss information, the parameters of the initial feature extraction module and the parameters of the first initial feature modulation layer are iteratively adjusted to obtain a first image super-resolution model; a second low-resolution image sample is obtained; the second low-resolution image sample is input into the first image super-resolution model to obtain a second reconstructed super-resolution image output by the first image super-resolution model; based on the super-resolution image labels corresponding to the second reconstructed super-resolution image and the second low-resolution image sample, a second loss information is constructed; based on the second loss information, the parameters of the second initial feature modulation layer are iteratively adjusted to obtain the image super-resolution model.

[0083] For example, in this round of training, all parameters of the VAE encoder, VAE decoder, U-Net encoder, and U-Net decoder in the initial image super-resolution model are fixed, as are the parameters of the feature extraction module and the first feature modulation layer after the previous training round. Only the parameters of the second feature modulation layer in the VAE decoder are fine-tuned. The core objective of this stage is to improve the visual fidelity of the generated super-resolution image. By introducing multi-scale perceptual loss to optimize the performance of the VAE decoder, and by combining the reconstruction loss with the multi-scale perceptual loss to obtain the second loss information, the finally trained image super-resolution model can generate super-resolution images with more realistic visual effects.

[0084] In one embodiment, the step of iteratively adjusting the parameters of the second initial feature modulation layer based on the second loss information to obtain the image super-resolution model can be implemented in the following way: Based on the second loss information, the parameters of the second initial feature modulation layer are iteratively adjusted to obtain a second image super-resolution model; a third low-resolution image sample is obtained; the third low-resolution image sample is input into the second image super-resolution model to obtain a third reconstructed super-resolution image output by the second image super-resolution model; based on the super-resolution image labels corresponding to the third reconstructed super-resolution image and the third low-resolution image sample, a third loss information is constructed; based on the third loss information, the parameters of the initial thinner are iteratively adjusted to obtain the image super-resolution model.

[0085] For example, in this round of training, all parameters of the VAE encoder, VAE decoder, U-Net encoder, and U-Net decoder in the initial image super-resolution model are fixed, as well as the parameters of the feature extraction module, the first feature modulation layer, and the second feature modulation layer after training. Only the parameters of the thinner are fine-tuned. The third loss information in this stage can be the gradient weight loss, which uses a gradient weighting mechanism to enhance the model's ability to reconstruct high-frequency details of the image. This gradient weight loss can effectively suppress the over-smoothing phenomenon in the image reconstruction process, while preserving the structural information and texture details of the image, thus adapting to the detail enhancement requirements of super-resolution tasks.

[0086] This phased fine-tuning strategy can gradually improve the reconstruction quality of low-resolution images during training, while preserving high-frequency details and maintaining visual fidelity, ultimately meeting the performance requirements of image super-resolution tasks. Furthermore, the phased local fine-tuning strategy significantly reduces computational resource consumption, avoids the loss of pre-training knowledge, and balances training efficiency with reconstruction quality.

[0087] It should be noted that the training dataset used in training the image super-resolution model is a publicly available real-world scene dataset. That is, the aforementioned low-resolution image samples and corresponding super-resolution image labels are all real-world scene data, which can be widely applied in multiple fields such as security monitoring, medical imaging, remote sensing, and high-definition display. For example, it provides clear reconstruction of low-resolution surveillance footage in security scenarios, aiding in accurate identification; it improves the detail resolution of medical images (such as computed tomography (CT) and magnetic resonance imaging (MRI), assisting doctors in diagnosis; at the same time, it can also meet the needs of high-definition display in consumer electronics and efficient analysis of remote sensing data, balancing reconstruction quality and efficiency. It can be directly transformed into software and hardware products or technical services, providing core technical support for enterprises to expand their markets and possessing market demand.

[0088] Figure 6 This is a schematic diagram of the comparative experimental visualization results provided in the embodiments of the present invention, such as... Figure 6As shown, low-resolution images of different scenes are input into the image super-resolution model, feature matching for super-resolution model (FeMaSR), blind image restoration with generative diffusion prior (DiffBIR), and stable super-resolution model (StableSR) of this invention, respectively, to obtain super-resolution images of different scenes output by each model. It can be seen that the super-resolution image reconstructed by the image super-resolution model of this invention has the closest spatial structure to the input low-resolution image and higher visual quality; moreover, compared with the true super-resolution image corresponding to the input low-resolution image, it is closer to the true super-resolution image. Therefore, the image super-resolution model of this invention can achieve good super-resolution results in different scenes.

[0089] It should be noted that, in practical applications, this invention can also input low-resolution synthetic images into a trained image super-resolution model to obtain super-resolution images output by the image super-resolution model, which performs excellently in the reconstruction of both low-resolution synthetic images and low-resolution images of real scenes.

[0090] The super-resolution image generation apparatus provided by the present invention will be described below. The super-resolution image generation apparatus described below can be referred to in correspondence with the super-resolution image generation method described above.

[0091] Figure 7 This is a schematic diagram of the structure of the super-resolution image generation device provided in an embodiment of the present invention, as shown below. Figure 7 As shown, the super-resolution image generation apparatus 700 includes a first encoding unit 701, an extraction unit 702, a second encoding unit 703, a modulation unit 704, and a thinning unit 705; wherein: The first encoding unit 701 is used to input the low-resolution image into the first encoder of the image super-resolution model to obtain the latent features output by the first encoder. Extraction unit 702 is used to input the latent features into the feature extraction module of the image super-resolution model to obtain multi-scale low-resolution spatial features output by the feature extraction module; The second encoding unit 703 is used to input random noise into the second encoder of the image super-resolution model to obtain the features corresponding to the random noise output by the second encoder; The modulation unit 704 is used to input the features corresponding to the random noise and the low-resolution spatial features of the multi-scale into the first feature modulation layer inserted in the first decoder of the image super-resolution model to obtain the denoised features output by the first feature modulation layer after iteration, and input the denoised features and the features output by the corresponding layer in the first encoder into the second feature modulation layer inserted in the second decoder of the image super-resolution model to obtain the initial high-resolution image output by the second feature modulation layer. The thinning unit 705 is used to input the latent features and the initial high-resolution image into the thinner of the image super-resolution model to obtain the super-resolution image output by the thinner.

[0092] The super-resolution image generation apparatus provided by this invention inputs a low-resolution image into the first encoder of an image super-resolution model to obtain latent features. These latent features are then input into the feature extraction module of the image super-resolution model to obtain multi-scale low-resolution spatial features. Random noise is input into the second encoder of the image super-resolution model to obtain features corresponding to the random noise. The features corresponding to the random noise and the multi-scale low-resolution spatial features are input into the first feature modulation layer inserted into the first decoder of the image super-resolution model to obtain denoised features after iterative output. The denoised features and the features output from the corresponding layer in the first encoder are then input into the second feature modulation layer inserted into the second decoder of the image super-resolution model to obtain an initial high-resolution image. Finally, the latent features and the initial high-resolution image are input into the thinner of the image super-resolution model to obtain the super-resolution image output by the thinner. As can be seen, the present invention designs a spatial prior feature extraction module to extract multi-scale low-resolution spatial features from the latent features of low-resolution images, and embeds the multi-scale low-resolution spatial features into the denoising process through the first feature modulation layer to constrain the spatial consistency between the initial high-resolution image and the low-resolution image. Furthermore, the initial high-resolution image is thinned by a thinner to obtain a super-resolution image, thereby improving the quality and fidelity of the generated super-resolution image.

[0093] Based on any of the above embodiments, the first encoding unit 701 is specifically used for: Obtain the original low-resolution image; The original low-resolution image is upsampled to obtain the upsampled image; The upsampled image is input as the low-resolution image into the first encoder to obtain the latent features output by the first encoder.

[0094] Based on any of the above embodiments, the feature extraction module includes a first convolutional layer, a feature enhancement module, a first downsampling module, and a processing module. The feature enhancement module includes a first residual network and a first self-attention layer. The processing module includes a second residual network, a second self-attention layer, and a channel shifting operation module. The extraction unit 702 is specifically used for: The latent features are input into the first convolutional layer to obtain the convolutional features output by the first convolutional layer. The convolutional features are sequentially input into the first residual network and the first self-attention layer of the feature enhancement module to obtain the first latent feature. The first potential feature is input into the first downsampling module to obtain the first feature output by the first downsampling module; The first feature is sequentially input into the second residual network, the second self-attention layer, and the channel displacement operation module of the processing module to obtain the second potential feature; Based on the first latent feature and the second latent feature, the multi-scale low-resolution spatial features are determined.

[0095] Based on any of the above embodiments, the feature extraction module further includes a second downsampling module and a depth feature extraction module, the depth feature extraction module including a third residual network and a shift operation module; the extraction unit 702 is further specifically used for: The second potential feature is input into the second downsampling module to obtain the second feature output by the second downsampling module; The second feature is sequentially input into the third residual network and the displacement operation module of the depth feature extraction module to obtain the third latent feature; Based on the first latent feature, the second latent feature, and the third latent feature, the multi-scale low-resolution spatial features are determined.

[0096] Based on any of the above embodiments, the first feature modulation layer includes a convolutional network and a normalization module; the modulation unit 704 is specifically used for: The multi-scale low-resolution spatial features are input into the convolutional network to obtain the affine parameter scaling factor and offset output by the convolutional network. The features corresponding to the random noise are input into the normalization module to obtain the mean and standard deviation of the features corresponding to the random noise output by the normalization module. Based on the features corresponding to the affine parameter scaling factor, the offset, the mean, the standard deviation, and the random noise, the fused features are obtained; The fused features are iterated to obtain the denoised features.

[0097] Based on any of the above embodiments, the refinement unit 705 is specifically used for: The latent features and the initial high-resolution image are concatenated along the channel dimension to obtain a joint feature representation; The joint feature representation is input into the fourth residual network to obtain the residual features output by the fourth residual network; The super-resolution image is obtained by controlling the contribution ratio of the latent features and the initial high-resolution image in the residual features based on the trade-off parameter.

[0098] Figure 8 This is a schematic diagram of the physical structure of the electronic device provided in the embodiments of the present invention, such as... Figure 8 As shown, the electronic device may include a processor 810, a communications interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communications interface 820, and the memory 830 communicate with each other via the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute a super-resolution image generation method, which includes: inputting a low-resolution image into a first encoder of an image super-resolution model to obtain latent features output by the first encoder; The latent features are input into the feature extraction module of the image super-resolution model to obtain the multi-scale low-resolution spatial features output by the feature extraction module. The random noise is input into the second encoder of the image super-resolution model to obtain the features corresponding to the random noise output by the second encoder; The features corresponding to the random noise and the low-resolution spatial features of the multi-scale are input into the first feature modulation layer inserted in the first decoder of the image super-resolution model to obtain the denoised features output by the first feature modulation layer after iteration. The denoised features and the features output by the corresponding layer in the first encoder are input into the second feature modulation layer inserted in the second decoder of the image super-resolution model to obtain the initial high-resolution image output by the second feature modulation layer. The latent features and the initial high-resolution image are input into the thinner of the image super-resolution model to obtain the super-resolution image output by the thinner.

[0099] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0100] On the other hand, the present invention also provides a computer program product, the computer program product including a computer program, the computer program being able to be stored on a non-transitory computer-readable storage medium, the computer program being executed by a processor, the computer being able to execute the super-resolution image generation method provided by the above methods, the method including: inputting a low-resolution image into a first encoder of an image super-resolution model to obtain latent features output by the first encoder; The latent features are input into the feature extraction module of the image super-resolution model to obtain the multi-scale low-resolution spatial features output by the feature extraction module. The random noise is input into the second encoder of the image super-resolution model to obtain the features corresponding to the random noise output by the second encoder; The features corresponding to the random noise and the low-resolution spatial features of the multi-scale are input into the first feature modulation layer inserted in the first decoder of the image super-resolution model to obtain the denoised features output by the first feature modulation layer after iteration. The denoised features and the features output by the corresponding layer in the first encoder are input into the second feature modulation layer inserted in the second decoder of the image super-resolution model to obtain the initial high-resolution image output by the second feature modulation layer. The latent features and the initial high-resolution image are input into the thinner of the image super-resolution model to obtain the super-resolution image output by the thinner.

[0101] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the super-resolution image generation method provided by the above methods, the method comprising: inputting a low-resolution image into a first encoder of an image super-resolution model to obtain latent features output by the first encoder; The latent features are input into the feature extraction module of the image super-resolution model to obtain the multi-scale low-resolution spatial features output by the feature extraction module. The random noise is input into the second encoder of the image super-resolution model to obtain the features corresponding to the random noise output by the second encoder; The features corresponding to the random noise and the low-resolution spatial features of the multi-scale are input into the first feature modulation layer inserted in the first decoder of the image super-resolution model to obtain the denoised features output by the first feature modulation layer after iteration. The denoised features and the features output by the corresponding layer in the first encoder are input into the second feature modulation layer inserted in the second decoder of the image super-resolution model to obtain the initial high-resolution image output by the second feature modulation layer. The latent features and the initial high-resolution image are input into the thinner of the image super-resolution model to obtain the super-resolution image output by the thinner.

[0102] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0103] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0104] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating super-resolution images, characterized in that, include: The low-resolution image is input into the first encoder of the image super-resolution model to obtain the latent features output by the first encoder. The latent features are input into the feature extraction module of the image super-resolution model to obtain the multi-scale low-resolution spatial features output by the feature extraction module. The random noise is input into the second encoder of the image super-resolution model to obtain the features corresponding to the random noise output by the second encoder; The features corresponding to the random noise and the low-resolution spatial features of the multi-scale are input into the first feature modulation layer inserted in the first decoder of the image super-resolution model to obtain the denoised features output by the first feature modulation layer after iteration. The denoised features and the features output by the corresponding layer in the first encoder are input into the second feature modulation layer inserted in the second decoder of the image super-resolution model to obtain the initial high-resolution image output by the second feature modulation layer. The latent features and the initial high-resolution image are input into the thinner of the image super-resolution model to obtain the super-resolution image output by the thinner.

2. The super-resolution image generation method according to claim 1, characterized in that, The process of inputting a low-resolution image into the first encoder of the image super-resolution model to obtain the latent features output by the first encoder includes: Obtain the original low-resolution image; The original low-resolution image is upsampled to obtain the upsampled image; The upsampled image is input as the low-resolution image into the first encoder to obtain the latent features output by the first encoder.

3. The super-resolution image generation method according to claim 1, characterized in that, The feature extraction module includes a first convolutional layer, a feature enhancement module, a first downsampling module, and a processing module. The feature enhancement module includes a first residual network and a first self-attention layer. The processing module includes a second residual network, a second self-attention layer, and a channel shifting operation module. The latent features are input into the feature extraction module of the image super-resolution model to obtain multi-scale low-resolution spatial features output by the feature extraction module, including: The latent features are input into the first convolutional layer to obtain the convolutional features output by the first convolutional layer. The convolutional features are sequentially input into the first residual network and the first self-attention layer of the feature enhancement module to obtain the first latent feature. The first potential feature is input into the first downsampling module to obtain the first feature output by the first downsampling module; The first feature is sequentially input into the second residual network, the second self-attention layer, and the channel displacement operation module of the processing module to obtain the second potential feature; Based on the first latent feature and the second latent feature, the multi-scale low-resolution spatial features are determined.

4. The super-resolution image generation method according to claim 3, characterized in that, The feature extraction module further includes a second downsampling module and a depth feature extraction module, wherein the depth feature extraction module includes a third residual network and a displacement operation module; The determination of the multi-scale low-resolution spatial features based on the first latent feature and the second latent feature includes: The second potential feature is input into the second downsampling module to obtain the second feature output by the second downsampling module; The second feature is sequentially input into the third residual network and the displacement operation module of the depth feature extraction module to obtain the third latent feature; Based on the first latent feature, the second latent feature, and the third latent feature, the multi-scale low-resolution spatial features are determined.

5. The super-resolution image generation method according to any one of claims 1-4, characterized in that, The first feature modulation layer includes a convolutional network and a normalization module; The step of inputting the features corresponding to the random noise and the multi-scale low-resolution spatial features into the first feature modulation layer inserted in the first decoder of the image super-resolution model to obtain the denoised features output by the first feature modulation layer after iterative processing includes: The multi-scale low-resolution spatial features are input into the convolutional network to obtain the affine parameter scaling factor and offset output by the convolutional network. The features corresponding to the random noise are input into the normalization module to obtain the mean and standard deviation of the features corresponding to the random noise output by the normalization module. Based on the features corresponding to the affine parameter scaling factor, the offset, the mean, the standard deviation, and the random noise, the fused features are obtained; The fused features are iterated to obtain the denoised features.

6. The super-resolution image generation method according to any one of claims 1-4, characterized in that, The step of inputting the latent features and the initial high-resolution image into the thinner of the image super-resolution model to obtain the super-resolution image output by the thinner includes: The latent features and the initial high-resolution image are concatenated along the channel dimension to obtain a joint feature representation; The joint feature representation is input into the fourth residual network to obtain the residual features output by the fourth residual network; The super-resolution image is obtained by controlling the contribution ratio of the latent features and the initial high-resolution image in the residual features based on the trade-off parameter.

7. A super-resolution image generation apparatus, characterized in that, include: The first encoding unit is used to input the low-resolution image into the first encoder of the image super-resolution model to obtain the latent features output by the first encoder. An extraction unit is used to input the latent features into the feature extraction module of the image super-resolution model to obtain multi-scale low-resolution spatial features output by the feature extraction module; The second encoding unit is used to input random noise into the second encoder of the image super-resolution model to obtain the features corresponding to the random noise output by the second encoder; The modulation unit is used to input the features corresponding to the random noise and the low-resolution spatial features of the multi-scale into the first feature modulation layer inserted in the first decoder of the image super-resolution model to obtain the denoised features output by the first feature modulation layer after iteration, and input the denoised features and the features output by the corresponding layer in the first encoder into the second feature modulation layer inserted in the second decoder of the image super-resolution model to obtain the initial high-resolution image output by the second feature modulation layer. A thinning unit is used to input the latent features and the initial high-resolution image into the thinner of the image super-resolution model to obtain the super-resolution image output by the thinner.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the super-resolution image generation method as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the super-resolution image generation method as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the super-resolution image generation method as described in any one of claims 1 to 6.

Images