Real scene image super-resolution method and system based on mean-shift diffusion

Abstract

The present invention belongs to the field of image super-resolution and provides a real-scene image super-resolution method and system based on mean-shift diffusion. A low-resolution image of a real scene is acquired, and a pre-trained variational autoencoder is used to project the low-resolution image into a latent space to obtain the latent features of the low-resolution image. Noise is added to the latent features of the low-resolution image to obtain an initial diffusion state. Based on the initial diffusion state, a pre-trained stable diffusion model is used to iteratively perform denoising, and the latent features of the low-resolution image are used as control signals to guide each iteration to obtain a super-resolution reconstructed image. The present invention proposes a mean-shift diffusion process to achieve consistency between training and testing. Based on the proposed mean-shift diffusion process, the present invention designs a corresponding training strategy and inference method, and uses this training strategy to fine-tune the existing stable diffusion model.

Description

Technical Field

[0001] The present invention belongs to the technical field of image super-resolution, and in particular relates to a real scene image super-resolution method and system based on mean shift diffusion. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Image super-resolution (SR) aims to restore high-resolution (HR) images from degraded low-resolution (LR) image observations. Starting with SRCNN, deep learning-based ISR has become popular. Various methods focusing on model design have been proposed to improve the accuracy of ISR reconstruction and achieve satisfactory performance on synthetic data with the same degradation. However, these SR methods have strict assumptions on the degradation process, such as bicubic downsampling and blurring with known parameters, which limits their effectiveness in dealing with complex and unknown degradations in the real world. Therefore, real-world super-resolution has gradually become a popular research direction in recent years.

[0004] Real-world image super-resolution (Real-ISR) aims to restore high-quality high-resolution versions of real-world low-resolution images with unknown and complex degradation. Some researchers have addressed this task by synthesizing low-resolution (LR)-to-high-resolution (HR) image pairs similar to real-world data. Specifically, BSRGAN employs a random shuffle degradation modeling strategy, while Real-ESRGAN employs a high-order degradation modeling procedure. Using training samples with more realistic degradation, both BSRGAN and Real-ESRGAN leverage GANs to reconstruct the desired HR images, achieving superior visual perception. However, these GAN-based methods often produce unnatural artifacts. Recent research based on diffusion models has further demonstrated superior performance in real-world image SR. Benefiting from the strong generative priors of pre-trained text-to-image (T2I) diffusion models, such as Stable Diffusion (SD), Real-ISR methods based on pre-trained SD models often generate numerous visually appealing details. StableSR, for the first time, uses SD to achieve Real-ISR and demonstrates impressive detail generation capabilities. DiffBIR utilizes ControlNet to provide appropriate control signals for SD, improving the model's generation performance. However, StableSR and DiffBIR only use images as control conditions and neither considers the role of semantic information in pre-trained T2I models. Therefore, PASD extracts low-level and high-level features from LR images and feeds them into a pre-trained SD model with a pixel-aware cross-attention module. SeeSR further explores ways to incorporate semantic information, using robust label-based text cues and leveraging soft cues to guide the diffusion process, allowing the model to produce more realistic texture details.

[0005] In other words, existing super-resolution methods based on pre-trained SD models suffer from the problem of inconsistency between training and testing. This inconsistency stems from the fact that the pre-trained stable diffusion model encodes high-resolution and low-resolution image pairs into the latent space, resulting in a large mean difference. This makes it difficult for models trained on noisy high-resolution images to adapt to the use of noisy low-resolution images as the diffusion starting point during testing. These methods use the latent features of HR images with added noise as input during the training phase, and use the latent features of LR images as input during the testing phase. However, after the pre-trained SD model encodes paired HR and LR images into the latent space, there will be a significant difference in the feature means between the two, which leads to obvious inconsistencies between training and testing of existing methods. Summary of the Invention

[0006] To address these issues, the present invention proposes a novel mean-shift diffusion process. This process shifts the distribution mean from the LR image to the HR image by replacing the mean of the Gaussian noise in the traditional diffusion process with a weight coefficient multiplied by the residual between the LR image and the HR image. This allows the reverse diffusion process during inference to naturally start from the noisy LR image and recover the corresponding high-quality HR image. This mean-shift diffusion supports using the noisy LR image as the diffusion starting point. Therefore, during testing, a high-quality, high-resolution image can be recovered through an iterative denoising process, simply using the latent features of the noisy LR image as the model input.

[0007] According to some embodiments, a first solution of the present invention provides a real scene image super-resolution method based on mean-shift diffusion, which adopts the following technical solutions:

[0008] A real scene image super-resolution method based on mean-shift diffusion, comprising:

[0009] Obtain low-resolution images from real scenes, and use pre-trained variational autoencoders to project the low-resolution images into the latent space to obtain the latent features of the low-resolution images;

[0010] Add noise to the latent features of the low-resolution image to obtain the initial diffusion state;

[0011] Based on the initial diffusion state, the pre-trained stable diffusion model is used iteratively for denoising, and the potential features of the low-resolution image are used as control signals to guide each round of iteration to obtain a super-resolution reconstructed image.

[0012] Among them, the mean shift diffusion process is used to determine the training strategy and stable diffusion model of the stable diffusion model.

[0013] Furthermore, the training strategy of the stable diffusion model is specifically as follows:

[0014] Acquire a high-resolution image and randomly degrade the high-resolution image to obtain a low-resolution image;

[0015] Use pre-trained variational autoencoders to obtain high-resolution image latent features and low-resolution image latent features;

[0016] Calculate the difference between the low-resolution latent features and the high-resolution image latent features as the residual;

[0017] Multiply the residual by the weight factor and add standard Gaussian noise to obtain the target noise;

[0018] The product of the target noise and the scale parameter is added to the potential features of the high-resolution image under a certain scale parameter to obtain the current state distribution;

[0019] The current state distribution is input into the noise prediction network model to predict the target noise, and the difference between the predicted target noise and the target noise is used as the loss to obtain the final trained noise prediction network model;

[0020] The final trained noise prediction network model is used as the pre-trained stable diffusion model.

[0021] Furthermore, the noise prediction network model is used to obtain the estimation of the potential features of the high-resolution image, specifically:

[0022]

[0023] Among them, the noise prediction network model ∈ θ (z t ,z lr ,t),z θ (z t ,z lr ,t) is the estimate of the potential features of the high-resolution image, and the distribution z at the t-th time step t , t=1,2,3,……T, T is an integer, z lr is the potential feature of the low-resolution image, α t is the scale parameter in the SD diffusion process, and θ is the model parameter.

[0024] Furthermore, the mean shift diffusion process includes a mean shift diffusion forward process and a mean shift diffusion reverse process, wherein:

[0025] Mean-shift diffusion forward process: The mean of the Gaussian noise is replaced by the residual between the low-resolution image latent feature and the original high-resolution image latent feature, and scaled according to the weight factor so that the distribution mean is transferred from the high-resolution image latent feature to the low-resolution image latent feature;

[0026] The reverse process of mean-shift diffusion is to replace the mean of Gaussian noise with the residual between the low-resolution image latent features and the high-resolution image latent features, and scale them according to the weight factor so that the distribution mean is transformed from the low-resolution image latent features to the high-resolution latent features.

[0027] Furthermore, in the forward process of mean transfer diffusion, the distribution z of each time step t is t satisfy:

[0028]

[0029] Among them, z hr is the high-resolution image potential feature, z lr is the potential feature of the low-resolution image, α t and βt is the scale parameter in the SD diffusion process, satisfying α t ,β t >0 and w t Represents a time-dependent weight curve.

[0030] Furthermore, the reverse process of mean shift diffusion is as follows:

[0031] p θ (z t-1 |z t ,z lr )=α t-1 z θ +(w t-1 β t-1 -w t η t )(z lr -z θ )+η t ∈′ θ +σ t ∈1;

[0032] in, z θ =z θ (z t ,z lr ,t),∈′ θ =∈′ θ (z t ,z lr ,t), And σ t is a variable parameter, z lr is the potential feature of the low-resolution image, the distribution z at the t-th time step t , the distribution z at the t-1th time step t-1 , α t-1 and β t-1 is the scale parameter in the SD diffusion process, satisfying α t-1 ,β t-1 >0 and w t Represents a time-dependent weight curve; noise prediction network model ∈ ′ θ (z t ,z lr ,t),z′ θ (z t ,z lr ,t) as the high-resolution image potential feature z hr Estimate of ∈′ θ is the target noise, ∈1 is the randomly sampled standard Gaussian noise, and θ is the model parameter.

[0033] Furthermore, a time-dependent weight curve w t , specifically:

[0034]

[0035] Among them, γ is a hyperparameter that controls the speed of the distribution mean shift, and T is the time step.

[0036] According to some embodiments, a second solution of the present invention provides a real scene image super-resolution system based on mean-shift diffusion, which adopts the following technical solutions:

[0037] Real-scene image super-resolution system based on mean-shift diffusion, including:

[0038] The latent feature acquisition module is configured to acquire low-resolution images in real scenes and use a pre-trained variational autoencoder to project the low-resolution images into a latent space to obtain latent features of the low-resolution images;

[0039] an initial diffusion state determination module, configured to add noise to the latent features of the low-resolution image to obtain an initial diffusion state;

[0040] The image super-resolution module is configured to iteratively use a pre-trained stable diffusion model to perform denoising based on the starting diffusion state, and use the latent features of the low-resolution image as a control signal to guide each iteration to obtain a super-resolution reconstructed image;

[0041] Among them, the mean shift diffusion process is used to determine the training strategy and stable diffusion model of the stable diffusion model.

[0042] According to some embodiments, a third aspect of the present invention provides a computer-readable storage medium.

[0043] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the real scene image super-resolution method based on mean-shift diffusion as described in the first aspect above.

[0044] According to some embodiments, a fourth aspect of the present invention provides a computer device.

[0045] A computer device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the real scene image super-resolution method based on mean-shift diffusion as described in the first aspect above are implemented.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] The present invention proposes a mean-shift diffusion process in which the distribution mean is transferred from the HR image to the LR image in the forward process, so that the reverse diffusion process during inference can naturally start from the noisy LR image and recover the corresponding high-quality HR image. This solves the training-test inconsistency of existing methods, allowing the model to more effectively utilize the trained knowledge during testing and achieve better performance.

[0048] Based on this mean-shift diffusion process, this paper proposes a comprehensive training strategy and a mean-shift sampler for inference. This strategy can be applied to all Real-ISR methods based on pre-trained SD, demonstrating its versatility. After fine-tuning using this diffusion process, the model can achieve better performance with one-fifth the number of sampling steps. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0050] Figure 1 This is a flow chart of a real scene image super-resolution method based on mean-shift diffusion in an embodiment of the present invention;

[0051] Figure 2 Schematic diagram of the change in pixel space of the mean of the HR and LR image pairs during the forward diffusion process described in an embodiment of the present invention;

[0052] Figure 3 Schematic diagram of the change of the mean of the HR and LR image pairs in the latent space during the forward diffusion process described in an embodiment of the present invention;

[0053] Figure 4 Schematic diagram of the distribution change of the mean shift diffusion process in the latent space in an embodiment of the present invention;

[0054] Figure 5 is the weight curve w in the embodiment of the present invention t Schematic diagram;

[0055] Figure 6 1 is a schematic diagram of the mean shift diffusion training process in an embodiment of the present invention;

[0056] Figure 7 2. It is a schematic diagram of the mean shift diffusion inference stage in an embodiment of the present invention;

[0057] Figure 8 It is a schematic diagram comparing the results of the method according to the embodiment of the present invention and other existing methods on the Real dataset. DETAILED DESCRIPTION

[0058] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0059] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0060] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0061] In the absence of conflict, the embodiments of the present invention and the features thereof may be combined with each other.

[0062] Example 1

[0063] like Figure 1 As shown, this embodiment provides a real scene image super-resolution method based on mean shift diffusion. This embodiment uses the method applied to the server as an example for illustration. It can be understood that the method can also be applied to the terminal, and can also be applied to a system including the terminal, the server, and the system, and is implemented through the interaction between the terminal and the server. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network servers, cloud communications, middleware services, domain name services, security services CDN, and big data and artificial intelligence platforms. The terminal can be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected by wired or wireless communication, and this application does not limit this. In this embodiment, the method includes the following steps:

[0064] Obtain low-resolution images from real scenes, and use pre-trained variational autoencoders to project the low-resolution images into the latent space to obtain the latent features of the low-resolution images;

[0065] Add noise to the latent features of the low-resolution image to obtain the initial diffusion state;

[0066] Based on the initial diffusion state, the pre-trained stable diffusion model is used iteratively for denoising, and the potential features of the low-resolution image are used as control signals to guide each round of iteration to obtain a super-resolution reconstructed image.

[0067] Among them, the mean shift diffusion process is used to determine the training strategy and stable diffusion model of the stable diffusion model.

[0068] The mean shift diffusion process includes a mean shift diffusion forward process and a mean shift diffusion reverse process, wherein,

[0069] Mean-shift diffusion forward process: The mean of the Gaussian noise is replaced by the residual between the low-resolution image latent feature and the original high-resolution image latent feature, and scaled according to the weight factor so that the distribution mean is transferred from the high-resolution image latent feature to the low-resolution image latent feature;

[0070] First, the diffusion process needs to start from the noisy LR latent features, which requires that the mean of the distribution is converted from the original HR latent features to the LR latent features during the diffusion process. According to the original diffusion process in stable diffusion (SD), at step t, the current state z t0 satisfy:

[0071] z t0 =α t z hr +β t ∈,t=1,2,…,T(1);

[0072] Among them, α t and β t is the scale parameter in the SD diffusion process, satisfying α t ,β t >0 and And z hr In the Real-ISR task, it is the HR latent feature. In the traditional diffusion process, the noise ∈ obeys the standard Gaussian distribution, and its mean is completely determined by α t z hr Since the noise in the original diffusion process does not change the mean, the mean is introduced as w t (z lr -z hr ) is Gaussian noise ∈′, rather than standard Gaussian noise ∈, to achieve the transfer of distribution mean. lr represents the LR potential feature, w t Represents a time-dependent weight curve.

[0073] Transformed distribution z t It can be expressed as follows:

[0074] z t =α t z hr +β t ∈′=α tz hr +w t β t (z lr -z hr )+β t ∈(2);

[0075] in, Obviously, when When z t can be viewed as LR latent features plus standard Gaussian noise, which is easy to obtain during inference. t = 0, the mean shift diffusion process is converted to the traditional diffusion process, and the mean of the distribution is the HR potential feature. The weight curve w designed by the present invention is t The form is as follows:

[0076]

[0077] Among them, γ is a hyperparameter that controls the speed of the mean shift. Finally, in the forward process of mean shift diffusion, the distribution z at each time step t is t satisfy:

[0078]

[0079] Based on this forward process, a new stable diffusion SD training strategy is designed for the Real-ISR task. First, the HR image latent features and LR image latent features are obtained by pre-training the VAE encoder (Variational Autoencoder, VAE), which are denoted as z and z respectively. hr and z lr Then, calculate z hr and z lr The residual is denoted as z res =z lr -z hr . Then, z res Multiply by the weight factor w t , and add the randomly sampled standard Gaussian noise ∈ to get the target noise ∈′. Finally, β t ∈′ added to α t z hr In the current distribution z t , and input it into the noise prediction network model to predict the target noise. It can be understood that the noise prediction network model includes but is not limited to using the Unet network model.

[0080] The optimization goal of the training phase is:

[0081]

[0082] Based on the loss in the above formula (5), the general stable diffusion model is fine-tuned to obtain the final trained noise prediction network model, and the final trained noise prediction network model is used as the pre-trained stable diffusion model in the method described in this example.

[0083] Mean shift diffusion inverse process: replace the mean of Gaussian noise with the residual between the low-resolution image latent features and the high-resolution image latent features, and scale them according to the weight factor so that the distribution mean is transferred from the low-resolution image latent features to the high-resolution latent features;

[0084] After obtaining the mean shift forward diffusion process and its corresponding training strategy, the corresponding mean shift sampler is still needed in the inference stage. The forward diffusion process is regarded as a non-Markov chain similar to DDIM, so that the distribution z t-1 It can be considered as the variance and the mean is z t ,z hr With z lr The normal distribution of the linear combination of . The formula is as follows:

[0085]

[0086] Among them, k t ,λ t and p t is the linear weighting coefficient, is the distribution z at each time step t-1 t-1 The estimated value of , the distribution z at the t-th time step t , the distribution z at the t-1th time step t-1 By using the method of undetermined coefficients, we can solve for k t ,λ t and p t , its solution is expressed as follows:

[0087]

[0088] Considering that z cannot be obtained in practical applications hr , so a noise prediction network model ∈′ can be trained θ (z t ,z lr ,t) to obtain z′ θ (z t ,z lr ,t) as z hr The estimates are as follows:

[0089]

[0090] By combining the above formulas, the mean shift diffusion inverse process can be expressed as:

[0091] p θ (z t-1 |z t ,z lr )=α t-1 z θ +(w t-1 β t-1 -w t η t )(z lr -z θ )+η t ∈′ θ +σ t ∈1(9);

[0092] in, z θ =z θ (z t ,z lr ,t),∈′ θ =∈′ θ (z t ,z lr ,t), And σ t is a variable parameter; the current state z at step t-1 during the diffusion process t-1 , α t-1 and β t-1 is the scale parameter in the SD diffusion process, satisfying α t-1 ,β t-1 >0 and w t surface

[0093] shows a time-dependent weight curve; noise prediction network model ∈′ θ (z t ,z lr ,y),z′ θ (z t ,z lr ,t) as the high-resolution image potential feature z ht Estimate of ∈′ θ is the target noise, ∈1 is the randomly sampled standard Gaussian noise, and θ is the model parameter.

[0094] Finally, this embodiment customizes the mean shift sampler for inference based on the reverse process expression. In practical applications, select σ t = 0 as the mean shift sampler in the inference process, that is, the DDIM version of the sampler.

[0095] In the inference stage, the present invention uses the latent features of the LR image to add noise as the diffusion starting point, and iteratively uses a sampler for denoising, ultimately obtaining a high-quality super-resolution result.

[0096] In the experiment, the present invention uses the most advanced real-scene super-resolution model SeeSR as the basic model architecture, and uses only 10 sampling steps to surpass the effect of the original SeeSR model with 50 sampling steps, achieving significant improvements in inference speed and result quality.

[0097] The mean-shift diffusion process shifts the mean of the distribution from high-resolution image features to low-resolution image features during the forward diffusion process, and shifts the mean of the distribution from low-resolution image features to high-resolution image features during the reverse diffusion process. In traditional diffusion processes, the mean of the distribution always remains at the high-resolution image. This gradually shifts the mean of the distribution from high-resolution image features to low-resolution image features during the forward diffusion process to address the training-test inconsistency of existing methods. High-quality image super-resolution is achieved by leveraging the generative capabilities of a pre-trained stable diffusion model, using the latent features of noisy low-resolution images as the diffusion starting point to restore details in the high-resolution image during the diffusion process. The mean-shift diffusion process is used to fine-tune the state-of-the-art real-scene image super-resolution model, significantly improving both inference speed and super-resolution quality.

[0098] Example 1: Mean-Shift Diffusion

[0099] like Figure 2 and Figure 3 As shown in , the HR and LR image pairs have similar means in the pixel space, but when they are encoded into the latent space, a significant mean difference appears. As the forward diffusion process continues to add noise, this mean difference gradually narrows but remains significant. Therefore, the present invention proposes a mean-shift diffusion method such as Figure 4As shown in Figure 2, after being projected into the latent space, the high-resolution (HR) distribution (blue area) and the low-resolution (LR) distribution (orange area) show significant differences in distribution. In the traditional forward diffusion process, although the HR and LR distributions gradually approach each other, they still maintain a significant distance due to the difference in mean, and eventually fall into the green and red areas, respectively. Since the current Real-ISR models based on diffusion models follow the traditional diffusion process and predict zero-mean standard Gaussian noise, they have difficulty in recovering the HR distribution from the noisy LR distribution during the backward diffusion process, and eventually move away from the purple area of ​​the HR distribution. In this study, we replace the mean of the Gaussian noise with the residual between the LR latent features and the HR latent features, and scale it by a weight factor. This strategy enables the distribution mean to naturally transition from the LR latent features to the HR latent features during the backward diffusion process. It is called the mean-shifted diffusion process. Under this setting, the backward diffusion process restores the noisy LR distribution to a position closer to the HR distribution (such as Figure 4 In practice, the Figure 5 The weight curve w shown t , where the hyperparameter γ = 50.

[0100] Example 2: Experimental Setup

[0101] The present invention is trained on the top 10K face images from DIV2K, DIV8K, Flickr2K, OST, and FFHQ. To synthesize low-quality images paired with high-quality images, the images in the dataset are degraded using the RealESRGAN degradation process and randomly cropped into 512×512 image blocks. For the network architecture, this embodiment uses the same network as the SeeSR model and fine-tunes the model weights based on the publicly released SeeSR model. The model is fine-tuned using the Adam optimizer for 50k iterations, with a batch size of 24 and a learning rate of 5×10 -6 , for training.

[0102] Example 3: Diffusion Model Training

[0103] 1. Model Architecture:

[0104] -Use an image restoration network based on a diffusion model, which is pre-trained on a large-scale dataset and accumulates rich prior knowledge.

[0105] 2. Training process:

[0106] -Using the images in the dataset as HR images, the two-stage degradation process proposed by RealESRGAN is used to degrade the images to obtain LR versions.

[0107] -Training process such as Figure 6As shown, the potential representations of HR images and LR images are obtained by pre-training the VAE encoder, which are denoted as z hr and z lr . Use z lr Input as control signal to ControlNet. Calculate z hr and z lr The residual is denoted as z res =z lr -z hr . Then, z res Multiply by the weight factor w t , and add the randomly sampled standard Gaussian noise ∈ to get the target noise ∈′. Finally, β t ∈′ added to α t z hr In the current distribution z t , and input it into the noise prediction network model to predict the target noise. The loss of formula (5) is used between the output of the noise prediction network model and the target noise to obtain the final noise prediction network model, which is used as the pre-trained stable diffusion model.

[0108] The pre-trained stable diffusion model, through pre-training on a large-scale dataset, accumulates rich prior knowledge to enhance the model's ability to generate and restore image details. Using the mean-shift diffusion model training strategy, the state-of-the-art real-scene image super-resolution model based on the stable diffusion model is fine-tuned. Using noisy low-resolution image features as the diffusion starting point, the proposed inference sampler implements the denoising process, gradually restoring high-resolution image details.

[0109] Example 4: Image Restoration

[0110] 1. Input processing:

[0111] - Take the LR image as the overall input and project it into the latent space through the VAE encoder to obtain z lr , add T time step noise to it, and get the initial diffusion state z T , t=1,2,3,……T, T is an integer. Use z lr As a control signal input to ControlNet to guide the denoising process, Figure 7 shown.

[0112] 2. Recovery process:

[0113] -The diffusion model gradually recovers high-frequency details in the image. In each iteration, the LR image and the extracted semantic information are used as control signals to guide the recovery process.

[0114] 3. Output generation:

[0115] -Finally, a high-definition HR image is generated. The content in the restored image is consistent with the LR image, and the degradation in the LR image is removed, with rich details and high quality.

[0116] Example 5: Experimental Verification

[0117] 1. Experimental setup:

[0118] -The DIV2K validation set and the RealSR, DRealSR, and LR200 datasets were selected as test datasets to compare the restoration effects of different methods.

[0119] - Use PSNR, SSIM, LPIPS and FID to measure image fidelity. For non-reference metrics, use CLIPIQ, MUSIQ, MANIQA, TOPIQ and QALIGN to measure image quality.

[0120] 2. Experimental results:

[0121] -Compared with the traditional SD-based Real-ISR method, the method of the present invention has significant improvements in reference-free indicators such as CLIPIQ, MUSIQ, MANIQA, TOPIQ and QALIGN.

[0122] -like Figure 8 As shown in the figure, the restored image shows excellent performance in terms of high-frequency detail generation ability and image content fidelity, especially on severely degraded images.

[0123] This embodiment aims to restore perceptually convincing high-resolution images from low-resolution images with complex and diverse degradations found in the real world. Many methods leverage the generative power of pre-trained stable diffusion models to achieve super-resolution, resulting in better super-resolution results. However, due to the discrepancy between training and testing, these methods are prone to producing blurry super-resolution results. To address this issue, this embodiment proposes a mean-shifted diffusion process to achieve consistency between model training and testing. Based on this diffusion process, a real-scene image super-resolution method is proposed, capable of achieving high-quality image super-resolution with a small number of diffusion steps. First, the limitations of existing methods based on pre-trained stable diffusion models are analyzed, revealing the inconsistency between training and testing. A mean-shifted diffusion process is then proposed to achieve consistency between training and testing. Based on the proposed mean-shifted diffusion process, a corresponding training strategy and inference method are designed, and the existing model is fine-tuned using this training strategy. Experiments demonstrate that the proposed method can achieve higher-quality image super-resolution results with less inference time. This invention provides new ideas and methods for the field of real-scene image super-resolution.

[0124] Example 2

[0125] This embodiment provides a real scene image super-resolution system based on mean-shift diffusion, including:

[0126] The latent feature acquisition module is configured to acquire low-resolution images in real scenes and use a pre-trained variational autoencoder to project the low-resolution images into a latent space to obtain latent features of the low-resolution images;

[0127] an initial diffusion state determination module, configured to add noise to the latent features of the low-resolution image to obtain an initial diffusion state;

[0128] The image super-resolution module is configured to iteratively use a pre-trained stable diffusion model to perform denoising based on the starting diffusion state, and use the latent features of the low-resolution image as a control signal to guide each iteration to obtain a super-resolution reconstructed image;

[0129] Among them, the mean shift diffusion process is used to determine the training strategy and stable diffusion model of the stable diffusion model.

[0130] The examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the contents disclosed in the above embodiment 1. It should be noted that the above modules as part of the system can be executed in a computer system such as a set of computer executable instructions.

[0131] The description of each embodiment in the above embodiments has different emphases. For parts not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0132] The proposed system can be implemented in other ways. For example, the system embodiment described above is merely illustrative. For example, the above module division is only a logical function division. In actual implementation, other division methods may be used. For example, multiple modules can be combined or integrated into another system, or some features can be ignored or not implemented.

[0133] Example 3

[0134] This embodiment provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the steps of the real scene image super-resolution method based on mean-shift diffusion as described in the first embodiment are implemented.

[0135] Example 4

[0136] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of the real scene image super-resolution method based on mean-shift diffusion as described in the first embodiment are implemented.

[0137] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage and optical storage, etc.) containing computer-usable program code.

[0138] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0139] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0140] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0141] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware through a computer program. The program can be stored in a computer-readable storage medium, and when executed, the program can include the processes in the above-described method embodiments. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0142] Although the above describes the specific embodiments of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without any creative work are still within the scope of protection of the present invention.

Claims

1. Real scene image super-resolution method based on mean-shift diffusion, characterized by: include: Obtain low-resolution images from real scenes, and use pre-trained variational autoencoders to project the low-resolution images into the latent space to obtain the latent features of the low-resolution images; Add noise to the latent features of the low-resolution image to obtain the initial diffusion state; Based on the initial diffusion state, the pre-trained stable diffusion model is used iteratively for denoising, and the potential features of the low-resolution image are used as control signals to guide each round of iteration to obtain a super-resolution reconstructed image. Among them, the training strategy and stable diffusion model of the stable diffusion model are determined by using the mean shift diffusion process; The training strategy of the stable diffusion model is specifically as follows: Acquire a high-resolution image and randomly degrade the high-resolution image to obtain a low-resolution image; Use pre-trained variational autoencoders to obtain high-resolution image latent features and low-resolution image latent features; Calculate the difference between the low-resolution latent features and the high-resolution image latent features as the residual; Multiply the residual by the weight factor and add standard Gaussian noise to obtain the target noise; The product of the target noise and the scale parameter is added to the potential features of the high-resolution image under a certain scale parameter to obtain the current state distribution; The current state distribution is input into the noise prediction network model to predict the target noise, and the difference between the predicted target noise and the target noise is used as the loss to obtain the final trained noise prediction network model; The final trained noise prediction network model is used as the pre-trained stable diffusion model; The mean shift diffusion process includes a mean shift diffusion forward process and a mean shift diffusion reverse process, wherein, Mean-shift diffusion forward process: The mean of the Gaussian noise is replaced by the residual between the low-resolution image latent feature and the original high-resolution image latent feature, and scaled according to the weight factor so that the distribution mean is transferred from the high-resolution image latent feature to the low-resolution image latent feature; The reverse process of mean-shift diffusion is to replace the mean of Gaussian noise with the residual between the low-resolution image latent features and the high-resolution image latent features, and scale them according to the weight factor so that the distribution mean is transformed from the low-resolution image latent features to the high-resolution latent features.

2. The real scene image super-resolution method based on mean-shift diffusion according to claim 1, characterized in that: The noise prediction network model is used to obtain an estimate of the potential features of the high-resolution image, specifically: Among them, the noise prediction network model , is the estimation of the latent features of the high-resolution image, The distribution of time steps , , is the potential feature of the low-resolution image, is the scale parameter in the SD diffusion process, is the model parameter 。 3. The real scene image super-resolution method based on mean-shift diffusion according to claim 1, characterized in that The mean transfer diffusion forward process, each time step Distribution satisfy: ； in, is the high-resolution image potential feature, is the potential feature of the low-resolution image, and is the scale parameter in the SD diffusion process, satisfying and , , Represents a time-dependent weight curve.

4. The real scene image super-resolution method based on mean-shift diffusion according to claim 1, characterized in that The reverse process of mean shift diffusion is as follows: ； in, and For variable parameters, is the potential feature of low-resolution image The distribution of time steps , No. The distribution of time steps , and is the scale parameter in the SD diffusion process, satisfying and , , Represents a time-dependent weight curve; noise prediction network model , As a latent feature of high-resolution images Estimates, is the target noise, is a randomly sampled standard Gaussian noise, are model parameters; A time-dependent weight curve , specifically: in, is a hyperparameter that controls the speed of distribution mean shift, is the time step.

5. Real scene image super-resolution system based on mean-shift diffusion, characterized by: include: The latent feature acquisition module is configured to acquire low-resolution images in real scenes and use a pre-trained variational autoencoder to project the low-resolution images into a latent space to obtain latent features of the low-resolution images; an initial diffusion state determination module, configured to add noise to the latent features of the low-resolution image to obtain an initial diffusion state; The image super-resolution module is configured to iteratively use a pre-trained stable diffusion model to perform denoising based on the starting diffusion state, and use the latent features of the low-resolution image as a control signal to guide each iteration to obtain a super-resolution reconstructed image; Among them, the training strategy and stable diffusion model of the stable diffusion model are determined by using the mean shift diffusion process; The training strategy of the stable diffusion model is specifically as follows: Acquire a high-resolution image and randomly degrade the high-resolution image to obtain a low-resolution image; Use pre-trained variational autoencoders to obtain high-resolution image latent features and low-resolution image latent features; Calculate the difference between the low-resolution latent features and the high-resolution image latent features as the residual; Multiply the residual by the weight factor and add standard Gaussian noise to obtain the target noise; The product of the target noise and the scale parameter is added to the potential features of the high-resolution image under a certain scale parameter to obtain the current state distribution; The current state distribution is input into the noise prediction network model to predict the target noise, and the difference between the predicted target noise and the target noise is used as the loss to obtain the final trained noise prediction network model; The final trained noise prediction network model is used as the pre-trained stable diffusion model; The mean shift diffusion process includes a mean shift diffusion forward process and a mean shift diffusion reverse process, wherein, Mean-shift diffusion forward process: The mean of the Gaussian noise is replaced by the residual between the low-resolution image latent feature and the original high-resolution image latent feature, and scaled according to the weight factor so that the distribution mean is transferred from the high-resolution image latent feature to the low-resolution image latent feature; The reverse process of mean-shift diffusion is to replace the mean of Gaussian noise with the residual between the low-resolution image latent features and the high-resolution image latent features, and scale them according to the weight factor so that the distribution mean is transformed from the low-resolution image latent features to the high-resolution latent features.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the real scene image super-resolution method based on mean-shift diffusion are implemented as described in any one of claims 1 to 4.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the real scene image super-resolution method based on mean shift diffusion are implemented as described in any one of claims 1 to 4.

Images