Image quality enhancement method and device, equipment and storage medium

Through the diffusion model and condition controller of the time step, the image generation is dynamically adjusted and combined with noise and reconstruction loss function for training, the problem of poor image quality enhancement in the existing technology is solved, and efficient image quality improvement and fidelity are achieved degree to maintain.

CN119048371AActive Publication Date: 2024-11-29GUANGZHOU HUYA TECH CO LTD

Patent Information

Application Number
CN202410843294.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-06-27
Publication Date
2024-11-29
Estimated Expiration
2044-06-27

AI Technical Summary

Technical Problem

The existing picture quality enhancement technology is poor when improving the picture quality of the display terminal, resulting in blurred picture quality and loss of information, and unable to effectively improve the user experience.

Method used

The time-step-based diffusion model and condition controller are adopted, and the image generation is dynamically adjusted through feature extraction and denoising processing, and the image is output enhanced, and the noise loss function and reconstruction loss function are combined for joint training, to optimize the diffusion model and condition controller.

Benefits of technology

It significantly improves the image quality enhancement effect of low-quality images, generates rich texture details, maintains fidelity, and is close to the performance of high-quality images, greatly improving the image quality enhancement effect.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119048371A_ABST
    Figure CN119048371A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of image processing, and discloses an image quality enhancement method and device, equipment and a storage medium. The image quality enhancement method comprises the following steps: firstly, obtaining a to-be-enhanced target image and a noise image satisfying Gaussian distribution; the target image is input into a trained condition controller for feature extraction, a control signal based on time steps is output, and the control signal comprises shallow features and deep features of the target image; and finally, inputting a noise image and a control signal into the trained diffusion model for de-noising processing, dynamically controlling image generation through a time step in the de-noising processing process, and outputting an enhanced image corresponding to the target image. According to the invention, rich texture details can be generated, so that the image is consistent with a high-image-quality image in fidelity, and the image quality enhancement effect of a low-image-quality image is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method, device, equipment and storage medium for image quality enhancement. Background Art

[0002] With the development of technology, mobile devices such as mobile phones and tablets are increasingly widely used due to their diverse functions and small portability. More and more users like to watch videos or live broadcasts on mobile devices. However, due to the influence of network bandwidth and the hardware conditions of mobile devices themselves, in order to reduce the amount of data, videos on mobile devices are usually compressed, that is, pixels are sampled and compressed, which will cause image distortion and information loss, and may cause problems such as blurred image quality and freezes during video playback. Therefore, in order to improve the user experience, it is necessary to enhance the image quality during video playback.

[0003] Existing image quality enhancement technologies usually use contrast enhancement algorithms and local light control algorithms. The contrast enhancement algorithm processes the image of the display terminal in the signal dimension. Through a series of complex calculations and adjustments, the contrast of the display terminal's image is significantly improved, the light and dark contrast in the image is more obvious, the details are richer, and the audience can get a more immersive viewing experience. The local light control algorithm adjusts the brightness of a specific area by controlling the local backlight, enhancing the contrast and details of the image, and bringing the audience a stronger visual enjoyment of the light and dark contrast of the image. When the existing technology enhances the image quality of the display terminal's image, it often only independently improves the image contrast of the display terminal from the signal dimension or the backlight dimension, resulting in poor image quality enhancement effect of the display terminal. Summary of the invention

[0004] The main purpose of the present invention is to provide a method, device, equipment and storage medium for image quality enhancement, aiming to solve the technical problem that the existing image quality enhancement has poor enhancement effect.

[0005] A first aspect of the present invention provides a method for enhancing image quality, the method comprising:

[0006] Obtaining a target image to be enhanced and a noise image satisfying Gaussian distribution;

[0007] Input the target image into a trained conditional controller for feature extraction, and output a control signal based on a time step, wherein the control signal includes shallow features and deep features of the target image;

[0008] The noise image and the control signal are input into a trained diffusion model for denoising, and image generation is dynamically controlled through time steps during the denoising process to output an enhanced image corresponding to the target image.

[0009] Optionally, in a first implementation of the first aspect of the present invention, the diffusion model is obtained by using the following training method:

[0010] Obtain pairs of low-quality images and high-quality images as training samples;

[0011] Inputting the low-quality image into the conditional controller to be trained for feature extraction, and outputting a control signal based on a time step, wherein the control signal includes shallow features and deep features of the low-quality image;

[0012] Inputting the high-quality image into the diffusion model to be trained for noise addition, so as to add the first noise used in the current time step to the corresponding high-quality image, and obtain the noisy image of the current time step;

[0013] Inputting the noisy image based on the time step and the time code corresponding to the time step into the diffusion model to be trained for noise prediction, and outputting the second noise corresponding to the time step;

[0014] The noisy image and the control signal are input into the diffusion model to be trained for denoising, and the image generation is dynamically controlled by the time step during the denoising process to obtain the denoised image of the current time step;

[0015] Based on the first noise, the second noise, the control signal, the denoised image and the high-quality image, the diffusion model and the conditional controller are jointly trained to obtain the trained diffusion model and conditional controller.

[0016] Optionally, in a second implementation of the first aspect of the present invention, the diffusion model and the condition controller are jointly trained based on the first noise, the second noise, the control signal, the denoised image, and the high-quality image to obtain the trained diffusion model and the condition controller, including:

[0017] Based on the noise error between the first noise and the second noise corresponding to each time step, construct a noise loss function corresponding to each time step;

[0018] Based on the reconstruction error between the denoised image corresponding to each time step and the high-quality image used for training, a reconstruction loss function corresponding to each time step is constructed;

[0019] Based on the control signal, dynamically adjust the mixed results of shallow features and deep features of the low-quality image at different time steps to control the generation focus of the diffusion model at each image generation stage;

[0020] Based on the noise loss function, the reconstruction loss function and the control signal, the diffusion model and the conditional controller are adjusted to obtain a trained diffusion model and conditional controller.

[0021] Optionally, in a third implementation of the first aspect of the present invention, the control signal is as follows:

[0022]

[0023] Among them, F represents the control signal, and the flatten(*) function represents flattening the multi-dimensional vector into a one-dimensional vector. represents the parameters related to the time step t, F S represents the shallow features of the image extracted by the conditional controller, F D represents the deep features of the image extracted by the conditional controller, C represents the number of channels, and H and W represent the height and width of the image input to the condition controller, respectively.

[0024] Optionally, in a fourth implementation of the first aspect of the present invention, the noisy image is as follows:

[0025]

[0026] Among them, Xt represents the noisy image at the tth time step, represents the parameters related to the time step t, e t represents the noise added at the tth time step, e t ∈R 3×H×W , H and W represent the height and width of the noisy image respectively.

[0027] Optionally, in a fifth implementation of the first aspect of the present invention, the noise loss function is as follows:

[0028]

[0029] Among them, L noise represents the noise loss function, e t represents the noise added at the tth time step, represents the noise predicted at the tth time step;

[0030] The reconstruction loss function is as follows:

[0031] L rec =λ 1 ·||X SR -X|| 2 +λ 2 ·||VGG(X SR )-VGG(X)|| 2 +λ 3 ||GAN(X SR )-GAN(X)|| 2 ;

[0032] Among them, L rec represents the reconstruction loss function, X SR represents the output image of the diffusion model, X represents the input image of the diffusion model, VGG(*) represents the VGG network, GAN(*) represents the GAN network, λ 1 ,λ 2 ,λ 3 Indicates the weight value.

[0033] Optionally, in a sixth implementation of the first aspect of the present invention, the loss function used in the joint training of the diffusion model and the conditional controller is as follows:

[0034]

[0035] Where L represents the loss function used in the joint training of the diffusion model and the conditional controller, represents the parameters related to the time step t, L noise represents the noise loss function, L rec represents the reconstruction loss function.

[0036] A second aspect of the present invention further provides a picture quality enhancement device, the picture quality enhancement device comprising:

[0037] An acquisition module, used for acquiring a target image to be enhanced and a noise image satisfying a Gaussian distribution;

[0038] An extraction module, used for inputting the target image into a trained condition controller for feature extraction, and outputting a control signal based on a time step, wherein the control signal includes shallow features and deep features of the target image;

[0039] The generation module is used to input the noise image and the control signal into a trained diffusion model for denoising, and dynamically control image generation through time steps during the denoising process to output an enhanced image corresponding to the target image.

[0040] Optionally, in a first implementation of the second aspect of the present invention, the image quality enhancement device further includes:

[0041] A sample acquisition module, used to acquire pairs of low-quality images and high-quality images as training samples;

[0042] A feature extraction module, used for inputting a low-quality image into a conditional controller to be trained for feature extraction, and outputting a control signal based on a time step, wherein the control signal includes shallow features and deep features of the low-quality image;

[0043] A noise adding module, used for inputting the high-quality image into the diffusion model to be trained for noise adding, so as to add the first noise used in the current time step to the corresponding high-quality image, and obtain the noisy image of the current time step;

[0044] A prediction module, used for inputting the noisy image based on the time step and the time code corresponding to the time step into the diffusion model to be trained for noise prediction, and outputting the second noise corresponding to the time step;

[0045] A denoising module is used to input the noisy image and the control signal into the diffusion model to be trained for denoising, and dynamically control the image generation through the time step during the denoising process to obtain the denoised image of the current time step;

[0046] The training module is used to train the diffusion model based on the first noise, the second noise, the control signal, the denoised image and the high-quality image to obtain a trained diffusion model.

[0047] Optionally, in a second implementation of the second aspect of the present invention, the training module is specifically used to:

[0048] Based on the noise error between the first noise and the second noise corresponding to each time step, construct a noise loss function corresponding to each time step;

[0049] Based on the reconstruction error between the denoised image corresponding to each time step and the high-quality image used for training, a reconstruction loss function corresponding to each time step is constructed;

[0050] Based on the control signal, dynamically adjust the mixed results of shallow features and deep features of the low-quality image at different time steps to control the generation focus of the diffusion model at each image generation stage;

[0051] Based on the noise loss function, the reconstruction loss function and the control signal, the diffusion model and the conditional controller are adjusted to obtain a trained diffusion model and conditional controller.

[0052] Optionally, in a third implementation of the second aspect of the present invention, the control signal is as follows:

[0053]

[0054] Among them, F represents the control signal, and the flatten(*) function represents flattening the multi-dimensional vector into a one-dimensional vector. represents the parameters related to the time step t, F S represents the shallow features of the image extracted by the conditional controller, F D represents the deep features of the image extracted by the conditional controller, C represents the number of channels, and H and W represent the height and width of the image input to the condition controller, respectively.

[0055] Optionally, in a fourth implementation of the second aspect of the present invention, the noisy image is as follows:

[0056]

[0057] Among them, X t represents the noisy image at the tth time step, represents the parameters related to the time step t, e t represents the noise added at the tth time step, e t ∈R 3×H×W , H and W represent the height and width of the noisy image respectively.

[0058] Optionally, in a fifth implementation of the second aspect of the present invention, the noise loss function is as follows:

[0059]

[0060] Among them, L noise represents the noise loss function, e t represents the noise added at the tth time step, represents the noise predicted at the tth time step;

[0061] The reconstruction loss function is as follows:

[0062] L rec =λ 1 ·||X SR -X|| 2 +λ 2 ·||VGG(X SR )-VGG(X)|| 2 +λ 3 ||GAN(X SR )-GAN(X)|| 2 ;

[0063] Among them, L rec represents the reconstruction loss function, X SR represents the output image of the diffusion model, X represents the input image of the diffusion model, VGG(*) represents the VGG network, GAN(*) represents the GAN network, λ 1 ,λ 2 ,λ 3 Indicates the weight value.

[0064] Optionally, in a sixth implementation of the second aspect of the present invention, the loss function used in the joint training of the diffusion model and the conditional controller is as follows:

[0065]

[0066] Where L represents the loss function used in the joint training of the diffusion model and the conditional controller, represents the parameters related to the time step t, L noise represents the noise loss function, L rec represents the reconstruction loss function.

[0067] A third aspect of the present invention provides a computer device, comprising: a memory and at least one processor, wherein the memory stores instructions; the at least one processor calls the instructions in the memory so that the computer device executes the above-mentioned image quality enhancement method.

[0068] A fourth aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the above-mentioned image quality enhancement method.

[0069] The embodiment of the present invention provides a method, device, equipment and storage medium for image quality enhancement. First, a target image to be enhanced and a noise image satisfying a Gaussian distribution are obtained. Then, the target image is input into a trained conditional controller for feature extraction, and a control signal based on a time step is output. The control signal includes shallow features and deep features of the target image. Finally, the noise image and the control signal are input into a trained diffusion model for denoising. During the denoising process, the mixed results of the two sets of features are dynamically adjusted at different time steps to dynamically control image generation, and finally an enhanced image corresponding to the target image is output. The enhanced image output by the embodiment of the present invention can generate richer texture details, and thus maintain the same fidelity as the high-quality image, greatly improving the image quality enhancement effect of the low-quality image. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 A schematic diagram of an embodiment of a method for enhancing image quality in an embodiment of the present invention;

[0071] Figure 2 A schematic diagram of a flow chart of a network model training method according to an embodiment of the present invention

[0072] Figure 3 A schematic diagram of an embodiment of a device for enhancing image quality according to an embodiment of the present invention;

[0073] Figure 4 FIG. 1 is a schematic diagram of an embodiment of a computer device in an embodiment of the present invention. DETAILED DESCRIPTION

[0074] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0075] For ease of understanding, the specific process of the image quality enhancement method in the embodiment of the present invention is described below. The image quality enhancement method of this embodiment is applied to an electronic device, which may be a terminal, such as a smart phone, a laptop, a tablet computer, a desktop computer, etc., or a development board, a chip system, etc. Please refer to Figure 1 , Figure 1 This is a schematic diagram of a first embodiment of a method for enhancing image quality in an embodiment of the present invention. In this embodiment, the method for enhancing image quality includes:

[0076] 101. Obtain a target image to be enhanced and a noise image satisfying a Gaussian distribution;

[0077] The image quality enhancement of this embodiment is specifically applied to the image quality enhancement of each frame of video images in a video stream. When a user turns on the image quality enhancement function, an image quality enhancement task can be generated to enhance the image quality of each frame of video images in the currently playing video stream.

[0078] In this embodiment, when performing the image quality enhancement task, the target image to be enhanced is first obtained, such as any frame of video image in the currently played video stream, and at the same time, a frame of image is collected from the pre-constructed noise image samples that satisfy the Gaussian distribution as a noise image, and the noise image preferably uses an image that satisfies the standard Gaussian distribution. A noise image refers to an image with unnecessary or redundant interference information in the image data, and the noise image in this embodiment can be generated based on the Gaussian noise model.

[0079] 102. Input the target image into a trained condition controller for feature extraction, and output a control signal based on a time step, wherein the control signal includes shallow features and deep features of the target image;

[0080] In this embodiment, the condition controller adopts a fully convolutional network (FCN), which is a special convolutional neural network used to extract shallow and deep features of the input image. The extracted image features are used to guide the diffusion model to generate images. The fully convolutional structure specifically includes:

[0081] (1) Fully convolutional part: This part is mainly composed of a series of classic convolutional neural networks (such as VGG, ResNet, etc.), which are used to extract features from input data. These networks gradually extract high-level features in the image through structures such as convolutional layers and pooling layers.

[0082] (2) Deconvolution part: Since convolution and pooling operations will reduce the size of the feature map, in order to obtain an output with the same size as the input image, a deconvolution (also called transposed convolution, upsampling) operation is required. The main purpose of this part is to restore the low-resolution feature map to the size of the original input image.

[0083] When a fully convolutional structure is used in a conditional controller, the output of the network can be affected according to the input conditions. Due to the flexibility and powerful feature extraction capabilities of the fully convolutional structure, it can generate corresponding outputs according to different conditions, thereby achieving the purpose of conditional control.

[0084] The shallow features extracted by the shallow convolution in the conditional controller are specifically the high-frequency information in the target image (such as texture details), while the deep features extracted by the deep convolution in the conditional controller are specifically the global features and low-frequency information (such as semantics, composition, etc.) in the target image.

[0085] 103. Input the noise image and the control signal into a trained diffusion model for denoising, and dynamically control image generation through time steps during the denoising process to output an enhanced image corresponding to the target image.

[0086] Diffusion Models is a generative model used to generate data similar to training data. The core idea is to gradually add noise to the data by simulating the diffusion process, and then learn to reverse the process to construct the required data samples from the noise. Specifically, the Markov chain is used to define the diffusion steps, and the data is gradually "destroyed" into pure noise through the state transition of each step. Then, by training a neural network to approximate the real reverse diffusion process, the diffusion model can gradually restore the original data from pure noise.

[0087] In this embodiment, the target image to be enhanced is equivalent to a low-quality image, and the enhanced image generated by the diffusion model is equivalent to a high-quality image corresponding to the low-quality image. The low-quality image is used as a conditional signal to guide the denoising process of the diffusion model, which can prevent the generated high-quality image from deviating from the low-quality image, thereby ensuring that the image effect generated by the diffusion model is closer to the high-quality image.

[0088] This embodiment requires pre-training of the diffusion model for image quality enhancement. The trained diffusion model can achieve an image quality enhancement effect. Specifically, a pre-constructed noise image and a control signal generated based on a low-quality image are input into the trained diffusion model to denoise the noise image. During the denoising process, the generation focus of the diffusion model is further adjusted based on the control signal, thereby achieving image generation control. The enhanced image finally outputted is close to the high-quality image corresponding to the target image.

[0089] The enhanced image output by the embodiment of the present invention can generate relatively rich texture details, thereby maintaining consistency with the high-quality image in terms of fidelity, thereby greatly improving the quality enhancement effect of the low-quality image.

[0090] The embodiment of the present invention also provides a network model training method for training a model used in the image quality enhancement method, including a condition controller and a diffusion model. The network model training method of the embodiment of the present invention needs to complete the model training in advance before performing the image quality enhancement task. Figure 2 , Figure 2 The network model training method of the present invention is a flowchart of an embodiment of the network model training method. The network model training method of this embodiment includes the following steps:

[0091] 201. Obtain pairs of low-quality images and high-quality images as training samples;

[0092] In this embodiment, the training samples used for model training preferably use pairs of low-quality images and high-quality images, that is, low-quality images and high-quality images corresponding to the same image. High-quality images have more detailed information in color, texture and other detail levels, which helps to more accurately represent and analyze the shape, structure and appearance characteristics of objects. Low-quality images lack detailed information and appear somewhat blurred as a whole, especially at the edges of sharp lines.

[0093] 202. Input the low-quality image into the conditional controller to be trained for feature extraction, and output a control signal based on a time step, wherein the control signal includes shallow features and deep features of the low-quality image;

[0094] In this embodiment, during the training process, it is necessary to inject the feature information in each low-quality image in the training sample as a control signal into the cross-attention module of the diffusion model, and then in the entire generation process of the enhanced image, it is used to control the generation result of the diffusion model, that is, to achieve image quality enhancement based on low-quality image information. Specifically, the control signal can be controlled according to the time step of the noise addition process.

[0095] In one embodiment, the control signal used is as follows:

[0096]

[0097] Among them, F represents the control signal, and the flatten(*) function represents flattening the multi-dimensional vector into a one-dimensional vector. represents the parameters related to the time step t, F S represents the shallow features of the image extracted by the conditional controller, F D represents the deep features of the image extracted by the conditional controller, C represents the number of channels, and H and W represent the height and width of the image input to the condition controller, respectively.

[0098] From the functional form of the above control signal, we can see that (the larger the time step t, The smaller the value of ), as the diffusion process becomes smaller and the generation process becomes larger, in the early stage of the generation process (when the time step t is large), the deep features account for a large proportion, and low-frequency information such as composition is used to guide the diffusion model to generate global features that are consistent with the structure of the low-quality image, so that the early generation results focus more on the overall structure; in the later stage of the generation process (when the time step t is small), the shallow features account for a large proportion, and high-frequency information such as texture is used to guide the diffusion model to generate local features that are relatively consistent with the texture details of the low-quality image, so that the later generation results focus more on local details. During the training process, as the time step changes, the generation focus of the diffusion model can be automatically controlled based on the image information provided by the low-quality image, thereby realizing the process of first generating the overall part of the image and then generating the local part of the image.

[0099] 203. Input the high-quality image into the diffusion model to be trained for noise addition, so as to add the first noise used in the current time step to the corresponding high-quality image to obtain the noisy image of the current time step;

[0100] In this embodiment, the diffusion model specifically includes the following two processes:

[0101] (1) Forward diffusion process: The model corrupts the training data by continuously adding Gaussian noise;

[0102] (2) Reverse diffusion process: The model learns the reverse denoising process to restore the original data. This process is achieved by training a neural network to approximate the true reverse diffusion process.

[0103] In this embodiment, the diffusion model gradually adjusts the original data into pure noise through the state transition of each step, and then approximates the real reverse diffusion process by training the neural network, thereby gradually restoring the original data from the pure noise data.

[0104] Based on the principle of the above diffusion model, the high-quality image is input into the diffusion model to be trained for noise processing, so that the first noise used in the current time step is added to the corresponding high-quality image to obtain the noisy image of the current time step.

[0105] In one embodiment, the noise-added image used is as follows:

[0106]

[0107] Where X represents the training sample (i.e., high-quality image, X∈R 3×H×W ), X t represents the noisy image at the tth time step, represents the parameters related to the time step t, which can be calculated automatically by the diffusion process, e t represents the noise added at the tth time step, e t ∈R 3×H×W , H and W represent the height and width of the noisy image respectively.

[0108] 204. Inputting the noisy image based on the time step and the time code corresponding to the time step into the diffusion model to be trained for noise prediction, and outputting the second noise corresponding to the time step;

[0109] In this embodiment, after the noisy image corresponding to the time step is generated in step 203, the noisy image and the time code of the time step t (using sin-cos coding) are used as the input of the diffusion model (preferably using the Unet network), and the diffusion model is input for noise prediction. The output result is the noise predicted for the corresponding time step.

[0110] In this embodiment, after the noise prediction training of the model is completed, the noise-added image X in step t can be used to predict the noise of the image. t and time step t predicts the noise Accordingly, assume that the noisy image used in the t-1th time step is where α t , σ t are all parameters automatically controlled by the time step t, based on the denoised image X t-1and time step t-1 to obtain the next noise And the denoised result X t-2 , and so on for other time steps.

[0111] Based on the above reasoning process, the reverse diffusion of the diffusion model can be obtained from the initial standard Gaussian noise X T Start denoising, and finally get a completely noise-free image X at step 0 SR , which is the enhanced image finally generated by the diffusion model. Among them, the noise prediction training can use the noise loss function to optimize the model parameter weights.

[0112] 205. Input the noisy image and the control signal into the diffusion model to be trained for denoising, and dynamically control image generation through the time step during the denoising process to obtain a denoised image at the current time step;

[0113] In this embodiment, in order to further improve the image quality and fidelity of the generated image, the characteristics of the low-quality image are introduced as a control signal when denoising the noisy image. The image generation focus of the diffusion model is automatically controlled based on the image information provided by the control signal, thereby ensuring the image quality and fidelity of the generated image.

[0114] In this embodiment, the noisy image and the control signal are input into the diffusion model to be trained for denoising, and the image generation is dynamically controlled by the time step during the denoising process to obtain the denoised image of the current time step. The diffusion model needs to perform denoising processing for multiple time steps to finally generate a completely noise-free image SR image (Super Resolution, SR, super resolution image, also known as X SR ).

[0115] In this embodiment, by continuously injecting feature information of low-quality images and dynamically adjusting them during the generation process, the model is prompted to generate different image focuses at different stages. At the same time, the dynamically adjustable reconstruction loss can be further combined to achieve improvements in image quality and fidelity of the SR images generated by the diffusion model.

[0116] 206. Based on the first noise, the second noise, the control signal, the denoised image and the high-quality image, jointly train the diffusion model and the conditional controller to obtain a trained diffusion model and conditional controller.

[0117] In this embodiment, image quality enhancement is achieved by iterative denoising to generate an SR image. This method is highly dependent on previous generation results. Therefore, simply optimizing the diffusion model through noise loss cannot completely avoid unnatural modifications to texture details of high-quality images. Therefore, in this embodiment, a reconstruction loss function is further introduced to optimize the model parameter weights, thereby improving the fidelity of the model-generated images.

[0118] In one embodiment, the step 206 further includes:

[0119] 2061. Construct a noise loss function corresponding to each time step based on a noise error between the first noise and the second noise corresponding to each time step;

[0120] In one embodiment, the noise loss function used is as follows:

[0121]

[0122] Among them, L noise represents the noise loss function, e t represents the noise added at the tth time step, represents the noise predicted at the tth time step;

[0123] 2062. Based on the reconstruction error between the denoised image corresponding to each time step and the high-quality image used for training, construct a reconstruction loss function corresponding to each time step;

[0124] The reconstruction loss function used in this embodiment is specifically composed of pixel error loss, perceptual loss and GAN loss. In one embodiment, the reconstruction loss function used is as follows:

[0125] L rec =λ 1 ·||X SR -X|| 2 +λ 2 ·||VGG(X SR )-VGG(X)|| 2 +λ 3 ||GAN(X SR )-GAN(X)|| 2 ;

[0126] Among them, L rec represents the reconstruction loss function, X SR represents the output image of the diffusion model, X represents the input image of the diffusion model, VGG(*) represents the VGG network, GAN(*) represents the GAN network, λ 1 ,λ 2 ,λ 3Represents the weight value. The VGG network is a deep convolutional neural network (CNN) architecture proposed by the Visual Geometry Group. Since the VGG network uses a deeper hierarchical structure and a smaller convolution kernel, it can extract richer image features and has good performance for complex visual tasks. The GAN network, or Generative Adversarial Networks, is a deep learning model. The network consists of two main parts: the generator and the discriminator, which are trained in an adversarial way to generate high-quality simulated data.

[0127] In this embodiment, the image output by the diffusion model can be expressed as From the calculation formula, we can see that according to the image X corresponding to the time step t t and the predicted noise The final generated image X can be obtained directly SR , but due to the noisy image X used in the early time step t is closer to pure noise, and carries very little effective information. Therefore, X is calculated based on the above formula. SR And find the reconstruction loss L rec This may easily lead to unstable training, so the noise loss is retained when training the diffusion model to generate images in this embodiment.

[0128] 2063. Based on the control signal, dynamically adjust the mixed results of the shallow features and the deep features of the low-quality image at different time steps to control the generation focus of the diffusion model at each image generation stage;

[0129] The control signals used in this embodiment are as follows:

[0130]

[0131] From the above, it can be seen that the control signal is actually a mixture of shallow features and deep features, and the mixing method is related to the time step. This embodiment uses a separate network to extract features from low-quality images, extracts shallow features and deep features respectively, dynamically adjusts the mixed results of the two sets of features at different time steps, and directly injects them into the cross-attention module of the diffusion model as control conditions to avoid information attenuation of low-quality images. During the model training process, not only noise errors are considered, but also image reconstruction losses. In the process of reducing the time step, the proportion of reconstruction losses between the SR image output by the model and the high-quality image input by the model is gradually increased to ensure that the SR image finally output by the model is closer to the high-quality image input by the model.

[0132] 2064. Based on the noise loss function, the reconstruction loss function and the control signal, adjust the diffusion model and the conditional controller to obtain a trained diffusion model and conditional controller.

[0133] In this embodiment, the diffusion model inputs a high-quality image to obtain a noisy image after denoising, and simultaneously inputs a low-quality image. Then, based on the noise loss and the reconstruction loss that can be dynamically adjusted, the diffusion model and the conditional controller are jointly trained to optimize the diffusion model and the conditional controller. After the training is completed, they can be applied to the image quality enhancement task.

[0134] In one embodiment, the loss function finally adopted by the joint training of the diffusion model and the conditional controller is as follows:

[0135]

[0136] Where L represents the loss function used in the joint training of the diffusion model and the conditional controller, represents the parameters related to the time step t, L noise represents the noise loss function (as described above), L rec Represents the reconstruction loss function (as described above). The conditional controller uses a full convolution structure, which needs to be jointly trained with the diffusion model to adjust the full convolution structure, thereby adjusting the feature information of the low-quality image extracted at different time steps. The conditional controller is specifically Optimize the control signal to further optimize the fully convolutional structure.

[0137] In the joint training process of the diffusion model and the conditional controller, this embodiment dynamically adjusts the conditional control and reconstruction loss through the time step to ensure that the final generation effect is close to the high-definition image, thereby generating richer texture details while maintaining the fidelity consistent with the high-definition image, greatly improving the image quality enhancement effect.

[0138] The above describes the image quality enhancement method in the embodiment of the present invention. The following describes the image quality enhancement device in the embodiment of the present invention. Figure 3 In one embodiment of the present invention, an image quality enhancement device includes:

[0139] An acquisition module 301 is used to acquire a target image to be enhanced and a noise image satisfying a Gaussian distribution;

[0140] An extraction module 302 is used to input the target image into a trained condition controller for feature extraction, and output a control signal based on a time step, wherein the control signal includes shallow features and deep features of the target image;

[0141] The generation module 303 is used to input the noise image and the control signal into the trained diffusion model for denoising, and dynamically control image generation through time steps during the denoising process to output an enhanced image corresponding to the target image.

[0142] Optionally, in one embodiment, the image quality enhancement device further includes:

[0143] The sample acquisition module 304 is used to acquire pairs of low-quality images and high-quality images as training samples;

[0144] A feature extraction module 305 is used to input the low-quality image into the condition controller to be trained for feature extraction, and output a control signal based on a time step, wherein the control signal includes shallow features and deep features of the low-quality image;

[0145] A noise adding module 306 is used to input the high-quality image into the diffusion model to be trained for noise adding, so as to add the first noise used in the current time step to the corresponding high-quality image to obtain the noisy image of the current time step;

[0146] A prediction module 307 is used to input the noisy image based on the time step and the time code corresponding to the time step into the diffusion model to be trained for noise prediction, and output the second noise corresponding to the time step;

[0147] A denoising module 308 is used to input the noisy image and the control signal into the diffusion model to be trained for denoising, and dynamically control the image generation through the time step during the denoising process to obtain a denoised image at the current time step;

[0148] The training module 309 is used to train the diffusion model based on the first noise, the second noise, the control signal, the denoised image and the high-quality image to obtain a trained diffusion model.

[0149] Optionally, in one embodiment, the training module 309 is specifically used for:

[0150] Based on the noise error between the first noise and the second noise corresponding to each time step, construct a noise loss function corresponding to each time step;

[0151] Based on the reconstruction error between the denoised image corresponding to each time step and the high-quality image used for training, a reconstruction loss function corresponding to each time step is constructed;

[0152] Based on the control signal, dynamically adjust the mixed results of shallow features and deep features of the low-quality image at different time steps to control the generation focus of the diffusion model at each image generation stage;

[0153] Based on the noise loss function, the reconstruction loss function and the control signal, the diffusion model and the conditional controller are adjusted to obtain a trained diffusion model and conditional controller.

[0154] Optionally, in one embodiment, the control signal is as follows:

[0155]

[0156] Among them, F represents the control signal, and the flatten(*) function represents flattening the multi-dimensional vector into a one-dimensional vector. represents the parameters related to the time step t, F S represents the shallow features of the image extracted by the conditional controller, F D represents the deep features of the image extracted by the conditional controller, C represents the number of channels, and H and W represent the height and width of the image input to the condition controller, respectively.

[0157] Optionally, in one embodiment, the noisy image is as follows:

[0158]

[0159] Among them, X t represents the noisy image at the tth time step, represents the parameters related to the time step t, e t represents the noise added at the tth time step, e t ∈R 3×H×W , H and W represent the height and width of the noisy image respectively.

[0160] Optionally, in one embodiment, the noise loss function is as follows:

[0161]

[0162] Among them, L noise represents the noise loss function, e t represents the noise added at the tth time step, represents the noise predicted at the tth time step;

[0163] The reconstruction loss function is as follows:

[0164] L rec =λ 1 ·||X SR -X|| 2 +λ 2 ·||VGG(X SR )-VGG(X)|| 2 +λ 3||GAN(X SR )-GAN(X)|| 2 ;

[0165] Among them, L rec represents the reconstruction loss function, X SR represents the output image of the diffusion model, X represents the input image of the diffusion model, VGG(*) represents the VGG network, GAN(*) represents the GAN network, λ 1 ,λ 2 ,λ 3 Indicates the weight value.

[0166] Optionally, in one embodiment, the loss function used in the joint training of the diffusion model and the conditional controller is as follows:

[0167]

[0168] Where L represents the loss function used in the joint training of the diffusion model and the conditional controller, represents the parameters related to the time step t, L noise represents the noise loss function, L rec represents the reconstruction loss function.

[0169] Since the embodiments of the device part correspond to the embodiments of the above-mentioned method, please refer to the above-mentioned method embodiments for the introduction of the image quality enhancement device provided by the present invention. The present invention will not be repeated here, and it has the same beneficial effects as the above-mentioned image quality enhancement method.

[0170] above Figure 3 The image quality enhancement apparatus in the embodiment of the present invention is described in detail from the perspective of modular functional entities, and the computer device in the embodiment of the present invention is described in detail from the perspective of hardware processing.

[0171] Figure 41 is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. The computer device 500 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 510 (for example, one or more processors) and a memory 520, and one or more storage media 530 (for example, one or more mass storage devices) storing application programs 533 or data 532. Among them, the memory 520 and the storage medium 530 can be short-term storage or permanent storage. The program stored in the storage medium 530 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the computer device 500. Furthermore, the processor 510 may be configured to communicate with the storage medium 530 to execute a series of instruction operations in the storage medium 530 on the computer device 500.

[0172] The computer device 500 may also include one or more power supplies 540, one or more wired or wireless network interfaces 550, one or more input and output interfaces 560, and / or one or more operating systems 531, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. It will be appreciated by those skilled in the art that Figure 4 The illustrated computer device structure does not constitute a limitation on the computer device, and may include more or fewer components than illustrated, or combine certain components, or arrange the components differently.

[0173] The present invention also provides a computer device, which includes a memory and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor executes the steps of the image quality enhancement method in the above-mentioned embodiments.

[0174] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer, the computer executes the steps of the image quality enhancement method.

[0175] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0176] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program codes.

[0177] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for enhancing image quality, characterized in that: The image quality enhancement method comprises: Obtaining a target image to be enhanced and a noise image satisfying Gaussian distribution; Input the target image into a trained conditional controller for feature extraction, and output a control signal based on a time step, wherein the control signal includes shallow features and deep features of the target image; The noise image and the control signal are input into a trained diffusion model for denoising, and image generation is dynamically controlled through time steps during the denoising process to output an enhanced image corresponding to the target image.

2. The image quality enhancement method according to claim 1, characterized in that: The diffusion model is obtained by the following training method: Obtain pairs of low-quality images and high-quality images as training samples; Inputting the low-quality image into the conditional controller to be trained for feature extraction, and outputting a control signal based on a time step, wherein the control signal includes shallow features and deep features of the low-quality image; Inputting the high-quality image into the diffusion model to be trained for noise addition, so as to add the first noise used in the current time step to the corresponding high-quality image, and obtain the noisy image of the current time step; Inputting the noisy image based on the time step and the time code corresponding to the time step into the diffusion model to be trained for noise prediction, and outputting the second noise corresponding to the time step; The noisy image and the control signal are input into the diffusion model to be trained for denoising, and the image generation is dynamically controlled by the time step during the denoising process to obtain the denoised image of the current time step; Based on the first noise, the second noise, the control signal, the denoised image and the high-quality image, the diffusion model and the conditional controller are jointly trained to obtain the trained diffusion model and conditional controller.

3. The image quality enhancement method according to claim 2, characterized in that: The method of jointly training the diffusion model and the condition controller based on the first noise, the second noise, the control signal, the denoised image and the high-quality image to obtain the trained diffusion model and the condition controller includes: Based on the noise error between the first noise and the second noise corresponding to each time step, construct a noise loss function corresponding to each time step; Based on the reconstruction error between the denoised image corresponding to each time step and the high-quality image used for training, a reconstruction loss function corresponding to each time step is constructed; Based on the control signal, dynamically adjust the mixed results of shallow features and deep features of the low-quality image at different time steps to control the generation focus of the diffusion model at each image generation stage; Based on the noise loss function, the reconstruction loss function and the control signal, the diffusion model and the conditional controller are adjusted to obtain a trained diffusion model and conditional controller.

4. The image quality enhancement method according to claim 2, characterized in that: The control signal is as follows: Among them, F represents the control signal, and the flatten(*) function represents flattening the multi-dimensional vector into a one-dimensional vector. represents the parameters related to the time step t, F s represents the shallow features of the image extracted by the conditional controller, F D represents the deep features of the image extracted by the conditional controller, C represents the number of channels, and H and W represent the height and width of the image input to the condition controller, respectively.

5. The image quality enhancement method according to claim 2, characterized in that: The noisy image is shown below: Among them, Xt represents the noisy image at the tth time step, represents the parameters related to the time step t, e t represents the noise added at the tth time step, e t ∈R 3×H×W , H and W represent the height and width of the noisy image respectively.

6. The image quality enhancement method according to claim 3, characterized in that: The noise loss function is as follows: Among them, L noise represents the noise loss function, e t represents the noise added at the tth time step, represents the noise predicted at the tth time step; The reconstruction loss function is as follows: L rec =λ1·||X SR -X||2+λ2·||VGG(X SR )-VGG(X)||2+λ3·||GAN(X SR )-GAN(X)||2; Among them, L rec represents the reconstruction loss function, X SR represents the output image of the diffusion model, X represents the input image of the diffusion model, VGG(*) represents the VGG network, GAN(*) represents the GAN network, λ1, λ2, λ3 represent the weight values.

7. The image quality enhancement method according to claim 6, characterized in that: The loss function used in the joint training of the diffusion model and the conditional controller is as follows: Where L represents the loss function used in the joint training of the diffusion model and the conditional controller, represents the parameters related to the time step t, L noise represents the noise loss function, L rec represents the reconstruction loss function.

8. A device for enhancing image quality, characterized in that: The image quality enhancement device comprises: An acquisition module, used for acquiring a target image to be enhanced and a noise image satisfying a Gaussian distribution; An extraction module, used for inputting the target image into a trained condition controller for feature extraction, and outputting a control signal based on a time step, wherein the control signal includes shallow features and deep features of the target image; The generation module is used to input the noise image and the control signal into a trained diffusion model for denoising, and dynamically control image generation through time steps during the denoising process to output an enhanced image corresponding to the target image.

9. A computer device, characterized in that: The computer device comprises: a memory and at least one processor, wherein instructions are stored in the memory; The at least one processor calls the instructions in the memory to enable the computer device to execute the image quality enhancement method according to any one of claims 1 to 7.

10. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instructions are executed by the processor, the image quality enhancement method according to any one of claims 1 to 7 is implemented.

Citation Information

Patent Citations

  • Image enhancement method and device based on diffusion model, equipment and storage medium

    CN116664450A

  • Low-dose DSA angiography method based on spatial diffusion model

    CN116725560A

  • Single image super-resolution method based on conditional diffusion model

    CN117173018A

  • Image Enhancement via Iterative Refinement based on Machine Learning Models

    US20230067841A1

  • Latent prior embedded network for restoration and enhancement of images

    WO2024049441A1

Cited By

  • Model training method and apparatus, image quality improvement method and apparatus, and display apparatus

    WO2026137253A1