High-fidelity low-illumination image enhancement method and system based on one-step diffusion
By combining a pre-trained diffusion model with a one-step diffusion technique, and utilizing conditional latent variable refinement and wavelet skip connections, the problems of generalization, computational cost, and fidelity in low-light image enhancement are solved, achieving efficient, deterministic, and high-fidelity image enhancement.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-02
- Publication Date
- 2026-03-13
Smart Images
Figure CN121660909A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image enhancement and image generation technology, specifically to a high-fidelity low-light image enhancement method and system based on one-step diffusion, and also to a corresponding image enhancement device and computer-readable storage medium. Background Technology
[0002] Low-light image enhancement improves brightness, detail, and color reproduction; one-step diffusion technology enables rapid inference and preserves generated quality; the combination of these two technologies allows for more efficient image restoration while maintaining image quality.
[0003] In the existing fields of low-light image enhancement and one-step diffusion, commonly used techniques include: A new color space for low-light image enhancement, abbreviated as CIDNet; A low-light image enhancement method based on a wavelet diffusion model, abbreviated as DiffLL; An efficient one-step diffusion network for real-world image super-resolution, abbreviated as OSEDiff.
[0004] The aforementioned CIDNet proposes a novel color space for low-light image enhancement—the Horizontal / Vertical Intensity (HVI) color space—and designs a color and intensity decoupling network based on this new color space. Although HVI effectively avoids the color bias and artifact problems easily caused by existing sRGB and HSV color spaces, its generalization ability to real images is poor due to domain offset (the difference between training data and complex and diverse real data), making it difficult to apply effectively to the real world.
[0005] The aforementioned DiffLL combines wavelet transform with a diffusion model to achieve high-quality low-light image enhancement. However, because DiffLL chooses to train the diffusion model from scratch rather than utilizing the strong generative priors of pre-trained diffusion models, it also faces challenges in generalization. Furthermore, the diffusion model it employs incurs high computational costs and long inference times.
[0006] OSEDiff, as described above, was proposed as an image super-resolution method, but by changing the training data, this framework can be adapted to various low-level visual tasks. However, in low-light scenes, on the one hand, due to the significant compression of the variational autoencoder (VAE), the conditional latent variables suffer irreversible information loss; on the other hand, existing methods struggle to generate accurate text cues from low-light images. This results in insufficient available information for the one-step diffusion model, making effective control impossible and leading to fidelity issues.
[0007] Currently, no descriptions or reports of technologies similar to this invention have been found, and no similar information has been collected domestically or internationally. Summary of the Invention
[0008] To address the aforementioned shortcomings in the prior art, this invention provides a high-fidelity low-light image enhancement method and system based on one-step diffusion, along with a corresponding image enhancement device and computer-readable storage medium.
[0009] According to a first aspect of the present invention, a high-fidelity low-light image enhancement method based on one-step diffusion is provided, comprising: Acquire normal lighting images, train a codebook for a vector quantization-based generative adversarial network (VQGAN), and obtain the normal lighting codebook and the latent variables of normal lighting. A VAE encoder with a pre-trained diffusion model is used to obtain conditional latent variables under low illumination. A latent variable refinement and alignment model is constructed. The conditional latent variables under low illumination are refined and aligned through the latent variable refinement and alignment model to obtain refined conditional latent variables and aligned conditional latent variables. Using the normal illumination codebook, the aligned conditional latent variables are quantized to obtain the quantized latent variables; A denoising network model based on one-step diffusion is constructed. The refined conditional latent variables are used as the diffusion starting point, and the quantized latent variables are used as the vector embedding to replace the text embedding, so as to obtain the enhanced latent variables corresponding to the normal illumination image. By utilizing a wavelet-based skip connection and a pre-trained diffusion model in the VAE decoder, high-frequency information of features from the input image in the VAE encoder is used as additional information beyond the enhanced latent variables to obtain the final enhancement result.
[0010] Preferably, the training is based on the codebook of the Vector Quantization Generative Adversarial Network (VQGAN) to obtain a normal illumination codebook, including: The normal illumination codebook consists of three parts: a VQGAN encoder, a codebook, and a VQGAN decoder, employing reconstruction loss. Perceived loss Codebook loss and combating losses The combination of is used as the training objective, and is represented as: In the formula, and These represent the target image and the result image, respectively. Represents the VGG19 feature extractor. The latent variables represent the input image obtained after passing through the VQGAN encoder. This represents the latent variables obtained after codebook retrieval. Represents the discriminator. For hyperparameters, This represents a pass-through gradient estimator used to facilitate backpropagation; The final complete loss function used to train the normal illumination codebook is the weighted sum of the four loss functions mentioned above.
[0011] Preferably, the VAE encoder employing a pre-trained diffusion model acquires conditional latent variables for low illumination; a latent variable refinement and alignment model is constructed, and the conditional latent variables for low illumination are refined and aligned using the latent variable refinement and alignment model to obtain refined and aligned conditional latent variables, including: A VAE encoder using a pre-trained diffusion model processes the input low-light image. Encoded as conditional latent variables under low light conditions ; A latent variable refinement and alignment model is constructed, comprising a shared module and two corresponding output heads connected to it, which respectively form branches for refinement. and branches used for alignment ; Conditional latent variables of low light The tensor obtained losslessly from the input image after pixel rearrangement connect; Branches for refinement Using tensors Lossless original image information supplements the conditional latent variables of low illumination. Information loss, the refined conditional latent variables are denoted as : Meanwhile, branches used for alignment Conditional latent variables of low light The distribution of is aligned with the normal illumination distribution on which the normal illumination codebook is based, and the aligned conditional latent variable is denoted as . : .
[0012] Preferably, the step of quantizing the aligned conditional latent variables to obtain quantized latent variables includes: By using nearest neighbor matching, the aligned conditional latent variables Each latent variable vector in Quantization is achieved by using a learnable codebook pre-trained on normally lit images. The corresponding code , is represented as: .
[0013] Preferably, the construction of a one-step diffusion-based denoising network model involves using the refined conditional latent variables as the diffusion starting point and the quantized latent variables as the vector embedding to replace the text embedding, to obtain the latent variables corresponding to the normal illumination image, including: A denoising network model based on one-step diffusion is constructed, wherein the denoising network model refines the conditional latent variables. As the starting point for diffusion, the quantized latent variables output from the codebook Vector embedding, as an alternative to text embedding, is represented as: In the formula, This represents a denoising network. This represents the predefined diffusion step size. This represents the latent variable corresponding to the final generated normal lighting image.
[0014] Preferably, the loss function of the one-step diffusion-based denoising network model includes: the data loss required by the one-step diffusion framework. Generative adversarial network loss for distillation and ensuring latent variable refinement and model alignment branches. Alignment loss in performance ;in: The data loss L1 loss Structural similarity loss With perceived loss The weighted sum is expressed as: In the formula, Represents the enhanced image. Represents the target image. and These are structural similarity loss and perceived loss Weighting coefficients; The generative adversarial network loss includes the generator. loss and discriminator loss , is represented as: In the formula, Representative target image Encoded latent variables, The latent variables representing the final generated normal lighting image, For expectation operators.
[0015] The VQGAN encoder of the normal illumination codebook For target image Encode and use as the alignment loss The goal is represented as: .
[0016] In the formula, This represents the aligned conditional latent variables.
[0017] Preferably, the VAE decoder utilizing wavelet-based skip connections and a pre-trained diffusion model incorporates high-frequency information from the features of the input image in the VAE encoder as additional information beyond the enhanced latent variables to obtain the final enhanced result, including: The VAE encoder, based on a pre-trained diffusion model, encodes low-light features at each resolution level of the input image. Performing a two-dimensional discrete wavelet transform yields high-frequency components containing image details and low-frequency components containing image degradation information, expressed as: In the formula, and They represent resolution levels respectively. Low light characteristics High-frequency and low-frequency components obtained by two-dimensional discrete wavelet transform (DWT); Only the high-frequency components are transmitted and compared with the enhanced features at the corresponding resolution level during the decoding process of the VAE decoder based on the pre-trained diffusion model. The low-frequency components are matched one by one, and then the fused features are reconstructed through two-dimensional inverse discrete wavelet transform. , is represented as: In the formula, Represents resolution level Enhanced features The low-frequency components are obtained by decomposition using two-dimensional discrete wavelet transform (DWT), while the high-frequency components are obtained simultaneously. It does not participate in the reconstruction of fused features. IDWT, on the other hand, represents the two-dimensional discrete wavelet inverse transform. The fusion features The decoding process is incorporated at each resolution level using pixel-by-pixel addition, providing features from the input image in the VAE encoder for decoding. High-frequency information, as enhanced latent variables Additional information beyond what is needed is ultimately decoded to obtain an enhanced result. .
[0018] According to a second aspect of the present invention, a high-fidelity low-light image enhancement system based on one-step diffusion is provided, comprising: Normal illumination codebook construction module: This module is used to acquire high-quality normal illumination images, train a codebook based on vector quantization generative adversarial network (VQGAN), and obtain a normal illumination codebook. The conditional latent variable refinement and alignment module is used to obtain the conditional latent variables under low illumination using a pre-trained diffusion model VAE encoder; construct a latent variable refinement and alignment model, and refine and align the conditional latent variables under low illumination using the latent variable refinement and alignment model to obtain the refined conditional latent variables and the aligned conditional latent variables. A conditional latent variable quantization module uses the normal illumination codebook to quantize the aligned conditional latent variables to obtain quantized latent variables; A one-step diffusion module is used to construct a denoising network model based on one-step diffusion. Through the denoising network model, the refined conditional latent variables are used as the diffusion starting point, and the quantized latent variables are used as the vector embedding to replace the text embedding to obtain the latent variables corresponding to the normal illumination image. The wavelet-based skip connection enhancement module utilizes a VAE decoder based on wavelet skip connections and a pre-trained diffusion model to incorporate high-frequency information from the features of the input image in the VAE encoder as additional information beyond the enhanced latent variables, thus obtaining the final enhancement result.
[0019] According to a third aspect of the present invention, an image enhancement device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, can perform the method described in any one of the above invention.
[0020] According to a fourth aspect of the present invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, can be used to perform the method described in any one of the preceding inventions.
[0021] By adopting the above technical solution, the present invention has at least one of the following beneficial effects compared with the prior art: Comprehensive advantages: Compared to CIDNet, this invention is based on one-step diffusion, which can utilize the pre-trained diffusion model as a natural prior, thereby achieving stronger generalization to real images; compared to DiffLL, this invention uses LoRA fine-tuning and distillation to inherit the generative prior of the pre-trained diffusion model, and solves the problems of computational cost and inference time through one-step diffusion; compared to OSEDiff, this invention increases the available information of the one-step diffusion model by supplementing the information lost in the VAE encoding process and providing effective prior knowledge as additional information, and further introduces wavelet-based skip connections, thereby achieving high-fidelity one-step diffusion under low-light conditions.
[0022] Generalization: Unlike CIDNet and DiffLL, this invention effectively utilizes a pre-trained diffusion model as a prior, thereby overcoming the domain offset problem and improving the generalization of the method to real images.
[0023] Computational cost and inference speed: Unlike DiffLL, this invention uses a novel one-step diffusion, which eliminates the need for multiple diffusion steps in traditional diffusion models, thereby significantly reducing computational cost and improving inference speed.
[0024] Fidelity: Unlike OSEDiff, this invention refines the conditional latent variables to supplement the lost information and replaces inaccurate text prompts with effective normal illumination codebook priors, thereby effectively controlling the one-step diffusion model. Combined with wavelet-based skip connections between VAEs, the fidelity of this method is significantly improved.
[0025] By integrating these advantages, the present invention enhances low-light images with high fidelity while ensuring generalizability for complex and diverse real-world images. Attached Figure Description
[0026] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a flowchart illustrating the workflow of a high-fidelity low-light image enhancement method based on one-step diffusion in a preferred embodiment of the present invention.
[0027] Figure 2 This is a schematic diagram of the constituent modules of a high-fidelity low-light image enhancement system based on one-step diffusion in a preferred embodiment of the present invention.
[0028] Figure 3 This is a schematic diagram of the working architecture of a high-fidelity low-light image enhancement system based on one-step diffusion in a preferred embodiment of the present invention.
[0029] Figure 4 This is a structural diagram of a normal illumination codebook obtained in a preferred embodiment of the present invention.
[0030] Figure 5 This is a structural diagram of a wavelet-based skip connection enhancement module in a preferred embodiment of the present invention. Detailed Implementation
[0031] The embodiments of the present invention are described in detail below: These embodiments are implemented based on the technical solution of the present invention, and provide detailed implementation methods and specific operation processes. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention.
[0032] In existing technologies, commonly used techniques in low-light image enhancement and one-step diffusion, such as CIDNet, DiffLL, and OSEDiff, still have many shortcomings in terms of comprehensiveness, generalization ability, computational cost and inference speed, and fidelity. To address these shortcomings, one embodiment of this invention provides a high-fidelity low-light image enhancement method based on one-step diffusion. This method achieves high-fidelity low-light image enhancement based on one-step diffusion, overcoming the problems of generalization to real images, computational cost and inference speed, fidelity, and uncertainties in the diffusion process inherent in existing technologies. Furthermore: Improving generalization to real images: Given that existing pre-trained diffusion models are trained on massive amounts of high-quality images, this method fully leverages this strong natural prior to overcome the domain offset problem and improve generalization to real images, thereby providing reliable, effective, and high-quality real-world low-light image enhancement. This addresses the problem of insufficient generalization to real images in existing techniques.
[0033] Reduced computational cost and improved inference speed: This method employs a novel one-step diffusion approach, significantly reducing the required diffusion steps to a single step without compromising the powerful image generation capabilities of the diffusion model. This results in a substantial reduction in computational cost and a significant increase in inference speed compared to traditional diffusion models. This addresses the high computational cost and slow inference speed of diffusion-based methods.
[0034] High fidelity: To address the fidelity issue of existing one-step diffusion models under low-light conditions, this method introduces conditional latent variable refinement and normal-light codebook priors to more effectively control the one-step diffusion model and improve fidelity. Furthermore, wavelet-based skip connections between VAEs can further enhance the fidelity of the results beyond the diffusion model, ensuring that the image enhancement result does not deviate from the original image. This solves the low-fidelity problem of one-step diffusion under low-light conditions.
[0035] Deterministic Diffusion: The uncertainty introduced by random noise sampling in diffusion models is detrimental to image enhancement tasks. To address this uncertainty, this method employs a one-step diffusion approach, directly using the input image rather than random noise as the starting point for diffusion. This eliminates the uncertainty in the diffusion process at its source, thereby avoiding generative distortion.
[0036] By solving the aforementioned technical problems, this method can be effectively applied to the field of high-quality, high-fidelity low-light image enhancement in the real world.
[0037] Specifically, such as Figure 1 As shown, the high-fidelity low-light image enhancement method based on one-step diffusion provided in this embodiment may include: S1. Obtain high-quality normal illumination images, train the codebook based on vector quantization generative adversarial network VQGAN, and obtain the normal illumination codebook. S2, using a pre-trained diffusion model VAE encoder to obtain conditional latent variables under low illumination; construct a latent variable refinement and alignment model, and refine and align the conditional latent variables under low illumination respectively through the latent variable refinement and alignment model to obtain refined conditional latent variables and aligned conditional latent variables. S3, using the normal illumination codebook, quantize the aligned conditional latent variables to obtain the quantized latent variables; S4. Construct a denoising network model based on one-step diffusion. Through the denoising network model, the refined conditional latent variables are used as the starting point for diffusion, and the quantized latent variables are used as the vector embedding to replace the text embedding, so as to obtain the enhanced latent variables corresponding to the normal illumination image. S5 utilizes a wavelet-based skip connection and a pre-trained diffusion model-based VAE decoder to incorporate high-frequency information from the input image features in the VAE encoder as additional information beyond the enhanced latent variables, thus obtaining the final enhanced result.
[0038] In some preferred embodiments, S1 above, training a codebook based on a vector quantization generative adversarial network (VQGAN) to obtain a codebook under normal lighting conditions, may further include: The normal illumination codebook consists of three parts: a VQGAN encoder, a codebook, and a VQGAN decoder, employing reconstruction loss. Perceived loss Codebook loss and combating losses The combination of is used as the training objective, and is represented as: In the formula, and These represent the target image and the result image, respectively. Represents the VGG19 feature extractor. The latent variables represent the input image obtained after passing through the VQGAN encoder. This represents the latent variables obtained after codebook retrieval. Represents the discriminator. For hyperparameters, This represents a pass-through gradient estimator used to facilitate backpropagation; The final complete loss function used to train the normal illumination codebook is the weighted sum of the four loss functions mentioned above.
[0039] In some preferred embodiments, S2 above uses a pre-trained diffusion model VAE encoder to obtain conditional latent variables for low illumination; a latent variable refinement and alignment model is constructed, and the conditional latent variables for low illumination are refined and aligned using the latent variable refinement and alignment model to obtain refined and aligned conditional latent variables. It may further include: S21, using a pre-trained diffusion model VAE encoder, processes the input low-light image. Encoded as conditional latent variables under low light conditions ; S22, Construct the latent variable refinement and alignment model. The latent variable refinement and alignment model includes a shared module and two corresponding output heads connected to it, which respectively form branches for refinement. and branches used for alignment ; S23, the conditional latent variable of low light. The tensor obtained losslessly from the input image through pixel-unshuffle. connect; Branches for refinement Using tensors Lossless original image information supplements the conditional latent variables of low illumination. Information loss, the refined conditional latent variables are denoted as : Meanwhile, branches used for alignment Conditional latent variables of low light The distribution of is aligned with the normal illumination distribution upon which the normal illumination codebook is based, and the aligned conditional latent variables are denoted as . : .
[0040] In some preferred embodiments, S3 above, which quantizes the aligned conditional latent variables to obtain quantized latent variables, may further include: By using nearest neighbor matching, the aligned conditional latent variables Each latent variable vector in Quantization is achieved by using a learnable codebook pre-trained on normally lit images. The corresponding code , is represented as: .
[0041] In some preferred embodiments, S4 above, constructing a denoising network model based on one-step diffusion, using the refined conditional latent variables as the diffusion starting point and the quantized latent variables as the vector embedding to replace the text embedding, to obtain the latent variables corresponding to the normal illumination image, may further include: Construct a denoising network model based on one-step diffusion. The denoising network model will refine the conditional latent variables. As the starting point for diffusion, the quantized latent variables output from the codebook Vector embedding, as an alternative to text embedding, is represented as: In the formula, This represents a denoising network. This represents the predefined diffusion step size. This represents the latent variable corresponding to the final generated normal lighting image.
[0042] In some preferred embodiments, the above-mentioned S4, based on the one-step diffusion denoising network model, may further include: S41, Construct the loss function for the denoising network model based on one-step diffusion, including: the data loss required by the single-step diffusion framework. Generative Adversarial Network (GAN) loss used for distillation and branch alignment to ensure latent variable refinement and model alignment. Alignment loss in performance ;in: S42, Data Loss L1 loss Structural Similarity (SSIM) Loss With perception (LPIPS) loss The weighted sum is expressed as: In the formula, Represents the enhanced image. Represents the target image. and These are structural similarity loss and perceived loss Weighting coefficients; S43, Generative Adversarial Network Loss includes the generator. loss and discriminator loss , is represented as: In the formula, Representative target image Encoded latent variables, The latent variables representing the final generated normal lighting image, Let be the expectation operator, representing the calculation of the mean.
[0043] S44, the VQGAN encoder with normal illumination codebook. For target image Encode and use as alignment loss The goal is represented as: .
[0044] In the formula, This represents the aligned conditional latent variables.
[0045] In some preferred embodiments, S5 above, utilizing a VAE decoder based on wavelet skip connections and a pre-trained diffusion model, uses high-frequency information of features from the input image in the VAE encoder as additional information beyond the enhanced latent variables to obtain the final enhanced result. It may further include: S51, encodes low-light features at each resolution level of the input image using a VAE encoder based on a pre-trained diffusion model. Performing a two-dimensional discrete wavelet transform (DWT) yields high-frequency components containing image details and low-frequency components containing image degradation information, represented as: In the formula, and They represent resolution levels respectively. Low light characteristics High-frequency and low-frequency components obtained by two-dimensional discrete wavelet transform (DWT); S52 transmits only high-frequency components and, during the decoding process of the VAE decoder based on a pre-trained diffusion model, enhances the features at the corresponding resolution level. The low-frequency components are matched one by one, and then the fused features are reconstructed through two-dimensional inverse discrete wavelet transform (IDWT). , is represented as: In the formula, Represents resolution level Enhanced features The low-frequency components are obtained by decomposition using two-dimensional discrete wavelet transform (DWT), while the high-frequency components are obtained simultaneously. It does not participate in the reconstruction of fused features. IDWT stands for Inverse Discrete Wavelet Transform; this selective transmission avoids the burden of degraded low-frequency information in the original image on the VAE decoder, while ensuring that high-frequency information, which is crucial for image fidelity, is included in the decoding process; S53 will integrate features The decoding process is incorporated at each resolution level using pixel-by-pixel addition, providing features from the input image in the VAE encoder for decoding. High-frequency information, as enhanced latent variables Additional information beyond what is needed is ultimately decoded to obtain an enhanced result. .
[0046] Based on the same inventive concept, an embodiment of the present invention also provides a high-fidelity low-light image enhancement system based on one-step diffusion.
[0047] Specifically, such as Figure 2 As shown, the high-fidelity low-light image enhancement system based on one-step diffusion provided in this embodiment may include: Normal illumination codebook construction module: This module is used to acquire high-quality normal illumination images, train a codebook based on vector quantization generative adversarial network (VQGAN), and obtain a normal illumination codebook. The Conditional Latent Variable Refinement and Alignment Module is used to obtain the conditional latent variables under low illumination using a pre-trained diffusion model VAE encoder; it constructs a latent variable refinement and alignment model to refine and align the conditional latent variables under low illumination, resulting in refined and aligned conditional latent variables. The conditional latent variable quantization module uses the normal illumination codebook to quantize the aligned conditional latent variables to obtain the quantized latent variables. The one-step diffusion module is used to construct a denoising network model based on one-step diffusion. Through the denoising network model, the refined conditional latent variables are used as the starting point for diffusion, and the quantized latent variables are used as the vector embedding to replace the text embedding to obtain the latent variables corresponding to the normal illumination image. The wavelet-based skip connection enhancement module utilizes a VAE decoder based on wavelet skip connections and a pre-trained diffusion model to incorporate high-frequency information from the features of the input image in the VAE encoder as additional information beyond the enhanced latent variables, thus obtaining the final enhancement result.
[0048] The following describes in detail, with reference to preferred embodiments, the specific implementation methods of each functional module constituting the system provided in the above embodiments of the present invention.
[0049] like Figure 3 The diagram shows the overall system architecture, which includes: a normal illumination codebook construction module for pre-training the required normal illumination codebook; a conditional latent variable refinement and alignment module, which uses the LRA module to generate refined and aligned latent variables respectively; a conditional latent variable quantization module, which quantizes the aligned conditional latent variables; a one-step diffusion module, which uses a unique loss function for one-step diffusion training to retrieve vector embeddings from the pre-trained normal illumination codebook as control conditions for replacing text embeddings; and a wavelet-based skip connection enhancement module, which uses wavelet-based skip connections and a VAE decoder to decode the results of the denoising network.
[0050] The normal illumination codebook construction module is a preparation phase module, including: In this module, a certain number of high-quality normal lighting images are collected and used to train a codebook based on Vector Quantization Generative Adversarial Network (VQGAN), serving as a priori normal lighting codebook. This normal lighting codebook consists of three parts: a VQGAN encoder, the codebook itself, and a VQGAN decoder. Figure 4 As shown. Reconstruction loss is used. Perceived loss Codebook loss and combating losses The combination of these elements is used as the training objective, and its mathematical expression is as follows: in, and These represent the target image and the result image, respectively. Represents the VGG19 feature extractor. The latent variables represent the input image obtained after passing through the VQGAN encoder. This represents the latent variables obtained after codebook retrieval. Represents the discriminator. These are hyperparameters. Furthermore, since the nearest neighbor (NN) matching used in codebook retrieval cannot backpropagate gradients, therefore... This represents the pass-through gradient estimator used to facilitate backpropagation. The final complete loss function used to train the normal illumination codebook is a weighted sum of the four loss functions mentioned above.
[0051] Existing text prompt generators struggle to produce accurate text prompts in low-light scenarios, while vector quantization codebooks built on VQGAN, through extensive training on a large number of normally lit images, are able to capture the inherent features of high-quality, normally lit images.
[0052] The conditional latent variable alignment module, namely the Latent Refinement and Alignment (LRA) module, includes: Encoding unit: In this unit, a pre-trained VAE encoder is applied to process the input low-light image. Encoding as conditional latent variables .
[0053] Conditional latent variable refinement unit: In this unit, conditional latent variables will be... The tensor obtained losslessly from the input image after pixel rearrangement The connection is then refined by the branches in the LRA module. (The two branches within the LRA consist of a shared module and two corresponding output headers) for processing, in order to utilize Lossless original image information supplementation condition latent variables Information loss. The refined conditional latent variables of the output are denoted as... This process can be represented as: .
[0054] Conditional latent variable alignment unit: This unit runs concurrently with the previous unit, and is performed by the alignment branch in the LRA module. The process involves aligning the conditional latent variables for low-light conditions with those used in the pre-trained normal-light codebook, thereby improving codebook retrieval accuracy and providing more accurate codebook prior knowledge for the one-step diffusion model. This step outputs the aligned conditional latent variables. , can be represented as: .
[0055] In the above unit, the normal illumination codebook is pre-trained on normal illumination images, so directly using the mismatched low illumination latent variables will affect the model's performance. Therefore, the alignment branch and the corresponding alignment loss aim to align the distribution of the low illumination conditional latent variables with that of the normal illumination latent variables, so that the aligned latent variables can be effectively quantized based on the normal illumination codebook.
[0056] Conditional latent variable quantification module: In this module, the aligned conditional latent variables are obtained through NN matching (nearest neighbor matching). Each latent variable vector in Quantization into a learnable codebook The corresponding code , is represented as: Among them, the quantified latent variables The text embeddings generated by the text prompts in the original one-step diffusion model will be replaced by vector embeddings, providing the one-step diffusion model with prior knowledge of natural, normally lit images.
[0057] A one-step diffusion module, used for one-step diffusion in denoising networks, includes: In this module, a one-step diffusion-based denoising network model is used to refine the conditional latent variables. As the starting point for diffusion, the quantized latent variables output from the codebook Vector embedding, as an alternative to text embedding, is represented as: in This represents a denoising network. This represents the predefined diffusion step size. This represents the latent variable corresponding to the final generated normal lighting image.
[0058] Furthermore, this diffusion module also includes: the model's loss function unit.
[0059] Unlike traditional diffusion models and end-to-end deep learning models, the loss function of the one-step diffusion model in this embodiment consists of three parts: the data loss required by the single-step diffusion framework. Compared to the generative adversarial network (GAN) loss used for distillation, and ensuring aligned branches Alignment loss in performance Furthermore: It is a weighted sum of L1 loss, structural similarity (SSIM) loss, and perception (LPIPS) loss: in, Represents the target image. and These are the weighting coefficients for SSIM loss and LPIPS loss, respectively.
[0060] GAN loss used for distillation is divided into generator loss and generator loss. and discriminator Two parts: in, Represents the target image Encoded latent variables.
[0061] VQGAN encoder using a pre-trained normal illumination codebook For target image Encode and use as alignment loss The goal is represented as: .
[0062] Wavelet-based skip connection enhancement module: In this module, the low-light features at each resolution level in the VAE encoder are first processed. (This low light characteristic) The VAE encoder encodes the input low-light image into a conditional latent variable for low light. The intermediate results at various resolution levels during the process are subjected to a two-dimensional discrete wavelet transform (DWT) to obtain high-frequency components containing image details and low-frequency components containing degradation information such as image brightness and contrast, as shown below: In the formula, and They represent resolution levels respectively. Low light characteristics The high-frequency and low-frequency components are obtained by two-dimensional discrete wavelet transform.
[0063] Then only the high-frequency components are transmitted and compared with the enhanced features at the corresponding resolution level in the VAE decoder. (This enhanced feature) The VAE decoder will enhance the latent variables. Decoded into an enhanced image The low-frequency components of the intermediate results at various resolution levels during the process are matched one by one, and then the fused features are reconstructed through two-dimensional inverse discrete wavelet transform (IDWT). , is represented as: In the formula, Represents resolution level Enhanced features The low-frequency components are obtained by decomposition using two-dimensional discrete wavelet transform (DWT), while the high-frequency components are obtained simultaneously. It does not participate in the reconstruction of fusion features.
[0064] This selective transmission avoids the burden on the VAE decoder caused by degraded low-frequency information in the original image, while ensuring that high-frequency information, which is crucial for image fidelity, is included in the decoding process. This fusion feature The decoding process is incorporated at each resolution level using pixel-by-pixel addition, providing features from the input image in the VAE encoder for decoding. High-frequency information, as enhanced latent variables Additional information beyond what is needed is ultimately decoded to obtain an enhanced result. .like Figure 5 As shown, in Figure 5 In this process, the wavelet-based skip connection enhancement module transmits only high-frequency information containing image details, thereby enhancing fidelity and texture details while avoiding the adverse effects of degraded low-frequency information on the decoder.
[0065] An embodiment of the present invention also provides an image enhancement device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it can be used to perform any of the methods described in the above embodiments of the present invention.
[0066] Optionally, the memory is used to store programs; the memory may include volatile memory, such as random-access memory (RAM), such as static random-access memory (SRAM), double data rate synchronous dynamic random-access memory (DDR SDRAM), etc.; the memory may also include non-volatile memory, such as flash memory. The memory is used to store computer programs (such as application programs and functional modules that implement the above methods), computer instructions, etc., and the aforementioned computer programs and computer instructions can be partitioned and stored in one or more memories. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by the processor.
[0067] A processor is used to execute computer programs stored in memory to implement the various steps of the methods or various modules of the systems involved in the above embodiments. For details, please refer to the relevant descriptions in the preceding method and system embodiments.
[0068] The processor and memory can be separate structures or integrated structures. When the processor and memory are separate structures, they can be coupled together via a bus.
[0069] An embodiment of the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can be used to perform the method of any of the above embodiments of the present invention.
[0070] Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transfer of computer programs from one place to another. Storage media can be any available medium accessible to a general-purpose or special-purpose computer. An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and storage medium can reside in an ASIC. Alternatively, the ASIC can reside in a user device. Of course, the processor and storage medium can also exist as discrete components in a communication device.
[0071] The high-fidelity low-light image enhancement method and system based on one-step diffusion provided in the above embodiments of the present invention improves the generalization ability to real images: by using a pre-trained diffusion model as a natural prior, the domain offset problem is overcome, improving the generalization ability to real images and enabling effective application to low-light image enhancement in real-world scenes. It reduces computational costs and improves inference speed: by employing one-step diffusion technology, while inheriting the image generation capabilities of the diffusion model, the diffusion steps are significantly reduced, thereby significantly reducing computational costs and improving inference speed. It achieves high fidelity: by using conditional latent variable refinement and a normal lighting codebook prior that replaces text prompts, the supplementation of loss information and effective additional information solve the problem of scarce available information in low-light scenes for the one-step diffusion model, thus achieving effective control of the one-step diffusion model and ensuring the fidelity of the results. Simultaneously, wavelet-based skip connections between VAEs play a role outside the diffusion model, further improving the fidelity of the results. It achieves deterministic diffusion: by employing one-step diffusion, the diffusion process no longer involves random noise, thereby fundamentally eliminating the uncertainty caused by random noise sampling and avoiding generative distortion in image-to-image restoration and enhancement tasks. In summary, the method and system provided in the above embodiments of the present invention, by combining the above technologies, achieve high-fidelity low-light image enhancement based on one-step diffusion, overcoming the problems of generalization to real images, computational cost and inference speed, fidelity and uncertainty in the diffusion process in the prior art.
[0072] Any matters not covered in the above embodiments of the present invention are well-known in the art.
[0073] The specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.
Claims
1. A high-fidelity low-illumination image enhancement method based on one-step diffusion, characterized in that, include: Obtain normal lighting images, train a codebook based on vector quantization generative adversarial network VQGAN, and obtain the normal lighting codebook; A VAE encoder with a pre-trained diffusion model is used to obtain conditional latent variables under low illumination. A latent variable refinement and alignment model is constructed. The conditional latent variables under low illumination are refined and aligned through the latent variable refinement and alignment model to obtain refined conditional latent variables and aligned conditional latent variables. Using the normal illumination codebook, the aligned conditional latent variables are quantized to obtain the quantized latent variables; A denoising network model based on one-step diffusion is constructed. The refined conditional latent variables are used as the diffusion starting point, and the quantized latent variables are used as the vector embedding to replace the text embedding, so as to obtain the enhanced latent variables corresponding to the normal illumination image. By utilizing a wavelet-based skip connection and a pre-trained diffusion model in the VAE decoder, high-frequency information of features from the input image in the VAE encoder is used as additional information beyond the enhanced latent variables to obtain the final enhancement result.
2. The high-fidelity low-light image enhancement method based on one-step diffusion according to claim 1, characterized in that, The training is based on the codebook of the Vector Quantization Generative Adversarial Network (VQGAN) to obtain a normal illumination codebook, including: The normal illumination codebook consists of three parts: a VQGAN encoder, a codebook, and a VQGAN decoder, employing reconstruction loss. Perceived loss Codebook loss and combating losses The combination of is used as the training objective, and is represented as: In the formula, and These represent the target image and the result image, respectively. Represents the VGG19 feature extractor. The latent variables represent the input image obtained after passing through the VQGAN encoder. This represents the latent variables obtained after codebook retrieval. Represents the discriminator. For hyperparameters, This represents a pass-through gradient estimator used to facilitate backpropagation; The final complete loss function used to train the normal illumination codebook is the weighted sum of the four loss functions mentioned above.
3. The high-fidelity low-light image enhancement method based on one-step diffusion according to claim 1, characterized in that, The VAE encoder employing a pre-trained diffusion model acquires conditional latent variables for low-light conditions. A latent variable refinement and alignment model is constructed to refine and align the conditional latent variables for low-light conditions, resulting in refined and aligned conditional latent variables, including: A VAE encoder using a pre-trained diffusion model processes the input low-light image. Encoded as conditional latent variables under low light conditions ; A latent variable refinement and alignment model is constructed, comprising a shared module and two corresponding output heads connected thereto, which respectively form branches for refinement. and branches used for alignment ; Conditional latent variables of low light The tensor obtained losslessly from the input image after pixel rearrangement connect; Branches for refinement Using tensors Lossless original image information supplements the conditional latent variables of low illumination. Information loss, the refined conditional latent variables are denoted as : Meanwhile, branches used for alignment Conditional latent variables of low light The distribution of is aligned with the normal illumination distribution on which the normal illumination codebook is based, and the aligned conditional latent variable is denoted as . : 。 4. The high-fidelity low-light image enhancement method based on one-step diffusion according to claim 1, characterized in that, The step of quantizing the aligned conditional latent variables to obtain quantized latent variables includes: By using nearest neighbor matching, the aligned conditional latent variables Each latent variable vector in Quantization is achieved by using a learnable codebook pre-trained on normally lit images. The corresponding code in , is represented as: 。 5. The high-fidelity low-light image enhancement method based on one-step diffusion according to claim 1, characterized in that, The construction of a one-step diffusion-based denoising network model involves using the refined conditional latent variables as the starting point for diffusion and the quantized latent variables as vector embeddings to replace text embeddings, thereby obtaining the latent variables corresponding to the normally illuminated image, including: A denoising network model based on one-step diffusion is constructed, wherein the denoising network model refines the conditional latent variables. As the starting point for diffusion, the quantized latent variables output from the codebook Vector embedding, as an alternative to text embedding, is represented as: In the formula, This represents a denoising network. This represents the predefined diffusion step size. This represents the latent variable corresponding to the final generated normal lighting image.
6. The high-fidelity low-light image enhancement method based on one-step diffusion according to claim 5, characterized in that, The loss function of the denoising network model based on one-step diffusion includes: the data loss required by the one-step diffusion framework. Generative adversarial network loss for distillation and ensuring latent variable refinement and model alignment branches. Alignment loss in performance ;in: The data loss L1 loss Structural similarity loss With perceived loss The weighted sum is expressed as: In the formula, Represents the enhanced image. Represents the target image. and These are structural similarity loss and perceived loss Weighting coefficients; The generative adversarial network loss includes the generator. loss and discriminator loss , is represented as: In the formula, Representative target image Encoded latent variables, The latent variables representing the final generated normal lighting image, For expectation operators; The VQGAN encoder of the normal illumination codebook For the target image Encode and use as the alignment loss The goal is represented as: In the formula, This represents the aligned conditional latent variables.
7. The high-fidelity low-light image enhancement method based on one-step diffusion according to claim 1, characterized in that, The VAE decoder, which utilizes wavelet-based skip connections and a pre-trained diffusion model, incorporates high-frequency information from the features of the input image in the VAE encoder as additional information beyond the enhanced latent variables, resulting in the final enhanced outcome, including: The VAE encoder, based on a pre-trained diffusion model, encodes low-light features at each resolution level of the input image. Performing a two-dimensional discrete wavelet transform yields high-frequency components containing image details and low-frequency components containing image degradation information, expressed as: In the formula, and They represent resolution levels respectively. Low light characteristics High-frequency and low-frequency components obtained by two-dimensional discrete wavelet transform (DWT); Only the high-frequency components are transmitted and compared with the enhanced features at the corresponding resolution level during the decoding process of the VAE decoder based on the pre-trained diffusion model. The low-frequency components are matched one by one, and then the fused features are reconstructed through two-dimensional inverse discrete wavelet transform. , is represented as: In the formula, Represents resolution level Enhanced features The low-frequency components are obtained by decomposition using two-dimensional discrete wavelet transform (DWT), while the high-frequency components are obtained simultaneously. It does not participate in the reconstruction of fused features; IDWT represents the two-dimensional discrete wavelet inverse transform. The fusion features The decoding process is incorporated at each resolution level using pixel-by-pixel addition, providing features from the input image in the VAE encoder for decoding. High-frequency information, as enhanced latent variables Additional information beyond what is needed is ultimately decoded to obtain an enhanced result. .
8. A high-fidelity low-light image enhancement system based on one-step diffusion, characterized in that, include: Normal illumination codebook construction module: This module is used to acquire high-quality normal illumination images, train a codebook based on vector quantization generative adversarial network (VQGAN), and obtain a normal illumination codebook. The conditional latent variable refinement and alignment module is used to obtain the conditional latent variables under low illumination using a pre-trained diffusion model VAE encoder; construct a latent variable refinement and alignment model, and refine and align the conditional latent variables under low illumination using the latent variable refinement and alignment model to obtain the refined conditional latent variables and the aligned conditional latent variables. A conditional latent variable quantization module uses the normal illumination codebook to quantize the aligned conditional latent variables to obtain quantized latent variables; A one-step diffusion module is used to construct a denoising network model based on one-step diffusion. Through the denoising network model, the refined conditional latent variables are used as the diffusion starting point, and the quantized latent variables are used as the vector embedding to replace the text embedding to obtain the latent variables corresponding to the normal illumination image. The wavelet-based skip connection enhancement module utilizes a VAE decoder based on wavelet skip connections and a pre-trained diffusion model to incorporate high-frequency information from the features of the input image in the VAE encoder as additional information beyond the enhanced latent variables, thus obtaining the final enhancement result.
9. An image enhancement device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it can be used to perform the method of any one of claims 1-7.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program can be used to perform the method of any one of claims 1-7.