Image low-light enhancement and joint deblurring method based on dual-layer planning light and shadow clear network

Abstract

The application provides an image low-light enhancement joint deblurring method based on a double-layer planning light and shadow clear network, and belongs to the field of computer vision. The method introduces a block adaptive brightness enhancement module, a blur kernel parameter optimization module and a Fourier prior guide module in a diffusion model. The block adaptive brightness enhancement module divides the image into different illumination regions and differentiates the processing of each illumination region, ensuring the naturalness and smoothness of image brightness adjustment and avoiding the problems of uneven brightness or unnatural transition caused by global processing. The Fourier prior guide module processes the brightness and structural information of the image respectively through the amplitude and phase decomposition of the Fourier domain, realizes the joint optimization of brightness enhancement and deblurring, and ensures the simultaneous improvement of image clarity and brightness. Through the double-layer planning framework, the optimization of the image enhancement curve in the upper layer and the domain adaptive optimization of the LoRA parameter in the lower layer are realized, ensuring the global optimality of the generated image in brightness and clarity.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular relates to an image low-light enhancement and joint deblurring method based on a double-layer planning light and shadow clarity network. Background Art

[0002] Image enhancement and deblurring involve processing low-quality images through algorithms to improve visual quality, such as brightness, contrast, and clarity. Images captured in low-light environments are particularly prone to blurring, noise, and loss of detail. Therefore, enhancing image brightness and clarity is a crucial task in image processing. Low-light image enhancement and deblurring technologies are widely used in smartphone photography, video surveillance, autonomous driving, and other fields.

[0003] With advances in deep learning technology, a large number of methods for low-light image enhancement and deblurring have been proposed in recent years. Existing low-light enhancement methods and deblurring algorithms typically address brightness and blur separately, lacking the ability to jointly address both. This separate approach can result in image blur remaining after brightness enhancement, or a loss of brightness during the deblurring process, making it difficult to achieve the best of both worlds simultaneously.

[0004] With the advancement of technology, some methods have begun to handle the tasks of low-light enhancement and deblurring simultaneously. LED Net proposes a network structure that combines low-light enhancement and deblurring to solve the problems of blur and insufficient brightness in low-light environments. This method uses paired data of low-light blurry images and normal-light clear images for supervised learning, and jointly optimizes image brightness and clarity, making it robust when facing actual scenes. Although LED Net performs well in the joint task of low-light enhancement and deblurring, it relies on a large amount of paired data for supervised learning. This method is easily limited by the size of the dataset in practical applications, especially for real-world low-light blurry scenes. It is difficult to obtain a large amount of high-quality paired data, resulting in insufficient model generalization ability.

[0005] Many existing methods use a global processing strategy during brightening. This global approach can easily lead to over- or under-enhancement of some regions and fails to differentiate processing based on the varying illumination regions within the image. In particular, in images with large illumination variations, the transition between bright and dark areas can be unnatural. Furthermore, without domain-adaptive adjustments based on scene-specific data, the model struggles to effectively adapt to the characteristics of different scenes, resulting in poor performance when migrating between domains. This global processing strategy, lacking fine-tuning, is particularly inadequate in scenes with complex and changing illumination, failing to adapt to the differentiated needs of each region, impacting the overall quality of image enhancement.

[0006] When processing images in complex low-light environments or with severe motion blur, existing technologies struggle to ensure naturalness and consistency. In particular, in scenes with extremely uneven lighting and severe blur, the resulting image may exhibit over-enhancement, loss of detail, or residual blur, failing to meet the requirements for high-quality image enhancement. Therefore, this paper proposes a combined low-light image enhancement and deblurring method based on a dual-layer planning light and shadow clarity network. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this paper proposes a method for low-light image enhancement and deblurring based on a dual-layer planning light and shadow clarity network. This method aims to address the issues of independent processing, global processing, and reliance on paired data in existing low-light enhancement and deblurring methods. Experiments have demonstrated that this method can effectively improve image brightness and clarity, as well as processing performance in complex scenes, while also enhancing the algorithm's computational efficiency.

[0008] The technical solution adopted in the present invention is:

[0009] A method for joint deblurring of low-light image enhancement based on a two-layer planning light and shadow clarity network, comprising:

[0010] S1. Construct a Two-Layer Optimization Network for Light and Sharpness (TOLS-Net), which includes a diffusion model, a block-adaptive brightness enhancement module, a blur kernel parameter optimization module, and a Fourier prior guidance module.

[0011] The diffusion model performs iterative processing on the input Gaussian noise image to obtain the image x t , and for image x t Remove noise and reweight to get image x 0|t , where t∈T; the block adaptive brightness enhancement module is used to enhance the image x obtained by the diffusion model 0|t Perform grayscale world prior processing to obtain image x t ', and the image x t 'Divide different lighting areas, and use adaptive brightness enhancement curve to enhance the brightness of each lighting area to obtain the image The blur kernel parameter optimization module deblurs the input guide image to obtain the image y t ; The Fourier prior guide module converts the image With the image y t Perform image decomposition and update the intermediate clear image, and obtain image x based on the intermediate clear image t-1 The blur kernel parameter optimization module is based on the intermediate clear image to image y t+1and blur kernel Update and get image y t and blur kernel Used to guide the image x in the diffusion model t-1 Iterative update; the weight of the diffusion model is fine-tuned using the LoRA method;

[0012] S2. Use the LOL-Blur dataset to train the two-layer regularized light and shadow clarity network (TOLS-Net) constructed in S1; the training method adopts a two-layer planning method, where the goal of the upper layer planning is to optimize the parameters of the adaptive brightness enhancement curve, and the goal of the lower layer planning is to fine-tune the parameters of LoRA;

[0013] S3: Input the low-light blurry image to be processed into the network model trained in S2 to obtain a normal brightness and clear image.

[0014] Furthermore, in the block adaptive brightness enhancement module, the image x obtained by the diffusion model is 0|t Perform grayscale world prior processing to obtain image x t ', the specific process is:

[0015] In natural scenes, uneven lighting often leads to color deviation in images. To ensure color balance before image enhancement, a grayscale prior is introduced. The grayscale prior assumes that under normal lighting conditions, the average value of the red, green, and blue (R, G, B) channels of an image should be close to gray (i.e., the average values ​​of the three channels are equal). By adjusting the gain coefficient of each channel to achieve this balance, color distortion can be removed. The gain coefficient is calculated as follows:

[0016]

[0017] Among them, G r ,G g ,G b are the gain coefficients of R, G, and B channels, μ r ,μ g ,μ b is the average value of each channel, μ gray is the target grayscale value. Through this step, the brightness of each color channel can be balanced, thus providing a uniform input for the subsequent enhancement step.

[0018] The pixel values ​​of each channel after adjustment are

[0019]

[0020] After processing, merge the three-channel images to get x t';The processed image is more balanced in color and provides a good input for subsequent brightness enhancement.

[0021] Furthermore, in the block adaptive brightness enhancement module, the image x t 'Divide different lighting areas, the specific process is:

[0022] Use k-means clustering method to cluster image x' t Different lighting areas are divided, corresponding to areas with normal brightness, insufficient brightness and overexposure; in the clustering process, the input is the image x' t , contains the brightness information of each pixel; the output is a set of three types of pixels C i , i ranges from 1 to 3, where C1 represents the area with normal brightness, C2 represents the area with insufficient brightness, and C3 represents the area with overexposure;

[0023] The goal of k-means clustering is to divide pixels into regions with similar brightness and minimize the difference between the brightness value μ of each pixel in the region and the brightness value of the center of the region. i To this end, k-means clustering achieves grouping by optimizing the following objective function:

[0024]

[0025] Where x represents image C i A pixel value in contains characteristic information such as brightness; μ i Represents the central brightness value of the i-th cluster area; by minimizing the difference between the pixel in each cluster area and the central brightness value of the area, the image x' t It is divided into three types of lighting areas, so that the brightness characteristics of each type of area are relatively consistent.

[0026] Furthermore, in the block adaptive brightness enhancement module, the brightness of each illuminated area is enhanced using an adaptive brightness enhancement curve to obtain an image The specific process is:

[0027] An iterative form is adopted. Each iteration continues to improve the brightness based on the previous enhancement, so that the brightness of each illuminated area gradually approaches the ideal value. The iterative formula is as follows:

[0028] LE n (C i )=LE n-1 (C i )+α i,n LE n-1 (C i )(1-LE n-1 (C i ))

[0029] Among them, LE n (C i ) represents the illuminated area C after the nth iteration i Brightness value, LE n-1 (C i ) is the illuminated area C after the previous iteration i Brightness value, α i,n is the control parameter at the nth iteration; through iteration, the formula will gradually improve the area C i The overall brightness of the image is gradually increased to the target brightness level, and the enhanced image is finally obtained.

[0030] Furthermore, in the blur kernel parameter optimization module, the input guide image is deblurred to obtain the image y t , and update the blur kernel. The specific process is as follows:

[0031] First, calculate the image of the Tt-1th iteration The autocorrelation coefficient A(|P yt+1 |):

[0032]

[0033] in, is an image that only retains amplitude information, ⊙ represents the element-wise multiplication operation; the above autocorrelation result is estimated as the initial blur kernel k of the input image t ;

[0034] Then update the blur kernel:

[0035]

[0036] Finally, the method of Pan et al. is used to solve the following formula to obtain the clear image y t and the optimized blur kernel

[0037]

[0038] Among them, λ1 and λ2 are custom hyperparameters. is the convolution operation, To find the gradient operation, For iteration The initial value of .

[0039] Furthermore, in the Fourier prior guidance module, the image With the image y t Perform image decomposition and update the intermediate clear image, and obtain image x based on the intermediate clear image t-1The specific process is:

[0040] For images and y t Use Fast Fourier Transform (FFT) to decompose it into amplitude A and phase P components:

[0041]

[0042] (A t ,P t )=FFT(y t )

[0043] In each sampling step, the image is transformed into and y t The amplitude of the deblurred image y t The updated intermediate clear image is obtained by combining the phase of

[0044]

[0045] This method effectively preserves the details and clarity of low-light blurry images while maintaining the image brightness enhancement effect, thereby gradually generating high-quality output images.

[0046] Finally, the image x generated by the diffusion model t and the intermediate clear image estimated from the current time step Combine and get the image x of the current time step t-1 .

[0047] Furthermore, in the blur kernel parameter optimization module, based on the intermediate clear image For image y t Update and get image y t-1 , the specific method is: from The updated blur kernel k is estimated from t-1 As the updated result, the blur kernel k is used again t-1 Generate image y t-1 .

[0048] Furthermore, in step S2, a two-level planning training method is used to achieve multi-objective optimization in low-light enhancement and deblurring tasks; wherein the goal of the upper-level planning is to optimize the enhancement effect of the adaptive brightness enhancement curve. The upper-level planning problem can be expressed as:

[0049] minL upper (I enhanced θ brightness )

[0050] Among them, θ brightnessis the adaptive brightness enhancement curve parameter α i,n A collection of enhanced is the final enhanced result image; L upper It is a loss function that measures the visual effect of an image, including indicators such as brightness, contrast, and color consistency. The loss consists of three parts:

[0051] L upper (I enhanced θ brightness )=L brightness (I enhanced )+L contrast (I enhanced )+L color (I enhanced )

[0052] Among them, L brightness The goal is to ensure that the brightness of the enhanced image conforms to the natural distribution and can be expressed as the average brightness value of the image L mean (I enhanced ) and target brightness L target The differences:

[0053] L brightness (I enhanced )=||L mean (I enhanced )-L target || 2

[0054] L contrast To ensure that the enhanced image has a good visual hierarchy, the image contrast variance or histogram distribution is used to measure the contrast loss:

[0055] L contrast (I enhanced )=||H(I enhanced )-H target || 2

[0056] Where H(I) represents the image histogram, H target is the target histogram;

[0057] L color To avoid color distortion caused by the enhancement process, the gain G of each channel is adjusted based on the grayscale world assumption to ensure color consistency loss. r ,G g ,G b , so that the average value of the RGB three channels is close to gray:

[0058]

[0059] Where, are the red, green, and blue channels of the enhanced image, respectively;

[0060] The goal of the lower-level planning is to fine-tune the parameters of LoRA so that the image generated by the diffusion model is consistent with the input image prior in the Fourier domain, thereby ensuring that the image structure information is not lost. In the Fourier prior guidance module, the image is decomposed into amplitude A and phase P, and the phase reflects the image structure information. By fine-tuning the parameter matrix, the amplitude adjustment of the generated image can be controlled to ensure consistency with the input image prior in the Fourier domain. The lower-level planning problem can be expressed as:

[0061] minL Lower (I generated θ LoRA )

[0062] Specifically:

[0063] L lower (I generated θ LoRA )=L Fourier (I generated ,I input )+L deblur (I generated )

[0064] I generated is the output image of the diffusion model at each time step. In order to ensure that the generated image I generated The amplitude (brightness information) and phase (structural information) of the input image I input In the Fourier domain, we need to be as consistent as possible and introduce the Fourier domain consistency loss L Fourier :

[0065] L Fourier (I generated ,I input )=||A generated -A input || 2 +λ||P generated -P input || 2

[0066] Among them, A generated and A input are the Fourier magnitudes of the generated image and the input image, P generated and P input are the Fourier phases of the generated image and the input image, respectively, and λ is the weight for controlling the amplitude and phase loss;

[0067] L deblur To achieve deblurring loss, we need to ensure that the generated image is clear and has good deblurring effect, as follows:

[0068]

[0069] in Represents the gradient operation, which is used to capture the edge and structural features of the image.

[0070] Beneficial effects of the present invention:

[0071] 1) Joint optimization of low-light enhancement and deblurring: Existing technologies typically address image brightness enhancement and deblurring separately, lacking the ability to jointly optimize the two. This results in blur remaining after brightness enhancement or loss of brightness during deblurring. This invention uses amplitude and phase decomposition in the Fourier domain to process the image's brightness and structural information separately, achieving joint optimization of brightness enhancement and deblurring. This method can effectively remove blur while enhancing brightness, ensuring both image clarity and brightness are improved.

[0072] 2) Adaptive Illumination Region Processing: Existing methods use a global processing strategy, which can easily lead to over-enhancement or under-enhancement of some image regions, especially in scenes with large lighting variations. This invention uses k-means clustering technology to divide the image into different illumination regions and perform differentiated processing on each illumination region, ensuring natural and smooth image brightness adjustment and avoiding the problems of uneven brightness or unnatural transitions caused by global processing.

[0073] 3) Domain Adaptive Fine-tuning: Existing technologies generally lack adaptive fine-tuning across different domains, especially when dealing with scenes with complex lighting changes, resulting in poor model performance. This paper uses LoRA (Low Rank Adaptation) technology to perform domain adaptive fine-tuning in the diffusion model, enabling the model to quickly adapt to different lighting conditions and scenes, improving the robustness and consistency of image enhancement, and showing stronger adaptability in complex and changing lighting conditions.

[0074] 4) Global Optimization through Two-Level Planning: Through a two-level planning framework, the present invention achieves upper-level optimization of image enhancement curves and lower-level domain-adaptive optimization of LoRA parameters, ensuring globally optimal brightness and clarity of the generated image. Existing technologies struggle to simultaneously achieve both illumination enhancement and structural preservation. This invention effectively preserves the image's structural information while enhancing image brightness, resolving the common problem of detail loss after brightness enhancement in existing technologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Figure 1 This is a global flow chart of an embodiment of the present invention.

[0076] Figure 2 Schematic diagram of Lora domain adaptive adjustment based on an embodiment of the present invention. DETAILED DESCRIPTION

[0077] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0078] like Figure 1 As shown, an embodiment of the present invention provides an image low-light enhancement and joint deblurring method based on a two-layer planning light and shadow clarity network, comprising:

[0079] Step 1: Use the LOL-Blur dataset, which is the first large-scale dataset for the joint task of low-light enhancement and deblurring. It contains 12,000 pairs of synthetic low-light blurry images paired with normal-light clear images, and there are various low-light levels and motion blur modes in the dataset.

[0080] Step 2: Construct a two-layer optimization network for light and sharpness (TOLS-Net), which includes a diffusion model, a block-adaptive brightness enhancement module, a blur kernel parameter optimization module, and a Fourier prior guidance module.

[0081] The diffusion model performs iterative processing on the input Gaussian noise image to obtain the image x t The specific process is as follows:

[0082] First, in the forward process, the diffusion model gradually adds Gaussian noise to the image, eventually transforming it into a pure noise image. This process can be represented as a Markov chain, where each step degrades the image by adding a small amount of noise. The formula is:

[0083]

[0084] Among them, x t represents the image state at step t in the diffusion process, β t is the noise intensity at step t. By gradually adding noise, x T It becomes a pure noise image.

[0085] In the reverse process, the model generates a high-quality output image by gradually restoring a clear image from a noisy image. The conditional distribution of the reverse process is:

[0086] p θ (x t-1 |x t )=N(x t-1 ;μ θ (x t ,t),Σ θ (x t ,t))

[0087] Among them, μ θ (x t ,t) and Σ θ (x t ,t) is the mean and variance estimated by the neural network, guiding the model denoising process; the mean μ θ (x t ,t) can be further decomposed into:

[0088]

[0089] Among them, α t =1-β t ; is a cumulative term representing the weight calculated from the predefined noise schedule; ε θ (x t ,t) is the predicted noise.

[0090] The final loss function is optimized by minimizing the mean square error between the model's predicted noise and the actual noise, so that the model can effectively restore a clear image during the denoising process.

[0091] The diffusion model lasts for T steps. After the denoising process of the diffusion model is completed, the generated image has a certain degree of clarity and brightness improvement. However, due to the uneven illumination in low-light and blurry scenes, it is necessary to further enhance the brightness of different areas to achieve a more natural effect. To this end, in the iterative process, the image x obtained by the Tt-th iteration is selected. t Enter the block adaptive brightness enhancement module for processing. The t is a custom number, usually half of the total number of iterations. The specific process is as follows:

[0092] First, the image x obtained by the diffusion model 0|t Perform grayscale world prior processing to obtain image x t ', as follows:

[0093] In natural scenes, uneven lighting often leads to color deviation in images. To ensure color balance before image enhancement, a grayscale prior is introduced. The grayscale prior assumes that under normal lighting conditions, the average value of the red, green, and blue (R, G, B) channels of an image should be close to gray (i.e., the average values ​​of the three channels are equal). By adjusting the gain coefficient of each channel to achieve this balance, color distortion can be removed. The gain coefficient is calculated as follows:

[0094]

[0095] Among them, G r ,G g ,G bare the gain coefficients of R, G, and B channels, μ r ,μ g ,μ b is the average value of each channel, μ gray is the target grayscale value. Through this step, the brightness of each color channel can be balanced, thus providing a uniform input for the subsequent enhancement step.

[0096] The pixel values ​​of each channel after adjustment are

[0097]

[0098] After processing, merge the three-channel images to get x t ';The processed image is more balanced in color and provides a good input for subsequent brightness enhancement.

[0099] Then, the k-means clustering method is used to cluster the image x t 'Divide different lighting areas, corresponding to areas with normal brightness, insufficient brightness and overexposure; in the clustering process, the input is image x t ', contains the brightness information of each pixel; the output is a set of three types of pixels C i , i ranges from 1 to 3, where C1 represents the area with normal brightness, C2 represents the area with insufficient brightness, and C3 represents the area with overexposure;

[0100] The goal of k-means clustering is to divide pixels into regions with similar brightness and minimize the difference between the brightness value μ of each pixel in each region and the brightness value of the center of the region. i To this end, k-means clustering achieves grouping by optimizing the following objective function:

[0101]

[0102] Where x represents image C i A pixel value in contains characteristic information such as brightness; μ i Represents the central brightness value of the i-th cluster area; by minimizing the difference between the pixel in each cluster area and the central brightness value of the area, the image x' t It is divided into three types of lighting areas, so that the brightness characteristics of each type of area are relatively consistent.

[0103] Then, the adaptive brightness enhancement curve is used to enhance the brightness of each illuminated area to obtain the image Specifically:

[0104] An iterative form is adopted. Each iteration continues to improve the brightness based on the previous enhancement, so that the brightness of each illuminated area gradually approaches the ideal value. The iterative formula is as follows:

[0105] LE n (C i )=LE n-1 (C i )+α i,n LE n-1 (C i )(1-LE n-1 (C i ))

[0106] Among them, LE n (C i ) represents the illuminated area C after the nth iteration i Brightness value, LE n-1 (C i ) is the illuminated area C after the previous iteration i Brightness value, α i,n is the control parameter at the nth iteration; through N iterations, the formula will gradually improve the area C i The overall brightness of the image is gradually increased to the target brightness level, and the enhanced image is finally obtained.

[0107] In order to realize the guidance of the low-light blurred image y on the diffusion model, the blur kernel parameter optimization module is used to deblur the input guidance image y to obtain the image y t The specific process is:

[0108] First, calculate the image of the Tt-1th iteration The autocorrelation coefficient

[0109]

[0110] in, is an image that only retains amplitude information, ⊙ represents the element-wise multiplication operation; the above autocorrelation result can be approximately estimated as the initial blur kernel k of the input image t .

[0111] Then update the blur kernel:

[0112]

[0113] Finally, we use the method of Pan et al. (Pan L, Hartley R, Liu M, et al. Phase-only image-based kernel estimation for single image blind deblurring [C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2019: 6034-6043.) to solve the following equation and obtain the clear image y t and the optimized blur kernel

[0114]

[0115] Among them, λ1 and λ2 are custom hyperparameters. is the convolution operation, To find the gradient operation, For iteration The initial value of .

[0116] In the Fourier prior guidance module, the image obtained by the block adaptive brightness enhancement module The image y obtained by the fuzzy kernel parameter optimization module t Perform image decomposition and update the intermediate clear image, and obtain image x based on the intermediate clear image t-1 The specific process is:

[0117] For images and y t Use Fast Fourier Transform (FFT) to decompose it into amplitude A and phase P components:

[0118]

[0119] (A t ,P t )=FFT(y t )

[0120] In each sampling step, the image is transformed into and y t The amplitude of the deblurred image y t The updated intermediate clear image is obtained by combining the phase of

[0121]

[0122] This method effectively preserves the details and clarity of low-light blurred images while maintaining the image brightness enhancement effect, thereby gradually generating high-quality output images. Compared to y t There are clearer boundaries.

[0123] Finally, the image x generated by the diffusion model t and the intermediate clear image estimated from the current time step Combine and get the image x of the current time step t-1 (Ho, J., Jain, A., & Abbeel, P. (2020). Denoising DiffusionProbabilistic Models. Advances in Neural Information Processing Systems, 33, 6840–6851.).

[0124] The blur kernel parameter optimization module is again based on the intermediate clear image Estimate the updated blur kernel k t-1 As the updated result, the blur kernel k is used again t-1 Generate image y t-1 , thereby guiding the image x in the diffusion model t-1 Iterative updates.

[0125] Step 3: Use the LOL-Blur dataset from step 1 to train the two-layer optimized light and shadow clarity network (TOLS-Net) constructed in step 2. The training method adopts a two-layer programming method, where the goal of the upper-layer programming is to optimize the enhancement effect of the adaptive brightness enhancement curve. The upper-layer programming problem can be expressed as:

[0126] min L upper (I enhanced θ brightness )

[0127] Among them, θ brightness is the adaptive brightness enhancement curve parameter α i,n A collection of enhanced is the final enhanced result image; L upper It is a loss function that measures the visual effect of an image, including indicators such as brightness, contrast, and color consistency. The loss consists of three parts:

[0128] L upper (I enhanced θ brightness )=L brightness (I enhanced )+L contrast(I enhanced )+L color (I enhanced )

[0129] Among them, L brightness The goal is to ensure that the brightness of the enhanced image conforms to the natural distribution and can be expressed as the average brightness value of the image L mean (I enhanced ) and target brightness L target The differences:

[0130] L brightness (I enhanced )=||L mean (I enhanced )-L target || 2

[0131] L contrast To reduce contrast loss, we need to ensure that the enhanced image has a good visual hierarchy. The variance or histogram distribution of image contrast is often used to measure it:

[0132] L contrast (I enhanced )=||H(I enhanced )-H target || 2

[0133] Where H(I) represents the image histogram, H target is the target histogram;

[0134] L color To avoid color distortion caused by the enhancement process, the gain G of each channel is adjusted based on the grayscale world assumption to ensure color consistency loss. r ,G g ,G b , so that the average value of the RGB three channels is close to gray:

[0135]

[0136] Where, are the red, green and blue channels of the enhanced image respectively.

[0137] The goal of the lower-level planning is to fine-tune the parameters of LoRA so that the image generated by the diffusion model is consistent with the input image prior in the Fourier domain, thereby ensuring that the image structure information is not lost. In the Fourier prior guidance module, the image is decomposed into amplitude A and phase P, and the phase reflects the image structure information. By fine-tuning the parameter matrix, the amplitude adjustment of the generated image can be controlled to ensure consistency with the input image prior in the Fourier domain. The lower-level planning problem can be expressed as:

[0138] min L Lower (I generated θ LoRA )

[0139] Specifically:

[0140] L lower (I generated θ LoRA )=L Fourier (I generated ,I input )+L deblur (I generated )

[0141] I generated is the output image of the diffusion model at each time step. In order to ensure that the generated image I generated The amplitude (brightness information) and phase (structural information) of the input image I input In the Fourier domain, we need to be as consistent as possible and introduce the Fourier domain consistency loss L Fourier :

[0142] L Fourier (I generated ,I input )=||A generated -A input || 2 +λ||P generated -P input || 2

[0143] Among them, A generated and A input are the Fourier magnitudes of the generated image and the input image, P generated and P input are the Fourier phases of the generated image and the input image, respectively, and λ is the weight for controlling the amplitude and phase loss;

[0144] L deblur To achieve deblurring loss, we need to ensure that the generated image is clear and has good deblurring effect, as follows:

[0145]

[0146] in Represents the gradient operation, which is used to capture the edge and structural features of the image.

[0147] Step 4: Input the low-light blurred image to be processed into the network model trained in step 3 to obtain a normal brightness and clear image.

[0148] The above-described embodiments are preferred embodiments of the present invention and do not limit the present invention in any other manner. Any person skilled in the art may utilize the above-described contents to make modifications or follow their own examples. However, any modifications to the above-described embodiments that do not depart from the scope of the present invention and are based on the essence of the present invention remain within the scope of protection of the present invention.

Claims

1. A low-light image enhancement and joint deblurring method based on a two-layer planning light and shadow clarity network, characterized by: include: S1. Construct a two-layer planning light and shadow clarity network, which includes a diffusion model, a block-adaptive brightness enhancement module, a blur kernel parameter optimization module, and a Fourier prior guidance module; The diffusion model performs iterative processing on the input Gaussian noise image to obtain the image x t , and for image x t Remove noise and reweight to get image x 0|t , where t∈T; The block adaptive brightness enhancement module is used to enhance the image x obtained by the diffusion model. 0|t Perform grayscale world prior processing to obtain image x′ t , and the image x′ t Different illumination areas are divided, and the brightness of each illumination area is enhanced using an adaptive brightness enhancement curve to obtain an image The blur kernel parameter optimization module deblurs the input guide image to obtain the image y t ; The Fourier prior guide module converts the image With the image y t Perform image decomposition and update the intermediate clear image, and obtain image x based on the intermediate clear image t-1 The blur kernel parameter optimization module is based on the intermediate clear image to image y t+1 and blur kernel Update and get image y t and blur kernel Used to guide the image x in the diffusion model t-1 Iterative update; the weight of the diffusion model is fine-tuned using the LoRA method; S2. Use the LOL-Blur dataset to train the double-layer regularized light and shadow clarity network constructed in S1. The training method adopts a double-layer planning method, where the goal of the upper layer planning is to optimize the parameters of the adaptive brightness enhancement curve, and the goal of the lower layer planning is to fine-tune the parameters of LoRA. S3: Input the low-light blurry image to be processed into the network model trained in S2 to obtain a normal brightness and clear image.

2. The image low-light enhancement and joint deblurring method based on a dual-layer planning light and shadow clarity network according to claim 1 is characterized in that: In the block adaptive brightness enhancement module, the image x obtained by iterating the diffusion model is 0|t Perform grayscale world prior processing to obtain image x′ t The specific process is: Grayscale world assumes that under normal lighting conditions, the average value of the R, G, and B channels of the image should be close to gray, that is, the average values ​​of the three channels are equal. By adjusting the gain coefficient of each channel to achieve this balance, color distortion can be removed. The gain coefficient is calculated as follows: Among them, G r ,G g ,G b are the gain coefficients of R, G, and B channels, μ r ,μ g ,μ b is the average value of each channel, μ gray is the target grayscale value; The pixel values ​​of each channel after adjustment are After processing, merge the three-channel images to get x′ t .

3. The image low-light enhancement and joint deblurring method based on a dual-layer planning light and shadow clarity network according to claim 2 is characterized in that: In the block adaptive brightness enhancement module, the image x′ is clustered using the k-means clustering method. t Divide different lighting areas; In the clustering process, the input is the image x′ t , contains the brightness information of each pixel; The output is a set of three types of pixels C i , i ranges from 1 to 3, where C1 represents the area with normal brightness, C2 represents the area with insufficient brightness, and C3 represents the area with overexposure; K-means clustering achieves grouping by optimizing the following objective function: Where x represents image C i A pixel value in , which contains feature information including brightness; μ i Represents the central brightness value of the i-th cluster area; by minimizing the difference between the pixel in each cluster area and the central brightness value of the area, the image x′ t It is divided into three types of lighting areas, so that the brightness characteristics of each type of area are relatively consistent.

4. The image low-light enhancement and joint deblurring method based on a dual-layer planning light and shadow clarity network according to claim 3 is characterized in that: In the block adaptive brightness enhancement module, the brightness of each illuminated area is enhanced using an adaptive brightness enhancement curve to obtain an image The specific process is: An iterative form is adopted. Each iteration continues to improve the brightness based on the previous enhancement, so that the brightness of each illuminated area gradually approaches the ideal value. The iterative formula is as follows: THE n (C i )=THE n-1 (C i )+α i,n THE n-1 (C i )(1-THE n-1 (C i )) Among them, LE n (C i ) represents the illuminated area C after the nth iteration i Brightness value, α i,n is the area C at the nth iteration i The control parameters of area C are gradually improved through iteration. i The overall brightness of the image is gradually increased to the target brightness level, and the enhanced image is finally obtained.

5. The image low-light enhancement and joint deblurring method based on a dual-layer planning light and shadow clarity network according to any one of claims 1 to 4, characterized in that: In the blur kernel parameter optimization module, the blur kernel is updated and the input guide image is deblurred to obtain the image y t The specific process is as follows: First, calculate the image of the Tt-1th iteration The autocorrelation coefficient in, is an image that only retains amplitude information, ⊙ represents the element-wise multiplication operation; the above autocorrelation result is estimated as the initial blur kernel k of the input image t ; Then update the blur kernel: Finally, solve the following formula to get the clear image y t and the optimized blur kernel Among them, λ1 and λ2 are custom hyperparameters. is the convolution operation, To find the gradient operation, For iteration The initial value of .

6. The image low-light enhancement and joint deblurring method based on a dual-layer planning light and shadow clarity network according to claim 5, characterized in that: In the Fourier prior guided module, the image With the image y t Perform image decomposition and update the intermediate clear image, and obtain image x based on the intermediate clear image t-1 The specific process is: For images and y t Use fast Fourier transform to decompose into amplitude A and phase P components: (A t ,P t )=FFT(y t ) In each sampling step, the image is transformed into and y t The amplitude of the deblurred image y t The updated intermediate clear image is obtained by combining the phase of Finally, the image x generated by the diffusion model t and the intermediate clear image estimated from the current time step Combine and get the image x of the next time step t-1 .

7. The image low-light enhancement and joint deblurring method based on a dual-layer planning light and shadow clarity network according to claim 6, characterized in that: In the blur kernel parameter optimization module, based on the intermediate clear image For image y t Update and get image y t-1 , the specific method is: from The updated blur kernel k is estimated from t-1 As the updated result, the blur kernel k is used again t-1 Generate image y t-1 .

8. The image low-light enhancement and joint deblurring method based on a dual-layer planning light and shadow clarity network according to claim 7, characterized in that: In step S2, a two-layer planning training method is adopted, wherein the goal of the upper-layer planning is to optimize the enhancement effect of the adaptive brightness enhancement curve. The upper-layer planning problem is expressed as: minL upper (I enhanced ;θ brightness ) Among them, θ brightness is the iterative adaptive brightness enhancement curve parameter α i,n A collection of enhanced is the final enhanced result image; L upper It is a loss function that measures the visual effect of an image, including indicators of brightness, contrast, and color consistency. The loss consists of three parts: L upper (I enhanced ;θ brightness )=L brightness (I enhanced )+L contrast (I enhanced )+L color (I enhanced ) Among them, L brightness The goal is to ensure that the brightness of the enhanced image conforms to the natural distribution, which is expressed as the average brightness value of the image L mean (I enhanced ) and target brightness L target The differences: L brightness (I enhanced )=||L mean (I enhanced )-L target || 2 L contrast To ensure that the enhanced image has a good visual hierarchy, the image contrast variance or histogram distribution is used to measure the contrast loss: L contrast (I enhanced )=||H(I enhanced )-H target || 2 Where H(I) represents the image histogram, H target is the target histogram; L color To avoid color distortion caused by the enhancement process, the gain G of each channel is adjusted based on the grayscale world assumption to ensure color consistency loss. r ,G g ,G b , so that the average value of the RGB three channels is close to gray: Where, are the red, green, and blue channels of the enhanced image, respectively; The goal of the lower-level planning is to fine-tune the parameters of LoRA so that the image generated by the diffusion model is consistent with the prior of the input image in the Fourier domain, thereby ensuring that the image structure information is not lost; the lower-level planning problem is expressed as: minL Lower (I generated ;θ LoRA ) Specifically: L lower (I generated ;θ LoRA )=L Fourier (I generated ,I input )+L deblur (I generated ) I generated Is the output image of the diffusion model at each time step. In order to ensure that the generated image I generated The amplitude and phase of the input image I input Try to be consistent in the Fourier domain and introduce the Fourier domain consistency loss L Fourier : L Fourier (I generated ,I input )=||A generated -A input || 2 +λ||P generated -P input || 2 Among them, A generated and A input are the Fourier magnitudes of the generated image and the input image, P generated and P input are the Fourier phases of the generated image and the input image, respectively, and λ is the weight for controlling the amplitude and phase loss; L deblur To achieve deblurring loss, we need to ensure that the generated image is clear and has good deblurring effect, as follows: in Represents the gradient operation, which is used to capture the edge and structural features of the image.

Images