Image low-light enhancement joint deblurring method based on bilevel planning light and shadow clearness network

Abstract

The invention provides an image low-light enhancement joint deblurring method based on a bilevel planning light and shadow clearness network, and belongs to the field of computer vision. According to the method, a block adaptive brightness enhancement module, a fuzzy kernel parameter optimization module and a Fourier prior guide module are introduced into a diffusion model; the image is divided into different illumination areas by the block adaptive brightness enhancement module, and differentiation processing is performed on each illumination area, so that the naturalness and smoothness of image brightness adjustment are ensured, and the problem of uneven brightness or unnatural transition caused by global processing is avoided; the Fourier prior guide module processes the brightness and structure information of the image through amplitude and phase decomposition of a Fourier domain, so that joint optimization of brightness enhancement and deblurring is realized, and simultaneous improvement of image definition and brightness is ensured; through a double-layer planning framework, optimization of an image enhancement curve by an upper layer and domain adaptive optimization of a LoRA parameter by a lower layer are realized, and global optimum of the generated image in brightness and definition is ensured.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, and in particular relates to an image low-light enhancement and joint deblurring method based on a double-layer planning light and shadow clarity network. Background Art

[0002] Image enhancement and deblurring refers to processing low-quality images through algorithms to improve the visual effects of the image, such as brightness, contrast, and clarity. In particular, images taken in low-light environments usually have problems such as blur, noise, and loss of details. Therefore, enhancing the brightness and clarity of the image is an important task in image processing. Low-light image enhancement and deblurring technology is widely used in smartphone photography, video surveillance, unmanned driving and other fields.

[0003] With the advancement of deep learning technology, a large number of methods for low-light image enhancement and deblurring have been proposed in recent years. Existing low-light enhancement methods and deblurring algorithms usually deal with brightness and blurring problems separately, lacking the ability to jointly process the two. This separate processing method may result in image blurring still existing after brightness enhancement, or loss of brightness effect during the deblurring process, making it difficult to achieve the best results in both aspects at the same time.

[0004] With the advancement of technology, some methods have begun to handle the tasks of low-light enhancement and deblurring at the same time. LED Net proposes a network structure that combines low-light enhancement and deblurring to solve the problems of blur and insufficient brightness in low-light environments. This method uses paired data of low-light blurry images and normal-light clear images for supervised learning, and jointly optimizes image brightness and clarity, making it robust when facing actual scenes. Although LED Net performs well in the joint task of low-light enhancement and deblurring, it relies on a large amount of paired data for supervised learning. This method is easily limited by the size of the data set in practical applications, especially for real-world low-light blurry scenes. It is difficult to obtain a large amount of high-quality paired data, resulting in insufficient model generalization ability.

[0005] Many existing methods use global processing strategies in the brightening process. This global approach easily leads to over-enhancement or under-enhancement of some areas, and it is impossible to perform differentiated processing according to different lighting areas in the image. Especially in images with large lighting changes, the transition between bright and dark areas is unnatural. In addition, without combining data from specific scenes for domain adaptive adjustment, it is difficult for the model to respond effectively to the characteristics of different scenes, resulting in poor results when migrating between domains. This global processing strategy that lacks fine-tuning is particularly inadequate in scenes with complex and changeable lighting, and is unable to adjust to the differentiated needs of each region, affecting the overall quality of image enhancement.

[0006] When processing images in complex low-light environments or with severe motion blur, existing technologies have difficulty ensuring the naturalness and consistency of images. In particular, in scenes with extremely uneven lighting and severe blur, the generated images may be over-enhanced, with loss of details or residual blur, which cannot meet the requirements of high-quality image enhancement. Therefore, the present invention proposes a low-light image enhancement and joint deblurring method based on a double-layer planning light and shadow clarity network. Summary of the invention

[0007] In view of the shortcomings of the prior art, the present invention proposes a method for low-light enhancement and joint deblurring of images based on a double-layer planning light and shadow clarity network, aiming to solve the problems of independent processing, global processing, and reliance on paired data in the existing methods when processing low-light enhancement and deblurring. Relevant experiments have proved that the present invention can effectively improve the brightness and clarity of images and the processing effect in complex scenes, while improving the computational efficiency of the algorithm.

[0008] The technical solution adopted by the present invention is:

[0009] A method for joint deblurring of low-light image enhancement based on a two-layer planning light and shadow clarity network, comprising:

[0010] S1. Construct a Two-Layer Optimization Network for Light and Sharpness (TOLS-Net), including a diffusion model, a block-based adaptive brightness enhancement module, a blur kernel parameter optimization module, and a Fourier prior guidance module.

[0011] The diffusion model performs iterative processing on the input Gaussian noise image to obtain the image x t , and for the image x t Remove noise and re-weight the image x 0|t , where t∈T; the block adaptive brightness enhancement module is used to enhance the image x obtained by the diffusion model 0|t Perform grayscale world prior processing to obtain image x t ', and the image x t 'Divide different lighting areas, use adaptive brightness enhancement curve to enhance the brightness of each lighting area, and get the image The blur kernel parameter optimization module deblurs the input guide image to obtain the image y t ; The Fourier prior guidance module converts the image With the image y t Perform image decomposition and update the intermediate clear image, and obtain image x based on the intermediate clear image t-1 The blur kernel parameter optimization module is based on the intermediate clear image to image y t+1and blur kernel Update and get image y t and blur kernel Used to guide the image x in the diffusion model t-1 Iterative update; the weight of the diffusion model is fine-tuned using the LoRA method;

[0012] S2, using the LOL-Blur dataset to train the two-layer regularized light and shadow clarity network (TOLS-Net) constructed in S1; the training method adopts a two-layer planning method, where the goal of the upper layer planning is to optimize the parameters of the adaptive brightness enhancement curve, and the goal of the lower layer planning is to fine-tune the parameters of LoRA;

[0013] S3: Input the low-light blurry image to be processed into the network model trained in S2 to obtain a normal brightness and clear image.

[0014] Furthermore, in the block adaptive brightness enhancement module, the image x obtained by the diffusion model is 0|t Perform grayscale world prior processing to obtain image x t ', the specific process is:

[0015] In natural scenes, uneven lighting often leads to color deviation in images. In order to ensure color balance before image enhancement, the grayscale world prior is introduced. The grayscale world assumes that under normal lighting conditions, the average value of the red, green, and blue (R, G, B) channels of the image should be close to gray (that is, the average values ​​of the three channels are equal). By adjusting the gain coefficient of each channel to achieve this balance, color distortion can be removed; the calculation formula of the gain coefficient is as follows:

[0016]

[0017] Among them, G r ,G g ,G b are the gain coefficients of R, G, and B channels respectively, μ r ,μ g ,μ b is the average value of each channel, μ gray is the target grayscale value. Through this step, the brightness of each color channel can be balanced, thereby providing a uniform input for the subsequent enhancement step.

[0018] The adjusted pixel values ​​of each channel are

[0019]

[0020] After processing, merge the three-channel images to get x t';The processed image is more balanced in color, providing a good input for subsequent brightness enhancement.

[0021] Furthermore, in the block adaptive brightness enhancement module, the image x t 'Divide different lighting areas, the specific process is:

[0022] Use k-means clustering method to cluster image x' t Different illumination areas are divided, corresponding to areas with normal brightness, insufficient brightness, and overexposure; in the clustering process, the input is the image x' t , contains the brightness information of each pixel; the output is a set of three types of pixels C i , i ranges from 1 to 3, where C1 represents an area with normal brightness, C2 represents an area with insufficient brightness, and C3 represents an area with overexposure;

[0023] The goal of k-means clustering is to divide pixels into regions with similar brightness and minimize the difference between the pixels in each region and the brightness value μ of the center of the region. i To this end, k-means clustering achieves grouping by optimizing the following objective function:

[0024]

[0025] Where x represents image C i A pixel value in contains characteristic information such as brightness; μ i Represents the central brightness value of the i-th cluster area; by minimizing the difference between the pixels in each cluster area and the central brightness value of the area, the image x' t It is divided into three types of lighting areas, so that the brightness characteristics of each type of area are relatively consistent.

[0026] Furthermore, in the block adaptive brightness enhancement module, an adaptive brightness enhancement curve is used to enhance the brightness of each illumination area to obtain an image The specific process is:

[0027] An iterative form is adopted. Each iteration continues to improve the brightness based on the previous enhancement, so that the brightness of each illuminated area gradually approaches the ideal value. The iterative formula is as follows:

[0028] LE n (C i )=LE n-1 (C i )+α i,n LE n-1 (C i )(1-LE n-1 (C i ))

[0029] Among them, LE n (C i ) represents the illuminated area C after the nth iteration i Brightness value, LE n-1 (C i ) is the illuminated area C after the previous iteration i Brightness value, α i,n is the control parameter at the nth iteration; through iteration, this formula will gradually improve the area C i The overall brightness of the image is gradually increased to the target brightness level, and the enhanced image is finally obtained.

[0030] Furthermore, in the blur kernel parameter optimization module, the input guide image is deblurred to obtain the image y t , and update the blur kernel. The specific process is as follows:

[0031] First, calculate the image of the Tt-1th iteration The autocorrelation coefficient A(|P yt+1 |):

[0032]

[0033] in, is an image that only retains amplitude information, ⊙ represents the element-by-element multiplication operation; the above autocorrelation result is estimated as the initial blur kernel k of the input image t ;

[0034] Then update the blur kernel:

[0035]

[0036] Finally, the method of Pan et al. is used to solve the following formula to obtain the clear image y t and the optimized blur kernel

[0037]

[0038] Among them, λ1 and λ2 are custom hyperparameters. is the convolution operation, To find the gradient operation, For iteration The initial value of .

[0039] Furthermore, in the Fourier prior guidance module, the image With the image y t Perform image decomposition and update the intermediate clear image, and obtain image x based on the intermediate clear image t-1, the specific process is:

[0040] For images and t The fast Fourier transform (FFT) is used to decompose it into amplitude A and phase P components:

[0041]

[0042] (A t ,P t )=FFT(y t )

[0043] In each sampling step, the image is transformed into and t The amplitude of is superimposed and combined with the deblurred image y t The updated intermediate clear image is obtained by combining the phase of

[0044]

[0045] This method effectively preserves the details and clarity of low-light blurry images while maintaining the image brightness enhancement effect, thereby gradually generating high-quality output images.

[0046] Finally, the image x generated by the diffusion model t and the intermediate clear image estimated from the current time step Combine them to get the image x at the current time step t-1 .

[0047] Furthermore, in the blur kernel parameter optimization module, based on the intermediate clear image For image y t Update and get image y t-1 The specific method is: from The updated blur kernel k is estimated from t-1 As the updated result, the blur kernel k is used again t-1 Generate image y t-1 .

[0048] Furthermore, in step S2, a two-layer planning training method is used to achieve multi-objective optimization in low-light enhancement and deblurring tasks; wherein the goal of the upper-layer planning is to optimize the enhancement effect of the adaptive brightness enhancement curve, and the upper-layer planning problem can be expressed as:

[0049] lm upper (I enhanced θ brightness )

[0050] Among them, θ brightnessis the adaptive brightness enhancement curve parameter α i,n A collection of enhanced is the final enhanced result image; L upper It is a loss function that measures the visual effect of an image, including indicators such as brightness, contrast, and color consistency. The loss consists of three parts:

[0051] L upper (I enhanced θ brightness )=L brightness (I enhanced )+L contrast (I enhanced )+L color (I enhanced )

[0052] Among them, L brightness The goal is to ensure that the brightness of the enhanced image conforms to the natural distribution, which can be expressed as the average brightness value L of the image. mean (I enhanced ) and target brightness L target The differences:

[0053] L brightness (I enhanced )=||L mean (I enhanced )-L target || 2

[0054] L contrast To ensure that the enhanced image has a good visual hierarchy, the image contrast variance or histogram distribution is used to measure the contrast loss:

[0055] L contrast (I enhanced )=||H(I enhanced )-H target || 2

[0056] Where H(I) represents the image histogram, H target is the target histogram;

[0057] L color To avoid color distortion caused by the enhancement process, the gain G of each channel is adjusted based on the grayscale world assumption to ensure color consistency loss. r ,G g ,G b , so that the average value of the RGB three channels is close to gray:

[0058]

[0059] In the formula, are the red, green, and blue channels of the enhanced image, respectively;

[0060] The goal of the lower-level planning is to fine-tune the parameters of LoRA so that the image generated by the diffusion model is consistent with the prior of the input image in the Fourier domain, thereby ensuring that the image structure information is not lost; in the Fourier prior guidance module, the image is decomposed into amplitude A and phase P, and the phase reflects the structural information of the image. By fine-tuning the parameter matrix, the amplitude adjustment of the generated image can be controlled to ensure consistency with the prior of the input image in the Fourier domain; the lower-level planning problem can be expressed as:

[0061] lm Lower (I generated θ LoRA )

[0062] Specifically:

[0063] L lower (I generated θ LoRA )=L Fourier (I generated ,I input )+L deblur (I generated )

[0064] I generated is the output image of the diffusion model at each time step. In order to ensure that the generated image I generated The amplitude (brightness information) and phase (structural information) of input As consistent as possible in the Fourier domain, the Fourier domain consistency loss L is introduced Fourier :

[0065] L Fourier (I generated ,I input )=||A generated -A input || 2 +λ||P generated -P input || 2

[0066] Among them, A generated and A input are the Fourier amplitudes of the generated image and the input image, respectively, generated and P input are the Fourier phases of the generated image and the input image, respectively, and λ is the weight for controlling the amplitude and phase loss;

[0067] L deblur To deblur the loss, ensure that the generated image is clear and the deblurring effect is good, as follows:

[0068]

[0069] in Represents the gradient operation, which is used to capture the edge and structural features of the image.

[0070] Beneficial effects of the present invention:

[0071] 1) Joint optimization of low-light enhancement and deblurring: The existing technology usually handles the brightness enhancement and deblurring of images separately, but lacks the ability to jointly optimize the two, resulting in blurring still existing after brightness enhancement, or loss of brightness effect during deblurring. The present invention processes the brightness and structural information of the image separately through amplitude and phase decomposition in the Fourier domain, achieving joint optimization of brightness enhancement and deblurring, which can effectively remove blurring while enhancing brightness, ensuring that both image clarity and brightness are improved;

[0072] 2) Adaptive illumination area processing: The existing methods adopt global processing strategies, which easily lead to over-enhancement or under-enhancement of some areas of the image, especially in scenes with large illumination changes. The present invention divides the image into different illumination areas through k-means clustering technology, and performs differentiated processing on each illumination area to ensure the naturalness and smoothness of image brightness adjustment, avoiding the problem of uneven brightness or unnatural transition caused by global processing;

[0073] 3) Domain adaptive fine-tuning: Existing technologies usually lack adaptive fine-tuning for different domains, especially when dealing with scenes with complex lighting changes, the model performs poorly. This invention uses LoRA (low-rank adaptation) technology to perform domain adaptive fine-tuning in the diffusion model, so that the model can quickly adapt to different lighting conditions and scenes, improve the robustness and consistency of image enhancement, and especially show stronger adaptability in complex and changing lighting conditions;

[0074] 4) Global optimization of two-layer planning: Through the two-layer planning framework, the present invention realizes the optimization of the image enhancement curve in the upper layer and the domain adaptive optimization of the LoRA parameters in the lower layer, ensuring the global optimization of the generated image in terms of brightness and clarity. It is difficult for the existing technology to take into account both illumination enhancement and structure preservation at the same time, while the present invention effectively maintains the structural information of the image while improving the image brightness, solving the problem of detail loss after brightness enhancement that is common in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Figure 1 This is a global flow chart of an embodiment of the present invention.

[0076] Figure 2 The figure is a schematic diagram of adaptive adjustment based on Lora domain in an embodiment of the present invention. DETAILED DESCRIPTION

[0077] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the specific implementation modes of the present invention will be described in detail below with reference to the accompanying drawings.

[0078] like Figure 1 As shown, an embodiment of the present invention provides an image low-light enhancement joint deblurring method based on a double-layer planning light and shadow clarity network, comprising:

[0079] Step 1, use the LOL-Blur dataset, which is the first large-scale dataset for the joint task of low-light enhancement and deblurring. It contains 12,000 pairs of synthetic low-light blurry images paired with normal-light clear images, and there are various low-light levels and motion blur modes in the dataset.

[0080] Step 2, construct a two-layer regularization light and sharpness network (Two-Layer Optimization Network for Light and Sharpness, referred to as TOLS-Net), including a diffusion model, a block adaptive brightness enhancement module, a blur kernel parameter optimization module and a Fourier prior guidance module.

[0081] The diffusion model performs iterative processing on the input Gaussian noise image to obtain the image x t , the specific process is as follows:

[0082] First, in the forward process, the diffusion model gradually adds Gaussian noise to the image, so that it is finally transformed into a pure noise image; this process can be expressed as a Markov chain, and each step degrades the image by adding a small amount of noise. The formula is:

[0083]

[0084] Among them, x t represents the image state at the tth step in the diffusion process, β t is the noise intensity at step t. By gradually adding noise, x T It becomes a pure noise image.

[0085] In the reverse process, the model generates a high-quality output image by gradually restoring a clear image from a noisy image; the conditional distribution of the reverse process is:

[0086] p θ (x t-1 |x t )=N(x t-1 ;μ θ (x t ,t),Σ θ (x t ,t))

[0087] Among them, μ θ (x t ,t) and Σ θ (x t ,t) is the mean and variance estimated by the neural network, guiding the model denoising process; the mean μ θ (x t ,t) can be further decomposed into:

[0088]

[0089] Among them, α t =1-β t ; is a cumulative term representing the weight calculated from the predefined noise schedule; ε θ (x t ,t) is the predicted noise.

[0090] The final loss function is optimized by minimizing the mean square error between the model predicted noise and the actual noise, so that the model can effectively restore the clear image during the denoising process.

[0091] The diffusion model lasts for T steps. After the denoising process of the diffusion model is completed, the generated image has a certain degree of clarity and brightness improvement. However, due to the uneven illumination in low-light and blurred scenes, the brightness of different areas needs to be further enhanced to achieve a more natural effect. To this end, in the iterative process, the image x obtained by the Ttth step iteration is selected. t Enter the block adaptive brightness enhancement module for processing, where t is a custom number, usually half of the total number of iterations. The specific process is as follows:

[0092] First, the image x obtained by the diffusion model 0|t Perform grayscale world prior processing to obtain image x t ', as follows:

[0093] In natural scenes, uneven lighting often leads to color deviation in images. In order to ensure color balance before image enhancement, the grayscale world prior is introduced. The grayscale world assumes that under normal lighting conditions, the average value of the red, green, and blue (R, G, B) channels of the image should be close to gray (that is, the average values ​​of the three channels are equal). By adjusting the gain coefficient of each channel to achieve this balance, color distortion can be removed; the calculation formula of the gain coefficient is as follows:

[0094]

[0095] Among them, G r ,G g ,G bare the gain coefficients of R, G, and B channels respectively, μ r ,μ g ,μ b is the average value of each channel, μ gray is the target grayscale value. Through this step, the brightness of each color channel can be balanced, thereby providing a uniform input for the subsequent enhancement step.

[0096] The adjusted pixel values ​​of each channel are

[0097]

[0098] After processing, merge the three-channel images to get x t ';The processed image is more balanced in color, providing a good input for subsequent brightness enhancement.

[0099] Next, we use the k-means clustering method to cluster the image x t 'Divide different lighting areas, corresponding to areas with normal brightness, insufficient brightness and overexposure; in the clustering process, the input is image x t ', contains the brightness information of each pixel; the output is a set of three types of pixels C i , i ranges from 1 to 3, where C1 represents an area with normal brightness, C2 represents an area with insufficient brightness, and C3 represents an area with overexposure;

[0100] The goal of k-means clustering is to divide pixels into regions with similar brightness and minimize the difference between the pixels in each region and the brightness value μ of the center of the region. i To this end, k-means clustering achieves grouping by optimizing the following objective function:

[0101]

[0102] Where x represents image C i A pixel value in contains characteristic information such as brightness; μ i Represents the central brightness value of the i-th cluster area; by minimizing the difference between the pixels in each cluster area and the central brightness value of the area, the image x' t It is divided into three types of lighting areas, so that the brightness characteristics of each type of area are relatively consistent.

[0103] Then, an adaptive brightness enhancement curve is used to enhance the brightness of each illuminated area to obtain an image Specifically:

[0104] An iterative form is adopted. Each iteration continues to improve the brightness based on the previous enhancement, so that the brightness of each illuminated area gradually approaches the ideal value. The iterative formula is as follows:

[0105] LE n (C i )=LE n-1 (C i )+α i,n LE n-1 (C i )(1-LE n-1 (C i ))

[0106] Among them, LE n (C i ) represents the illuminated area C after the nth iteration i Brightness value, LE n-1 (C i ) is the illuminated area C after the previous iteration i Brightness value, α i,n is the control parameter at the nth iteration; through N iterations, this formula will gradually improve the area C i The overall brightness of the image is gradually increased to the target brightness level, and the enhanced image is finally obtained.

[0107] In order to realize the guidance of the low-light blurred image y to the diffusion model, the blur kernel parameter optimization module is used to deblur the input guided image y to obtain the image y t , the specific process is:

[0108] First, calculate the image of the Tt-1th iteration The autocorrelation coefficient

[0109]

[0110] in, is an image that only retains amplitude information, ⊙ represents the element-by-element multiplication operation; the above autocorrelation result can be approximately estimated as the initial blur kernel k of the input image t .

[0111] Then update the blur kernel:

[0112]

[0113] Finally, we use the method of Pan et al. (Pan L, Hartley R, Liu M, et al. Phase-only image-based kernel estimation for single image blind deblurring [C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2019: 6034-6043.) to solve the following formula and get the clear image y t and the optimized blur kernel

[0114]

[0115] Among them, λ1 and λ2 are custom hyperparameters. is the convolution operation, To find the gradient operation, For iteration The initial value of .

[0116] In the Fourier prior guidance module, the image obtained by the block adaptive brightness enhancement module The image y obtained by the fuzzy kernel parameter optimization module t Perform image decomposition and update the intermediate clear image, and obtain image x based on the intermediate clear image t-1 , the specific process is:

[0117] For images and t The fast Fourier transform (FFT) is used to decompose it into amplitude A and phase P components:

[0118]

[0119] (A t ,P t )=FFT(y t )

[0120] In each sampling step, the image is transformed into and t The amplitude of is superimposed and combined with the deblurred image y t The updated intermediate clear image is obtained by combining the phase of

[0121]

[0122] This method effectively preserves the details and clarity of low-light blurry images while maintaining the image brightness enhancement effect, thereby gradually generating high-quality output images. Compared to y t There are clearer boundaries.

[0123] Finally, the image x generated by the diffusion model t and the intermediate clear image estimated from the current time step Combine them to get the image x at the current time step t-1 (Ho, J., Jain, A., & Abbeel, P. (2020). Denoising DiffusionProbabilistic Models. Advances in Neural Information Processing Systems, 33, 6840–6851.).

[0124] The blur kernel parameter optimization module is again based on the intermediate clear image Estimate the updated blur kernel k t-1 As the updated result, the blur kernel k is used again t-1 Generate image y t-1 , thereby guiding the image x in the diffusion model t-1 Iterative updates.

[0125] Step 3, using the LOL-Blur dataset of step 1 to train the two-layer optimized light and shadow clarity network (TOLS-Net) constructed in step 2; the training method adopts a two-layer programming method, wherein the goal of the upper-layer programming is to optimize the enhancement effect of the adaptive brightness enhancement curve, and the upper-layer programming problem can be expressed as:

[0126] min L upper (I enhanced θ brightness )

[0127] Among them, θ brightness is the adaptive brightness enhancement curve parameter α i,n A collection of enhanced is the final enhanced result image; L upper It is a loss function that measures the visual effect of an image, including indicators such as brightness, contrast, and color consistency. The loss consists of three parts:

[0128] L upper (I enhanced θ brightness )=L brightness (I enhanced )+L contrast(I enhanced )+L color (I enhanced )

[0129] Among them, L brightness The goal is to ensure that the brightness of the enhanced image conforms to the natural distribution, which can be expressed as the average brightness value L of the image. mean (I enhanced ) and target brightness L target The differences:

[0130] L brightness (I enhanced )=||L mean (I enhanced )-L target || 2

[0131] L contrast To reduce the contrast loss, we ensure that the enhanced image has a good visual hierarchy. The variance or histogram distribution of image contrast is often used to measure it:

[0132] L contrast (I enhanced )=||H(I enhanced )-H target || 2

[0133] Where H(I) represents the image histogram, H target is the target histogram;

[0134] L color To avoid color distortion caused by the enhancement process, the gain G of each channel is adjusted based on the grayscale world assumption to ensure color consistency loss. r ,G g ,G b , so that the average value of the RGB three channels is close to gray:

[0135]

[0136] In the formula, They are the red, green and blue channels of the enhanced image respectively.

[0137] The goal of the lower-level planning is to fine-tune the parameters of LoRA so that the image generated by the diffusion model is consistent with the prior of the input image in the Fourier domain, thereby ensuring that the image structure information is not lost; in the Fourier prior guidance module, the image is decomposed into amplitude A and phase P, and the phase reflects the structural information of the image. By fine-tuning the parameter matrix, the amplitude adjustment of the generated image can be controlled to ensure consistency with the prior of the input image in the Fourier domain; the lower-level planning problem can be expressed as:

[0138] min L Lower (I generated θ LoRA )

[0139] Specifically:

[0140] L lower (I generated θ LoRA )=L Fourier (I generated ,I input )+L deblur (I generated )

[0141] I generated is the output image of the diffusion model at each time step. In order to ensure that the generated image I generated The amplitude (brightness information) and phase (structural information) of input As consistent as possible in the Fourier domain, the Fourier domain consistency loss L is introduced Fourier :

[0142] L Fourier (I generated ,I input )=||A generated -A input || 2 +λ||P generated -P input || 2

[0143] Among them, A generated and A input are the Fourier amplitudes of the generated image and the input image, respectively, generated and P input are the Fourier phases of the generated image and the input image, respectively, and λ is the weight for controlling the amplitude and phase loss;

[0144] L deblur To deblur the loss, ensure that the generated image is clear and the deblurring effect is good, as follows:

[0145]

[0146] in Represents the gradient operation, which is used to capture the edge and structural features of the image.

[0147] Step 4: Input the low-light blurry image to be processed into the network model trained in step 3 to obtain a clear image with normal brightness.

[0148] The above-described embodiments are all preferred implementations of the present invention, and do not limit the present invention in other forms. Any technician familiar with the profession may use the above content to make changes or follow suit. However, any changes made to the above embodiments based on the essence of the method of the present invention without departing from the content of the scheme in the present invention still fall within the scope of protection of the present invention.

Claims

1. A joint deblurring method for low-light enhancement of images based on a two-layer planning light and shadow clarity network, characterized in that: include: S1. Construct a two-layer planning light and shadow clarity network, which includes a diffusion model, a block adaptive brightness enhancement module, a blur kernel parameter optimization module, and a Fourier prior guidance module; The diffusion model performs iterative processing on the input Gaussian noise image to obtain the image x t , and for the image x t Remove noise and re-weight the image x 0|t , where t∈T; The block adaptive brightness enhancement module is used to enhance the image x obtained by the diffusion model. 0|t Perform grayscale world prior processing to obtain image x′ t , and the image x′ t Different illumination areas are divided, and the brightness of each illumination area is enhanced using an adaptive brightness enhancement curve to obtain an image The blur kernel parameter optimization module deblurs the input guide image to obtain the image y t ; The Fourier prior guidance module converts the image With the image y t Perform image decomposition and update the intermediate clear image, and obtain image x based on the intermediate clear image t-1 The blur kernel parameter optimization module is based on the intermediate clear image to image y t+1 and blur kernel Update and get image y t and blur kernel Used to guide the image x in the diffusion model t-1 Iterative update; the weight of the diffusion model is fine-tuned using the LoRA method; S2. The LOL-Blur dataset is used to train the double-layer regularized light and shadow clarity network constructed in S1. The training method adopts a double-layer planning method, wherein the goal of the upper-layer planning is to optimize the parameters of the adaptive brightness enhancement curve, and the goal of the lower-layer planning is to fine-tune the parameters of LoRA. S3: Input the low-light blurry image to be processed into the network model trained in S2 to obtain a normal brightness and clear image.

2. The image low-light enhancement joint deblurring method based on a double-layer planning light and shadow clarity network according to claim 1 is characterized in that: In the block adaptive brightness enhancement module, the image x obtained by iterating the diffusion model is 0|t Perform grayscale world prior processing to obtain image x′ t , the specific process is: Grayscale world assumes that under normal lighting conditions, the average values ​​of the three channels of R, G, and B of the image should be close to gray, that is, the average values ​​of the three channels are equal. By adjusting the gain coefficient of each channel, it can reach this balance state, thereby removing color distortion; the calculation formula of the gain coefficient is as follows: Among them, G r ,G g ,G b are the gain coefficients of the R, G, and B channels, μ r ,μ g ,μ b is the average value of each channel, μ gray is the target gray value; The adjusted pixel values ​​of each channel are After processing, merge the three-channel images to get x′ t .

3. The image low-light enhancement joint deblurring method based on a double-layer planning light and shadow clarity network according to claim 2 is characterized in that: In the block adaptive brightness enhancement module, the image x′ is clustered using the k-means clustering method. t Divide different lighting areas; In the clustering process, the input is the image x′ t , contains the brightness information of each pixel; The output is a set of three types of pixels C i , i ranges from 1 to 3, where C1 represents an area with normal brightness, C2 represents an area with insufficient brightness, and C3 represents an area with overexposure; K-means clustering achieves grouping by optimizing the following objective function: Where x represents image C i A pixel value in contains characteristic information including brightness; μ i Represents the central brightness value of the i-th cluster area; by minimizing the difference between the pixels in each cluster area and the central brightness value of the area, the image x′ t It is divided into three types of lighting areas, so that the brightness characteristics of each type of area are relatively consistent.

4. The image low-light enhancement joint deblurring method based on a double-layer planning light and shadow clarity network according to claim 3 is characterized in that: In the block adaptive brightness enhancement module, an adaptive brightness enhancement curve is used to enhance the brightness of each illuminated area to obtain an image The specific process is: An iterative form is adopted. Each iteration continues to improve the brightness based on the previous enhancement, so that the brightness of each illuminated area gradually approaches the ideal value. The iterative formula is as follows: THE n (C i )=THE n-1 (C i )+α i,n THE n-1 (C i )(1-THE n-1 (C i )) Among them, LE n (C i ) represents the illuminated area C after the nth iteration i Brightness value, α i,n is the area C at the nth iteration i The control parameters of area C are gradually improved through iteration. i The overall brightness of the image is gradually increased to the target brightness level, and the enhanced image is finally obtained.

5. The image low-light enhancement and joint deblurring method based on a double-layer planning light and shadow clarity network according to any one of claims 1 to 4, characterized in that: In the blur kernel parameter optimization module, the blur kernel is updated, and the input guide image is deblurred to obtain the image y t , the specific process is as follows: First, calculate the image of the Tt-1th iteration The autocorrelation coefficient in, is an image that only retains amplitude information, ⊙ represents the element-by-element multiplication operation; the above autocorrelation result is estimated as the initial blur kernel k of the input image t ; Then update the blur kernel: Finally, solve the following formula to get the clear image y t and the optimized blur kernel Among them, λ1 and λ2 are custom hyperparameters. is the convolution operation, To find the gradient operation, For iteration The initial value of .

6. The image low-light enhancement joint deblurring method based on a double-layer planning light and shadow clarity network according to claim 5, characterized in that: In the Fourier prior guidance module, the image With the image y t Perform image decomposition and update the intermediate clear image, and obtain image x based on the intermediate clear image t-1 , the specific process is: For images and t The fast Fourier transform is used to decompose them into amplitude A and phase P components: (A t ,P t )=FFT(y t ) In each sampling step, the image is transformed into and t The amplitude of is superimposed and combined with the deblurred image y t The updated intermediate clear image is obtained by combining the phase of Finally, the image x generated by the diffusion model t and the intermediate clear image estimated from the current time step Combine and get the image x of the next time step t-1 .

7. The image low-light enhancement joint deblurring method based on a double-layer planning light and shadow clarity network according to claim 6, characterized in that: In the blur kernel parameter optimization module, based on the intermediate clear image For image y t Update and get image y t-1 The specific method is: from The updated blur kernel k is estimated from t-1 As the updated result, the blur kernel k is used again t-1 Generate image y t-1 .

8. The image low-light enhancement joint deblurring method based on a double-layer planning light and shadow clarity network according to claim 7, characterized in that: In step S2, a double-layer planning training method is adopted, wherein the upper-layer planning goal is to optimize the enhancement effect of the adaptive brightness enhancement curve, and the upper-layer planning problem is expressed as: minL upper (I enhanced ;θ brightness ) Among them, θ brightness is the iterative adaptive brightness enhancement curve parameter α i,n A collection of enhanced is the final enhanced result image; L upper It is a loss function that measures the visual effect of an image, including indicators of brightness, contrast, and color consistency. The loss consists of three parts: L upper (I enhanced ;θ brightness )=L brightness (I enhanced )+L contrast (I enhanced )+L color (I enhanced ) Among them, L brightness The goal is to ensure that the brightness of the enhanced image conforms to the natural distribution, which is expressed as the average brightness value of the image L mean (I enhanced ) and target brightness L target The differences: L brightness (I enhanced )=||L mean (I enhanced )-L target || 2 L contrast To ensure that the enhanced image has a good visual hierarchy, the image contrast variance or histogram distribution is used to measure the contrast loss: L contrast (I enhanced )=||H(I enhanced )-H target || 2 Where H(I) represents the image histogram, H target is the target histogram; L color To avoid color distortion caused by the enhancement process, the gain G of each channel is adjusted based on the grayscale world assumption to ensure color consistency loss. r ,G g ,G b , so that the average value of the RGB three channels is close to gray: In the formula, are the red, green, and blue channels of the enhanced image, respectively; The goal of the lower-level planning is to fine-tune the parameters of LoRA so that the image generated by the diffusion model is consistent with the prior of the input image in the Fourier domain, thereby ensuring that the image structure information is not lost; the lower-level planning problem is expressed as: minL Lower (I generated ;θ LoRA ) Specifically: L lower (I generated ;θ LoRA )=L Fourier (I generated ,I input )+L deblur (I generated ) I generated is the output image of the diffusion model at each time step. In order to ensure that the generated image I generated The amplitude and phase of the input image I input Try to be consistent in the Fourier domain and introduce the Fourier domain consistency loss L Fourier : L Fourier (I generated ,I input )=||A generated -A input || 2 +λ||P generated -P input || 2 Among them, A generated and A input are the Fourier amplitudes of the generated image and the input image, respectively, generated and P input are the Fourier phases of the generated image and the input image, respectively, and λ is the weight for controlling the amplitude and phase loss; L deblur To deblur the loss, ensure that the generated image is clear and the deblurring effect is good, as follows: in Represents the gradient operation, which is used to capture the edge and structural features of the image.

Images