Weak light enhancement method based on adaptive sparse attention diffusion model

Abstract

The application discloses a weak light enhancement method based on an adaptive sparse attention diffusion model. First, a normal light image is added with noise to obtain a pure noise picture. Then, a low light image is preprocessed, including curve fine-tuning, initial denoising enhancement and edge detection, to obtain better input. Next, an improved noise prediction module is used to combine the pure noise image and the preprocessed image to obtain a noise feature map. Based on these preparations, a weak light image enhancement network based on a sparse attention diffusion model is constructed. In the reverse process, the denoising formula of the diffusion model is applied to iteratively denoise the pure noise image, and finally an enhanced image is generated. By training the sparse attention diffusion model network, different types of weak light images can be adaptively processed. Overall, this method can significantly improve the quality of weak light images and enhance the details and clarity of the images through effective data processing, noise modeling and adaptive network optimization.

Description

Technical Field

[0001] This invention belongs to the field of computer digital image processing technology, specifically relating to a low-light enhancement method based on an adaptive sparse attention diffusion model. Background Technology

[0002] With the development of digital imaging devices, people can capture images in various environments. However, insufficient light entering the camera severely degrades image quality, resulting in low-light images. These low-light images often suffer from a series of problems, including low visibility, loss of detail, dull colors, and excessive noise. This not only creates an unpleasant visual experience but also poses significant challenges to applications such as night surveillance, machine vision, and mobile photography. Therefore, research on low-light image enhancement methods has become crucial, aiming to improve the quality and visual effects of images captured in low-light environments. Existing low-light image enhancement methods include traditional methods and deep learning-based methods.

[0003] Currently, low-light image enhancement methods based on deep learning can be broadly categorized as follows: methods based on convolutional neural networks (CNNs), methods based on generative adversarial networks (GANs), methods based on attention mechanisms, methods based on residual learning, and methods based on joint training. Although existing low-light image enhancement algorithms have achieved satisfactory results in terms of generation effects, some problems still exist, such as over-enhancement, the introduction of noise, color distortion, and insufficient adaptability to specific scenes. Summary of the Invention

[0004] The purpose of this invention is to provide a low-light enhancement method based on an adaptive sparse attention diffusion model, which solves the problems of over-enhancement, noise amplification, excessive smoothing of texture details, and color distortion that exist in previous low-light image enhancement methods.

[0005] The technical solution adopted in this invention is a low-light enhancement method based on an adaptive sparse attention diffusion model. It involves acquiring normal light images corresponding to low-light images from the LOL dataset, forward diffusion of the normal light images to obtain pure noise images, designing an improved noise prediction module to calculate the noise feature map of the low-light image, designing a low-light enhancement network, and inputting the low-light image x. low Noise feature map ε θ Inverse denoising is performed by using the diffusion model denoising formula for iterative denoising to obtain the final low-light enhancement network. The final low-light enhancement network is then trained by inputting the low-light image to be enhanced to obtain the enhanced low-light image.

[0006] The invention is further characterized by:

[0007] The specific steps are as follows:

[0008] Step 1: Use a diffusion model to perform forward diffusion on the normal light image x0 corresponding to the low-light image in the LOL dataset to obtain the pure noise image x. T ;

[0009] Step 2: Design low-light image x low The preprocessing module consists of a curve fine-tuning module, an initial denoising and enhancement module, and a Canny edge detection module, which processes low-light images x... low Input to the preprocessing module to obtain the preprocessed image x llpm ;

[0010] Step 3: Design an improved noise prediction module. Replace the convolutional layers in the existing noise prediction module with a sparse attention enhancement module, and replace the skip connections in the existing noise prediction module with a noise consistency constraint module. This will improve the noise prediction of the pure noise image x. T and the preprocessed image x llpm The noise feature map ε is obtained by inputting it into the improved noise prediction module. θ ;

[0011] Step 4: Design a low-light enhancement network incorporating a diffusion model to enhance the low-light image x. low Noise feature map ε θ The low-light enhancement network is used for enhancement. During the enhancement process, the denoising formula in the inverse denoising process of the diffusion model is used to denoise the purely noisy image x. T Denoising was performed, and the denoising formula was iterated 2000 times to obtain the final low-light enhancement network;

[0012] Step 5: Use the low-light images and their corresponding normal-light images from the LOL dataset as training samples to train the final low-light enhancement network for 2060 epochs, and obtain the optimal enhancement network model after training.

[0013] Step 6: Input the low-light image to be enhanced into the trained optimal enhancement network model to obtain the enhanced low-light image.

[0014] Step 1 is as follows:

[0015] Select low-light images from the LOL dataset. low The corresponding normal light image x0 is input into the diffusion model, where it is forward diffused. Through this forward diffusion process, the normal light image x0 is transformed into a purely noisy image x. T The forward diffusion process is shown in the formula:

[0016]

[0017]

[0018]

[0019]

[0020] in ε and Ι follow a standard normal distribution N(0,1). The diffusion model is a parameterized Markov chain. Due to its reparameterization characteristics, given x0 and randomly sampled Gaussian noise ε, the pure noise image x is obtained according to formulas (3) and (4). T .

[0021] Step 2 is as follows:

[0022] Step 2.1: Convert the low-light image x low Input the Canny edge detection module to perform edge detection and obtain edge information x canny edge information x canny Blended into the original low-light image x low In the process, the edge information x' of the fused low-light image is obtained. canny The formula for this process is:

[0023]

[0024] Where canny() is the canny edge detector;

[0025] Step 2.2: The initial denoising and enhancement module consists of five convolutional layers forming a dense residual connection block, which denoises the low-light image x. low After input, each convolutional layer is used to extract features from the image, and the residuals between the features are accumulated to obtain the initial denoising and enhancement results;

[0026] Step 2.3: Convert the low-light image x low The number of channels is increased from 3 to 12 through a convolutional layer. Then, the channel is divided into four 3-channel adjustment factors through channel segmentation, which are input into the four iterations of curve fine-tuning. The initial denoising and enhancement result is used as the adjustment target and input into the curve fine-tuning block for iterative enhancement to obtain x. curve The curve fine-tuning module is shown in the following formula.

[0027]

[0028] In the formula, For the initial enhancement results, Here is the formula for curve fine-tuning, where i is the number of iterations. Since there are a total of 4 adjustments with 3 channels, the number of iterations i is 4.

[0029] Step 2.4: Adjust the curve result x curve The result of edge detection x' cannyPerform a simple summation, then multiply the result by... Obtain the preprocessed image x llpm The specific process is shown in the following formula.

[0030]

[0031] Step 3 specifically involves:

[0032] Step 3.1: Design an improved noise prediction module. Replace the convolutional layers in the existing noise prediction module with a sparse attention enhancement module, and replace the skip connections in the existing noise prediction module with a noise consistency constraint module. Then, process the preprocessed image... images with pure noise The input is fed into the improved noise prediction module through channel-level connections. After the connections, the image size remains unchanged, but the number of channels increases from 3 to 6. After a dimensionality increase operation, the input 6-channel tensor is expanded into a 64-channel tensor. The result of the dimensionality increase operation is used for feature extraction through a residual network. During the feature extraction process, the intermediate feature tensors obtained will enter the sparse attention enhancement module while keeping the number of channels and size unchanged.

[0033] Step 3.2: In the sparse attention enhancement module, process the obtained intermediate feature tensor. Group normalization is performed to normalize the input data of different channels, resulting in feature maps.

[0034] Step 3.3: Map the obtained features Perform channel transformation from the original bchw shape to the bhwc shape, then divide the feature maps of the transformed channels into n×n non-overlapping regions, such that each feature map contains m feature regions, where m is calculated using the formula:

[0035]

[0036] Where H and W represent the length and width of the feature map, respectively, and n is the length and width of the m small windows. 2 It is the area of ​​the divided square windows;

[0037] Step 3.4: After dividing the window into smaller windows, it makes... Become The query, key, and value tensors of m small windows are obtained through linear projection calculation. The formula is as follows:

[0038] Q = x m W q K = x m Wk V = x m W v (9);

[0039] in The projection weights for Q, K, and V are respectively.

[0040] Construct a directed graph to determine the most relevant key-value pairs in each region, and calculate the average of Q and K over m small windows to obtain the region-level query and build values. The adjacency matrix between regions is obtained using formula (10).

[0041] A m =Q m (K m ) T (10);

[0042] Adjacency matrix A k To represent the semantic correlation between two regions, we retain the K most crucial pieces of information from each region, resulting in an index matrix I. m The process is as follows:

[0043]

[0044] Obtain the index matrix I m Then, a fine-grained region-by-region attention mechanism is applied. For each query tag in region i, all key-value pairs are processed, and I is collected. m The formulas for k and v are as follows:

[0045] K g =collection(K,I m ),V g =collection(V,I) m ), (12)

[0046] in The k and v tensors after the set are given, and the formula for calculating attention is as follows;

[0047]

[0048] Through nested operations, including dimensionality transformation, linear transformation, attention computation, and shape rearrangement, the final feature representation x is obtained. SAEM ;

[0049] The constant factor is used to prevent weight concentration and attention collapse;

[0050] Step 3.5: Noise suppression and constraint are applied to the output of the sparse attention enhancement module using the noise consistency constraint module. The specific steps are as follows: Convert the feature representation x... SAEM After feature enhancement using noise_block1, x is obtained. noise_block1 ; through residual blocks on x noise_block1 Residual connections and feature extraction are performed, and an attention mechanism is used to further enhance the correlation between features and improve the quality of feature representation. This module yields feature information x. noise_atten ;

[0051] feature information x noise_atten An adaptive filter is used for correction, correcting the previously obtained feature information to obtain x. curve_filt ;

[0052] Take the original input x SAEM x noise_block1 x noise_atten and x curve_filt The features are added together in a weighted manner to obtain the final modified feature block x. ncm , making x SAEM Through multiple feature transformations, residual connections, attention mechanisms, and adaptive filtering, noise in the input data is effectively processed, and features are extracted and corrected to produce more accurate and robust feature representations, which are then passed as more accurate inputs to the next layer's task. The formula is as follows:

[0053] x ncm =x SAEM +x noise_block1 +x noise_atten +x noise_filt (14);

[0054] Step 3.6, x ncm With x SAEM Simple summation, summing the result with respect to x SAEM The upsampling results are connected at the channel level. Each upsampling layer passes through an existing residual connection and a sparse attention enhancement module, and the noise feature map ε is obtained by passing through a ConvBlock output. θ .

[0055] In step 3.5, the feature representation x is... SAEM The specific process of feature enhancement after noise_block1 is as follows: x SAEM The process involves sequentially performing convolution, batch normalization, and ReLU activation, followed by random channel-level dropout on the results. This process is then repeated. These operations help the model predict potential noise in the input data. By learning the weights between different feature transformations, the extracted feature information is further enhanced to obtain x.noise_block1 .

[0056] Step 4 is as follows:

[0057] Step 4.1: In the noise prediction stage, the loss function L2 is used for constraint, and the specific formula is as follows:

[0058]

[0059] Where ε is Gaussian noise that follows a normal distribution, ε θ It is a noise feature map predicted by the noise prediction module;

[0060] Step 4.2: Based on the principle of the inverse process of the existing diffusion model, and combined with the predicted noise, the isotropic standard Gaussian distribution is transformed into the target data distribution. This allows the input low-light image to be iteratively transformed into an enhanced normal-light image. During this process, the existing diffusion model denoising formula is used to denoise the pure noise image using x... T The denoising formula for the reverse process is shown in formula (16):

[0061]

[0062] The mean and variance of noise are derived from the formula for the normal distribution as follows:

[0063]

[0064]

[0065] Where β t =1-α t Since x0 is not available during the testing phase, the approximate value f of x0 is calculated based on formula (4) for the forward diffusion process. θ (x t The specific calculation process is shown in formula (18):

[0066]

[0067] The approximate value of x0 calculated by formula (17) is used to replace x0 in formula (16) to obtain formula (19). In order to apply it to low-light image enhancement, a condition x is added to the inverse process. llpm To guide the reverse denoising process, so as to achieve low-light image enhancement, formula (19) becomes formula (20). By iterating formula (20) until t=0, the enhanced image is obtained.

[0068]

[0069]

[0070] Step 4.3: In the reverse process, iterate formula (20) 2000 times to obtain the final enhanced network model.

[0071] Step 5 is as follows:

[0072] Step 5.1: Use the low-light images and their corresponding normal-light images from the LOL dataset as training samples. Train the network on a GPU with 12GB of VRAM. Use the existing ADAM optimizer as the network optimizer and set the initial learning rate to 1×10⁻⁶. -4 The scheduler strategy is a multi-step decay strategy with decay steps of 25 and 100 respectively, decaying by half each time, and the number of iterations is set to 1,000,000, with 200 iterations per iteration, for a total of 2067 epochs of training.

[0073] Step 5.2: Save the neural network parameters trained in Step 5.1, the number of epochs being trained, the number of iterations, the optimizer ADAM, and the scheduler to obtain the optimal augmented network model after training.

[0074] The beneficial effects of this invention are:

[0075] This invention presents a low-light image enhancement method based on an adaptive sparse attention diffusion model. The method first adds noise to a normal light image to obtain a purely noisy image. Then, it preprocesses the low-light image, including curve fine-tuning, initial denoising enhancement, and edge detection, to obtain better input. An improved noise prediction module combines the purely noisy image and the preprocessed image to obtain a noise feature map. A low-light image enhancement network, SADiff, based on the sparse attention diffusion model, is constructed. In the inverse process, the denoising formula of the diffusion model is applied iteratively to denoise the purely noisy image, ultimately generating the enhanced image. By training the SADiff network, different types of low-light images can be adaptively processed. This invention's method, through effective data processing, noise modeling, and adaptive network optimization, can significantly improve the quality of low-light images, enhancing image detail and clarity. Attached Figure Description

[0076] Figure 1 This is a schematic diagram of the overall structure of the low-light image enhancement method based on the adaptive sparse attention diffusion model of the present invention;

[0077] Figure 2 This is a schematic diagram of the preprocessing module of the low-light image enhancement method based on the adaptive sparse attention diffusion model of the present invention;

[0078] Figure 3 This is a schematic diagram of the sparse attention enhancement network module in the low-light image enhancement method based on the adaptive sparse attention diffusion model of the present invention;

[0079] Figure 4 This is a schematic diagram of the structure of the low-light image enhancement method based on the adaptive sparse attention diffusion model in the noise consistency constraint module of the present invention;

[0080] Figure 5 This is a visualization comparison of the low-light image enhancement method based on the adaptive sparse attention diffusion model of this invention with other different low-light enhancement methods on the LOL dataset. Detailed Implementation

[0081] The present invention will now be described in detail with reference to specific embodiments.

[0082] Example 1

[0083] This invention proposes a weak light enhancement method based on an adaptive sparse attention diffusion model. The network structure diagram is shown below. Figure 1 As shown, normal light images corresponding to low-light images in the LOL dataset are collected. The normal light images are then forward-divided to obtain pure noise images. An improved noise prediction module is designed to calculate the noise feature map of the low-light image. A weak light enhancement network is designed, with the low-light image x as input. low Noise feature map ε θ Inverse denoising is performed using the diffusion model denoising formula through iterative denoising to obtain the final low-light enhancement network. This final low-light enhancement network is then trained by inputting the low-light image to be enhanced, resulting in the enhanced low-light image. The specific steps are as follows:

[0084] Step 1: Use a diffusion model to perform forward diffusion on the normal light image x0 corresponding to the low-light image in the LOL dataset to obtain the pure noise image x. T The specific process is as follows:

[0085] Select low-light images from the LOL dataset. low The corresponding normal light image x0 is input into the diffusion model, where it is forward diffused. Through this forward diffusion process, the normal light image x0 is transformed into a purely noisy image x. T The forward diffusion process is shown in the formula:

[0086]

[0087]

[0088]

[0089]

[0090] in ε and Ι follow a standard normal distribution N(0,1). The diffusion model is a parameterized Markov chain. Due to its reparameterization characteristics, given x0 and randomly sampled Gaussian noise ε, the pure noise image x is obtained according to formulas (3) and (4). T .

[0091] Step 2, as follows Figure 2 As shown, design a low-light image x low The preprocessing module (LLPM) consists of a curve fine-tuning module, an initial denoising and enhancement module, and a Canny edge detection module, which processes low-light images x... low Input to the preprocessing module to obtain the preprocessed image x llpm This is used to extract feature information from low-light images, which serves as a condition to guide noise prediction in the inverse process in step 3; the specific process is as follows:

[0092] Step 2.1: Convert the low-light image x low Input the Canny edge detection module to perform edge detection and obtain edge information x canny edge information x canny Blended into the original low-light image x low In the process, the edge information x' of the fused low-light image is obtained. canny The formula for this process is:

[0093]

[0094] Where canny() is the canny edge detector;

[0095] Step 2.2: The initial denoising and enhancement module consists of five convolutional layers forming a dense residual connection block, which denoises the low-light image x. low After input, each convolutional layer is used to extract features from the image, and the residuals between the features are accumulated to obtain the initial denoising and enhancement results;

[0096] Step 2.3: Convert the low-light image x low The number of channels is increased from 3 to 12 through a convolutional layer. Then, the channel is divided into four 3-channel adjustment factors through channel segmentation, which are input into the four iterations of curve fine-tuning. The initial denoising and enhancement result is used as the adjustment target and input into the curve fine-tuning block for iterative enhancement to obtain x. curve The curve fine-tuning module is shown in the following formula.

[0097]

[0098] In the formula, For the initial enhancement results, Here is the formula for curve fine-tuning, where i is the number of iterations. Since there are a total of 4 adjustments with 3 channels, the number of iterations i is 4.

[0099] Step 2.4: Adjust the curve result x curve The result of edge detection x' canny To perform a simple summation and avoid overexposure of the preprocessed result, the summation result needs to be multiplied by [a factor]. Obtain the preprocessed image x llpm The specific process is shown in the following formula.

[0100]

[0101] Step 3, as follows Figure 3 As shown, an improved noise prediction module is designed to enhance the quality of low-light images, thereby discarding noisy regions in low-light images and highlighting texture details in extremely dark regions. To better discard noisy regions in low-light images and highlight texture details in extremely dark regions, a sparse attention enhancement module replaces the convolutional layers in the existing noise prediction module. The overall network structure is based on the UET network, but differs from UET in that the original UET network structure performs a simple convolution operation after downsampling or upsampling, followed by direct downsampling or upsampling. The proposed network replaces the original convolution with a residual module containing a sparse attention mechanism, and replaces the skip connections in the existing noise prediction module with a noise consistency constraint module. This improves the rendering of the pure noise image x... T and the preprocessed image x llpm The noise feature map ε is obtained by inputting it into the improved noise prediction module. θ Specifically:

[0102] Step 3.1: Design an improved noise prediction module. Replace the convolutional layers in the existing noise prediction module with a sparse attention enhancement module, and replace the skip connections in the existing noise prediction module with a noise consistency constraint module. Then, process the preprocessed image... images with pure noise The input is fed into the improved noise prediction module through channel-level connections. After the connections, the image size remains unchanged, but the number of channels increases from 3 to 6. After a dimensionality upscaling operation, the input 6-channel tensor is expanded into a 64-channel tensor. The result after the dimensionality upscaling operation is then processed by a residual network for feature extraction. During the feature extraction process, the intermediate feature tensors obtained, while keeping the number of channels and size unchanged, are fed into the sparse attention enhancement module. This module combines the characteristics of local and global attention to capture different patterns and relationships in the input data.

[0103] Step 3.2: In the sparse attention enhancement module, to prevent attention collapse, the obtained intermediate feature tensor is... Group normalization is performed to normalize the input data of different channels, resulting in feature maps.

[0104] Step 3.3: Map the obtained features Perform channel transformation from the original bchw shape to the bhwc shape, then divide the feature maps of the transformed channels into n×n non-overlapping regions, such that each feature map contains m feature regions, where m is calculated using the formula:

[0105]

[0106] Where H and W are the length and width of the feature map, respectively, and n is the length and width of the m small windows. 2 It is the area of ​​the divided square windows;

[0107] Step 3.4: After dividing the window into smaller windows, as shown... Figure 3 As shown, this makes Become The query, key, and value tensors of m small windows are obtained through linear projection calculation. The formula is as follows:

[0108] Q = x m W q K = x m W k V = x m W v (9);

[0109] in The projection weights for Q, K, and V are respectively.

[0110] Construct a directed graph to determine the most relevant key-value pairs in each region, and calculate the average of Q and K over m small windows to obtain the region-level query and build values. The adjacency matrix between regions is obtained using formula (10).

[0111] A m =Q m (K m ) T (10);

[0112] Adjacency matrix A k To represent the semantic correlation between two regions, we retain the K most crucial pieces of information from each region, resulting in an index matrix I. m The process is as follows:

[0113] I m =topKIndex(A m )∈N n2×k (11);

[0114] Obtain the index matrix I m Then, a fine-grained region-by-region attention mechanism is applied. For each query tag in region i, all key-value pairs are processed, and I is collected. m The formulas for k and v are as follows:

[0115] K g =collection(K,I m ),V g =collection(V,I) m ), (12)

[0116] in The k and v tensors after the set are given, and the formula for calculating attention is as follows;

[0117]

[0118] Through nested operations, including dimensionality transformation, linear transformation, attention computation, and shape rearrangement, a weighted and mapped output tensor is generated, effectively modeling the multi-level relationships of the input data and ultimately obtaining feature information x. SAEM .

[0119] The constant factor is used to prevent weight concentration and attention collapse;

[0120] Step 3.5, as follows Figure 4 As shown, the Noise Consistency Constraint Module (NCM) constrains the noise predicted by the neural network to avoid feature loss during downsampling and ensure that the denoised image does not produce artifacts. The NCM suppresses and constrains the output of the sparse attention enhancement module by means of the following steps: The feature representation x... SAEM After feature enhancement via noise_block1, x SAEM The process involves sequentially performing convolution, batch normalization, and ReLU activation, followed by random channel-level dropout on the results. This process is then repeated. These operations help the model predict potential noise in the input data. By learning the weights between different feature transformations, the extracted feature information is further enhanced to obtain x. noise_block1 x through residual blocks noise_block1Residual connections and feature extraction are performed, and an attention mechanism is used to further enhance the correlation between features and improve the quality of feature representation. This module yields feature information x. noise_atten ;

[0121] feature information x noise_atten An adaptive filter is used for correction, correcting the previously obtained feature information to obtain x. curve_filt ;

[0122] Take the original input x SAEM x noise_block1 x noise_atten and x curve_filt The features are added together in a weighted manner to obtain the final modified feature block x. ncm , making x SAEM Through multiple feature transformations, residual connections, attention mechanisms, and adaptive filtering, noise in the input data is effectively processed, and features are extracted and corrected to produce more accurate and robust feature representations, which are then passed as more accurate inputs to the next layer's task. The formula is as follows:

[0123] x ncm =x SAEM +x noise_block1 +x noise_atten +x noise_filt (14);

[0124] Step 3.6, x ncm With x SAEM Simple summation, summing the result with respect to x SAEM The upsampling results are connected at the channel level. Each upsampling layer passes through an existing residual connection and a sparse attention enhancement module, and the noise feature map ε is obtained by passing through a ConvBlock output. θ .

[0125] Step 4: Design a low-light enhancement network incorporating a diffusion model to enhance the low-light image x. low Noise feature map ε θ The low-light enhancement network is used for enhancement. During the enhancement process, the denoising formula in the inverse denoising process of the diffusion model is used to denoise the purely noisy image x. T Denoising was performed, and the denoising formula was iterated 2000 times to obtain the final low-light enhancement network;

[0126] Based on the idea of ​​existing diffusion models, the designed weak light enhancement network based on the sparse attention diffusion model also includes a forward diffusion process and a reverse denoising process. The forward diffusion process is as shown in step 1.

[0127] The specific process is as follows:

[0128] Step 4.1: In the noise prediction stage, the loss function L2 is used for constraint, and the specific formula is as follows:

[0129]

[0130] Where ε is Gaussian noise that follows a normal distribution, ε θ It is a noise feature map predicted by the noise prediction module;

[0131] Step 4.2: Based on the principle of the inverse process of the existing diffusion model, and combined with the predicted noise, the isotropic standard Gaussian distribution is transformed into the target data distribution. This allows the input low-light image to be iteratively transformed into an enhanced normal-light image. During this process, the existing diffusion model denoising formula is used to denoise the pure noise image using x... T The denoising formula for the reverse process is shown in formula (16):

[0132]

[0133] The mean and variance of noise are derived from the formula for the normal distribution as follows:

[0134]

[0135]

[0136] Where β t =1-α t Since x0 is not available during the testing phase, the approximate value f of x0 is calculated based on formula (4) for the forward diffusion process. θ (x t The specific calculation process is shown in formula (18):

[0137]

[0138] The approximate value of x0 calculated by formula (17) is used to replace x0 in formula (16) to obtain formula (19). In order to apply it to low-light image enhancement, a condition x is added to the inverse process. llpm To guide the reverse denoising process, so as to achieve low-light image enhancement, formula (19) becomes formula (20). By iterating formula (20) until t=0, the enhanced image is obtained.

[0139]

[0140]

[0141] Step 4.3: In the reverse process, iterate formula (20) 2000 times to obtain the final enhanced network model.

[0142] Step 5: Using low-light images and their corresponding normal-light images from the LOL dataset as training samples, train the final low-light enhancement network for 2060 epochs to obtain the optimal enhancement network model after training; the specific process is as follows:

[0143] Step 5.1: Use the low-light images and their corresponding normal-light images from the LOL dataset as training samples. Train the network on a GPU with 12GB of VRAM. Use the existing ADAM optimizer as the network optimizer and set the initial learning rate to 1×10⁻⁶. -4 The scheduler strategy is a multi-step decay strategy with decay steps of 25 and 100 respectively, decaying by half each time, and the number of iterations is set to 1,000,000, with 200 iterations per iteration, for a total of 2067 epochs of training.

[0144] Step 5.2: Save the neural network parameters trained in Step 5.1, the number of epochs being trained, the number of iterations, the optimizer ADAM, and the scheduler to obtain the optimal augmented network model after training.

[0145] Step 6: Input the low-light image to be enhanced into the trained optimal enhancement network model to obtain the enhanced low-light image.

[0146] Example

[0147] The method of this invention has been compared with several state-of-the-art methods, including EnlightenGAN, EnlightenGAN+BM3D, LIME, RetinexNet, RetinexNet+BM3D, RUAS, Zero, and Zero+BM3d. Figure 5 As shown, a visualization comparison of different methods is performed on the LOL-real dataset. These images were obtained by testing the pre-trained models provided by the corresponding methods. `ours` represents the results obtained by the method of this invention, `Ground Truth` is the reference value, and `input` is the input low-light image. RetinexNet and RetinexNet+BM3D exhibit significant noise and color bias. RUAS has significant color bias and uneven wall color. Zero and Zero+BM3D, while still exhibiting significant noise, appear to be slightly better than RetinexNet. EnlightenGAN+EnglightenGAN shows color bias, noise, and uneven wall color. LIME has comfortable visual performance but lacks good recovery in dark areas. Compared to the ground truth, LIME achieves limited enhancement on the edges and textures of black clothing.

[0148] In summary, the method of this invention offers the most comfortable visual experience and the most accurate color information, while also effectively removing noise and preserving texture details in low-light images. Therefore, the method of this invention is superior to other methods in both subjective and objective evaluation.

[0149] This invention presents a low-light image enhancement method based on an adaptive sparse attention diffusion model (SADiff). Building upon the traditional diffusion model (DDPM), it introduces a pre-processed low-light image to guide the inverse denoising process, ultimately generating an enhanced normal-light image. Since the diffusion model has excellent noise removal capabilities, the preprocessing module LLPM can use curve fine-tuning to enhance the brightness of the low-light image without amplifying noise. Furthermore, Canny edge detection is combined to enhance edge information in the image, and the enhancement result is input into the diffusion process, allowing the model to restore an image with higher contrast and clearer edge details. An adaptive sparse attention enhancement network (SAEM) is used to improve the quality of the low-light image. This mechanism discards noisy areas and highlights texture details in extremely dark areas, resulting in a clearer and brighter enhanced image. Simultaneously, the proposed Noise Consistency Constraint (NCM) module corrects predicted noise information by predicting, modeling, and correcting noise in the input data, thereby improving the model's robustness, performance, and output quality, enabling the model to achieve a visually better enhancement effect.

Claims

1. A weak light enhancement method based on an adaptive sparse attention diffusion model, characterized in that, We collect normal light images corresponding to low-light images from the LOL dataset, forward diffuse the normal light images to obtain pure noise images, design an improved noise prediction module, calculate the noise feature map of the low-light images, and design a low-light enhancement network, taking the low-light images as input. Noise feature map Inverse denoising is performed, and the denoising iterative process is carried out using the diffusion model denoising formula to obtain the final low-light enhancement network. The final low-light enhancement network is trained by inputting the low-light image to be enhanced to obtain the enhanced low-light image. In the noise prediction stage, the loss function L2 is used for constraint, and the specific formula is as follows: (15)； in For Gaussian noise that follows a normal distribution, It is a noise feature map predicted by the noise prediction module; Based on the principle of the inverse process of existing diffusion models, and incorporating predicted noise, the isotropic standard Gaussian distribution is transformed into the target data distribution. This allows the input low-light image to be iteratively transformed into an enhanced normal-light image. During this process, existing diffusion model denoising formulas are used to denoise the purely noisy image. The denoising formula for the reverse process is shown in formula (16): (16)； The mean and variance of noise are derived from the formula for the normal distribution as follows: (17); in Because there was no testing phase Therefore, according to formula (4) for the forward diffusion process, the following can be calculated: approximation The specific calculation process is shown in formula (18): ，(18)； The result calculated using formula (17) The approximate value replacement formula (16) Formula (19) is obtained. In order to apply it to low-light image enhancement, a condition is added to the inverse process. To guide the reverse denoising process, so as to achieve low-light image enhancement, formula (19) becomes formula (20). By iterating formula (20) until t=0, the enhanced image is obtained. (19)； (20)； In the reverse process, formula (20) is iterated 2000 times to obtain the final enhanced network model.

2. The weak light enhancement method based on the adaptive sparse attention diffusion model according to claim 1, characterized in that, The specific steps are as follows: Step 1: Use a diffusion model to analyze the normal light images corresponding to the low-light images in the LOL dataset. Forward diffusion is performed to obtain a pure noise image. ; Step 2: Design low-light images The preprocessing module consists of a curve fine-tuning module, an initial denoising and enhancement module, and a Canny edge detection module, which processes low-light images. Input to the preprocessing module to obtain the preprocessed image. ; Step 3: Design an improved noise prediction module. Replace the convolutional layers in the existing noise prediction module with a sparse attention enhancement module, and replace the skip connections in the existing noise prediction module with a noise consistency constraint module, thus transforming the pure noise image... and preprocessed images The noise feature map is obtained by inputting it into the improved noise prediction module. ; Step 4: Design a low-light enhancement network incorporating a diffusion model to enhance low-light images. Noise feature map The image is input into a low-light enhancement network for enhancement. During the enhancement process, the denoising formula from the inverse denoising process of the diffusion model is used to denoise the purely noisy image. Denoising was performed, and the denoising formula was iterated 2000 times to obtain the final low-light enhancement network; Step 5: Use the low-light images and their corresponding normal-light images from the LOL dataset as training samples to train the final low-light enhancement network for 2060 epochs, and obtain the optimal enhancement network model after training. Step 6: Input the low-light image to be enhanced into the trained optimal enhancement network model to obtain the enhanced low-light image.

3. The weak light enhancement method based on the sparse attention diffusion model according to claim 2, characterized in that, Step 1 is as follows: Select low-light images from the LOL dataset The corresponding normal light image Input a diffusion model, and then perform forward diffusion within the model. This forward diffusion process will transform the normal light image... Turn into a picture of pure noise. The forward diffusion process is shown in the formula: (1)； (2)； (3)； (4)； in , and Follows a standard normal distribution The diffusion model is a parameterized Markov chain, and due to its reparameterization property, it is known that... and randomly sampled Gaussian noise Pure noise images are obtained according to formulas (3) and (4). .

4. The weak light enhancement method based on the sparse attention diffusion model according to claim 2, characterized in that, Step 2 is as follows: Step 2.1: Transfer the low-light image Input the Canny edge detection module to perform edge detection and obtain edge information. edge information Blended into the original low-light image In the process, edge information of the fused low-light image is obtained. The formula for this process is: (5); Where canny() is the canny edge detector; Step 2.2: The initial denoising and enhancement module consists of five convolutional layers forming a dense residual connection block, which denoises the low-light image. After input, each convolutional layer is used to extract features from the image, and the residuals between the features are accumulated to obtain the initial denoising and enhancement results; Step 2.3: Transfer the low-light image The number of channels is increased from 3 to 12 by a convolutional layer, and then the adjustment factor is divided into four 3-channel adjustment factors by a channel segmentation operation, which are then input into the four iterations of curve fine-tuning. The initial denoising enhancement result is used as the adjustment target and input into the curve fine-tuning block for iterative enhancement. The curve fine-tuning module is shown in the following formula. (6); In the formula, For the initial enhancement results, Here is the formula for curve fine-tuning, where i is the number of iterations. Since there are a total of 4 adjustments with 3 channels, the number of iterations i is 4. Step 2.4: Fine-tune the curve. Results of edge detection Perform a simple summation, then multiply the result by... To obtain the preprocessed image The specific process is shown in the following formula. (7)。 5. The weak light enhancement method based on the sparse attention diffusion model according to claim 2, characterized in that, Step 3 specifically involves: Step 3.1: Design an improved noise prediction module. Replace the convolutional layers in the existing noise prediction module with a sparse attention enhancement module, and replace the skip connections in the existing noise prediction module with a noise consistency constraint module. Then, process the preprocessed image... images with pure noise The input is fed into the improved noise prediction module through channel-level connections. After the connections, the image size remains unchanged, but the number of channels increases from 3 to 6. After a dimensionality increase operation, the input 6-channel tensor is expanded into a 64-channel tensor. The result of the dimensionality increase operation is used for feature extraction through a residual network. During the feature extraction process, the intermediate feature tensors obtained will enter the sparse attention enhancement module while keeping the number of channels and size unchanged. Step 3.2: In the sparse attention enhancement module, process the obtained intermediate feature tensor. Group normalization is performed to normalize the input data from different channels, resulting in feature maps. Step 3.3: Map the obtained features Perform channel transformation, changing the original bchw shape to a bhwc shape, and then divide the feature maps of the transformed channels into... There are m non-overlapping regions, such that each feature map contains m feature regions, where m is calculated using the formula: (8)； Where H and W are the length and width of the feature map, respectively, and n is the length and width of the m small windows. It is the area of ​​the divided square windows; Step 3.4: After dividing the window into smaller windows, it makes... Become The query, key, and value tensors of m small windows are obtained through linear projection calculation. The formula is as follows: (9)； in The projection weights for Q, K, and V are respectively. Construct a directed graph to determine the most relevant key-value pairs in each region, and calculate the average of Q and K over m small windows to obtain the region-level query and build values. The adjacency matrix between regions is obtained through formula (10). : (10)； Adjacency matrix This represents the semantic correlation between two regions. By retaining the K most crucial pieces of information from each region, an index matrix is ​​obtained. The process is as follows: (11); Obtain the index matrix Then, a fine-grained region-based attention mechanism is applied to each query tag in region i, processing all key-value pairs and collecting... The formulas for k and v are as follows: (12) in The k and v tensors after the set are given, and the formula for calculating attention is as follows; (13); Through nested operations, including dimensionality transformation, linear transformation, attention computation, and shape rearrangement, the final feature representation is obtained. ; The constant factor is used to prevent weight concentration and attention collapse; Step 3.5: Noise suppression and constraint are applied to the output of the sparse attention enhancement module using the noise consistency constraint module. The specific steps are as follows: Represent the features... After feature enhancement using noise_block1, we obtain ; through residual blocks Residual connections and feature extraction are performed, and an attention mechanism is used to further enhance the correlation between features and improve the quality of feature representation. Feature information is obtained through this module. ; Feature information An adaptive filter is used for correction, correcting the previously obtained feature information to obtain... ; Original input , , and The weighted sums are then used to obtain the final modified feature block. , making Through multiple feature transformations, residual connections, attention mechanisms, and adaptive filtering, noise in the input data is effectively processed, and features are extracted and corrected to produce more accurate and robust feature representations, which are then passed as more accurate inputs to the next layer's task. The formula is as follows: ，（14）； Step 3.6, and Simple summation, summing the result and the pair The upsampling results are connected at the channel level. Each upsampling layer passes through an existing residual connection and a sparse attention enhancement module, and the noise feature map is obtained by passing through a ConvBlock output. .

6. The weak light enhancement method based on the sparse attention diffusion model according to claim 5, characterized in that, The feature representation described in step 3.5 The specific process of feature enhancement after noise_block1 is as follows: The process involves sequentially performing convolution, batch normalization, and ReLU activation, followed by random channel-level dropout on the results. This process is then repeated. These operations help the model predict potential noise in the input data. By learning the weights between different feature transformations, the extracted feature information is further enhanced. .

7. The weak light enhancement method based on the sparse attention diffusion model according to claim 1, characterized in that, Step 5 is as follows: Step 5.1: Use the low-light images and their corresponding normal-light images from the LOL dataset as training samples. Train the network on a GPU with 12GB of VRAM. Use the existing ADAM optimizer as the network optimizer and set the initial learning rate to [missing value]. The scheduler strategy is a multi-step decay strategy with decay steps of 25 and 100 respectively, decaying by half each time, and the number of iterations is set to 1,000,000, with 200 iterations per iteration, for a total of 2067 epochs of training. Step 5.2: Save the neural network parameters trained in Step 5.1, the number of epochs being trained, the number of iterations, the optimizer ADAM, and the scheduler to obtain the optimal augmented network model after training.

Images