Signal-to-noise ratio guided and HVI color assisted three-branch dark light image enhancement method

Abstract

The invention provides a signal-to-noise ratio guided and HVI color assisted three-branch dark light image enhancement method. The method comprises the following steps: inputting a dark light image and calculating a signal-to-noise ratio map of the dark light image; a signal-to-noise ratio guided dark-light image enhancement main branch composed of an encoder, a long-distance middle branch, a short-distance middle branch and a decoder is designed to perform preliminary enhancement on an input dark-light image, the long-distance branch performs global modeling on the dark-light image by using a signal-to-noise ratio graph, and the short-distance branch performs local modeling on the dark-light image; the long-distance branch and the short-distance branch are dynamically fused through a signal-to-noise ratio adaptive fusion module; designing an HVI color auxiliary branch to extract color consistency features; and finally, fusing output results of the main branch and the auxiliary branch to obtain a final enhanced image. Compared with an existing mainstream dark light image enhancement method, the method provided by the invention further optimizes local detail textures, color consistency and adaptability to a complex degraded environment, and obtains better efficiency and performance balance at the same time.

Description

Technical Field

[0001] This invention relates to the fields of digital image processing and deep learning, and in particular to methods for enhancing low-light images. Background Technology

[0002] Low-light imaging is prevalent due to objective environmental conditions (such as insufficient lighting) and hardware limitations (such as limited exposure time). These images not only severely affect the visual perception and recognition capabilities of the human eye, but more importantly, they struggle to meet the demands of modern multimedia computing and various downstream visual tasks. The core objective of low-light image enhancement technology is to effectively recover the hidden details in dark areas of an image through algorithmic methods, thereby ensuring the accuracy and reliability of subsequent visual analysis tasks.

[0003] Existing low-light image enhancement methods mainly employ histogram equalization [1], Retinex-based methods [2], and learning-based methods. Histogram equalization and Retinex-based methods have low generalization ability and are difficult to efficiently solve the problem of local image degradation in complex scenes. With the breakthrough of deep learning technology, convolutional neural networks [3] and Transformer [4] architectures have gradually become the mainstream backbone networks for feature extraction in this field. Convolutional neural networks significantly improve the enhancement effect by efficiently extracting local features, but their local receptive fields are difficult to effectively model long-distance dependencies, resulting in the restoration effect of darker areas in the image, i.e., low signal-to-noise ratio areas, not meeting the actual needs. Transformer enhances the ability to model long-distance dependencies through self-attention mechanism, but its computational complexity is quadratic with the input length, inevitably bringing a high computational burden. The proposal of VMamaba [5] network architecture and HVI color space [6] has laid a solid theoretical foundation for solving the above problems.

[0004] References: 1.WANG S., ZHENG J., HU H.-M., et al. Naturalness PreservedEnhancement Algorithm for Non-uniform Illumination Images [J]. IEEETransactions on Image Processing, 2013, 22(9):3538-48. 2.PARK S., YU S., MOON B., et al. Low-light Image Enhancement UsingVariational Optimization-based Retinex Model[J]. IEEE Transactions onConsumer Electronics, 2017, 63(2):178-84. 3.Ma L., Ma T., Liu R., et al. Toward Fast, Flexible, and Robust Low-light Image Enhancement[C] / / IEEE / CVF Conference on Computer Vision andPattern Recognition (CVPR). 2022: 5637-5646. 4.Cai Y., Bian H., Lin J., et al. Retinexformer: One-stage Retinex-based Transformer for Low-light Image Enhancement[C] / / Proceedings of theIEEE / CVF International Conference on Computer Vision. 2023: 12504-12513. 5.Liu Y., Tian Y., Zhao Y., et al. Vmamba: Visual State Space Model[C] / / Proceedings of the 36th International Conference on Neural InformationProcessing Systems. 2024: 103031-103063. 6.Yan Q., Feng Y., Zhang C., et al. HVI: A New Color Space for Low-light Image Enhancement [C] / / IEEE / CVF Conference on Computer Vision andPattern Recognition (CVPR). 2025: 5678-5687. Summary of the Invention

[0005] This invention addresses the problems of long inference times and loss of local details and color artifacts in enhanced images caused by existing methods. It proposes a three-branch low-light image enhancement method guided by signal-to-noise ratio and assisted by HVI color, mainly comprising the following steps: Step S1: Input a low-light image and calculate its signal-to-noise ratio (SNR). Step S2: Design a signal-to-noise ratio-guided low-light image enhancement main branch to perform preliminary enhancement on the input low-light image; Step S3: Design an HVI color auxiliary branch to extract color consistency features; Step S4: Fuse the outputs of the main branch and the auxiliary branch to obtain the final enhanced image.

[0006] Compared with current mainstream low-light image enhancement methods, this invention designs a long-range branch to globally model low SNR regions in low-light images using a signal-to-noise ratio (SNR) map, and a short-range branch to locally model high SNR regions in low-light images. The long-range and short-range branches are dynamically fused using an adaptive SNR fusion module. An HVI color auxiliary branch is designed to extract color consistency features. The final enhanced image is obtained by fusing the outputs of the main and auxiliary branches. Compared with existing mainstream low-light image enhancement methods, the method provided by this invention further optimizes local detail texture, color consistency, and adaptability to complex degradation environments, while achieving a better balance between efficiency and performance. Attached Figure Description

[0007] Figure 1 This is a schematic diagram of the process of the present invention; Figure 2 This is a structural diagram of the signal-to-noise ratio sensing embedding module of the present invention; Figure 3 This is a structural diagram of the bidirectional cross-attention module for VMamba fusion according to the present invention; Figure 4 This is a comparison of the visual quality of the present invention on the LOLv1 dataset; Figure 5 This is a comparison of the visual quality of the present invention on the LOLv2-synthetic dataset; Figure 6 This is a comparison chart of the visual quality of the present invention on the SMID dataset. Detailed Implementation

[0008] like Figure 1The flowchart shown is a three-branch low-light image enhancement method guided by signal-to-noise ratio (SNR) and assisted by HVI color, according to the present invention. The method mainly includes the following steps: inputting a low-light image and calculating its SNR map; designing a long-distance branch to globally model the low SNR region of the low-light image using the SNR map; designing a short-distance branch to locally model the high SNR region of the low-light image; dynamically fusing the long-distance and short-distance branches through an adaptive SNR fusion module; designing an HVI color-assisted branch to extract color consistency features; and fusing the outputs of the main branch and the auxiliary branch to obtain the final enhanced image. The specific implementation details of each step are as follows: Step S1: Input a low-light image and calculate its signal-to-noise ratio (SNR) map. The specific method for calculating the SNR map is as follows: Given an input image ,calculate grayscale, that is Then calculate the signal-to-noise ratio map. The mathematical formula is expressed as follows: , Where DN represents a deblurring operation, and abs represents the absolute value. This represents the estimated noise map.

[0009] Step S2: Design a signal-to-noise ratio-guided low-light image enhancement main branch to perform preliminary enhancement on the input low-light image, as follows: Step S21: Apply an encoder network to encode features in the low-light image.

[0010] Step S22: Design a long-range branch to perform global modeling of the low-light image using the signal-to-noise ratio map. Specifically, the backbone of the long-range branch consists of a signal-to-noise ratio sensing embedding module (SEM), a 2D selective scanning mechanism (SS2D), layer normalization (LN), and a feedforward neural network (FFN), given the input features. Signal-to-noise ratio diagram The long-distance branch can be expressed mathematically as follows: , Among them, SEM is a signal-to-noise ratio sensing embedding module, which uses cross-attention operation, firstly... Adjusted through convolutional layers to and For features of the same dimension, after layer normalization (LN), the feature tensor is transformed by changing its size. and Adjusted to a format that better suits the attention of bulls: , in , d m=C / m is the dimension of each feature head, where C is the dimension of the input features, m is the number of attention heads, and n=1,2,3…,m. The SEM with only 1 attention head is shown below. Figure 2 As shown, Considered Input low-light image As The representations of Q, K, and V are as follows: , in The signal-to-noise ratio attention map is a learnable parameter matrix constructed through convolutional layers, and therefore its representation is as follows: , in This is a learnable parameter that acts as a scaling factor to adjust the attention score. It adaptively scales matrix multiplication. Through the above operations, an output feature with noisy information is obtained. Then, the original image is used to optimize the noisy output image, resulting in a relatively clean image. After that, continue to The process is refined and is described below: , SS2D is the 2D selective scanning mechanism in the VMamba model, while CSF is composed of the CBAM attention mechanism and the ShuffleAttention attention mechanism connected in series.

[0011] Step S23: Design a short-range branch consisting of basic feature extraction and local detail enhancement to locally model the low-light image. Specifically, in the basic feature extraction stage, two consecutive 3×3 convolutional layers combined with the ReLU activation function are used to extract the input features. Skip connections are used to preserve the original illumination distribution information, and a He initialization strategy (weights initialized to a Gaussian distribution with 0.1 variance) is adopted to ensure training stability. In the local detail enhancement stage, the intermediate features F obtained in the basic feature extraction stage are used to extract spatial details independently in each channel using a 3×3 depthwise separable convolution. These details are then further fused using a 1×1 point convolution, and a non-linear mapping is constructed using the GELU activation function. Finally, a 1×1 convolution is used to adjust the enhanced features back to their original dimensions to obtain the final output features of the short-range branch. .

[0012] Step S24: Design an adaptive signal-to-noise ratio (SNR) fusion module to dynamically fuse the outputs of long-range and short-range branches. Specifically, adjust the size of the SNR map to H / 4 × W / 4, normalize its value to the range of [0,1], and then normalize the SNR map... As a feature of weight fusion and Its mathematical expression is: , in To provide output features for the decoder to generate the final output image, the signal-to-noise ratio (SNR) map dynamically quantizes the noise intensity distribution in each region, driving the feature fusion network to construct an adaptive weight allocation mechanism. This achieves the synergistic integration of short-range correlation features and long-range dependency features, ultimately generating fused features. .

[0013] Step S25: Apply the decoder network to perform feature decoding on the fused result from S24 to obtain a preliminary enhanced image.

[0014] Step S3: Design an HVI color auxiliary branch to extract color consistency features, as follows: Step S31: Obtain the intensity map of the input low-light image. Specifically, the intensity map of the image is represented by the maximum value in the RGB channels. The mathematical formula for calculating the image intensity map is as follows: .

[0015] Step S32: After converting the dark light image to HSV space, perform HV transformation, specifically the HV transformation in reference [6].

[0016] Step S33: Apply a convolutional neural network to extract features from the outputs of S31 and S32 respectively.

[0017] Step S34: Design a bidirectional cross-attention module for Vmamba fusion to perform feature fusion on the intensity and chromaticity components output from S33. Specifically, as follows: Figure 3 As shown, the intensity map is first used as Q, and the HV features are used as K and V. Cross-channel attention weights are calculated through a cross-attention mechanism to obtain the HV attention map. Then, according to Retinex theory, the HV attention map is decomposed into reflectance and illuminance components. After processing with separable convolution and Tanh activation function, the two components are multiplied. To further enhance the representation ability of chromaticity features, a 2D selective scanning mechanism is added to effectively capture the long-range dependencies in the HV chromaticity components, making the image colors smoother and more fluid. Then, the HV features are used as Q, and the intensity map is used as K and V to complete the reverse propagation of luminance information to the chromaticity domain.

[0018] Step S35: Convert the output of S34 to the RGB color space via the HSV color space to obtain color consistency features. The specific conversion method is the perceptual inversion HVI transformation in reference [6].

[0019] Step S4: The outputs of the main branch and the auxiliary branch are fused to obtain the final enhanced image. The specific fusion method is element-wise addition.

[0020] Compared with current mainstream low-light image enhancement methods, this invention designs a long-range branch to globally model low-signal-noise ratio (SNR) regions in low-light images using a signal-to-noise ratio (SNR) map, and a short-range branch to locally model high-SNR regions in low-light images. The long-range and short-range branches are dynamically fused through an adaptive SNR fusion module. An HVI color auxiliary branch is designed to extract color consistency features. The final enhanced image is obtained by fusing the outputs of the main branch and the auxiliary branch. Figure 4 It can be seen that the method provided by the present invention effectively enhances low-light images and better maintains the consistency of image colors; by Figure 5 It can be seen that the method provided by this invention improves visual quality while optimizing local details; by Figure 6 As can be seen, the method provided by this invention can effectively reshape texture and color in complex degraded environments, such as the enhancement effect of runways and reflective glass. Table 1 shows that the method provided by this invention achieves the best performance in PSNR, SSIM, and LPIPS on the LOLv2-synthetic dataset. Table 2 shows that the method provided by this invention achieves the best PSNR on the LOLv1 dataset while maintaining a parameter count in the millions. Therefore, compared with existing mainstream low-light image enhancement methods, the method provided by this invention further optimizes local detail texture, color consistency, and adaptability to complex degraded environments, while achieving a better balance between efficiency and performance.

[0021] Table 1. Quantitative comparison of the present invention on the LOLv2-synthetic dataset.

[0022] Table 2 Comparison of computational complexity of the present invention on the LOLv1 dataset

Claims

1. A three-branch low-light image enhancement method guided by signal-to-noise ratio and assisted by HVI color, characterized in that, The method specifically includes the following steps: Step S1: Input a low-light image and calculate its signal-to-noise ratio (SNR). Step S2: Design a signal-to-noise ratio-guided low-light image enhancement main branch to perform preliminary enhancement on the input low-light image; Step S3: Design an HVI color auxiliary branch to extract color consistency features; Step S4: Fuse the outputs of the main branch and the auxiliary branch to obtain the final enhanced image.

2. The low-light image enhancement method according to claim 1, characterized in that... Step S2 includes the following steps: Step S21: Apply an encoder network to encode features in the low-light image; Step S22: Design a long-range branch to perform global modeling of the low-light image using the signal-to-noise ratio map. Specifically, the backbone of the long-range branch consists of a signal-to-noise ratio sensing embedding module (SEM), a 2D selective scanning mechanism (SS2D), layer normalization (LN), and a feedforward neural network (FFN), given the input features. Signal-to-noise ratio diagram The long-distance branch can be expressed mathematically as follows: ， Among them, SEM is a signal-to-noise ratio sensing embedding module, which uses cross-attention operation, firstly... Adjusted through convolutional layers to and For features of the same dimension, after layer normalization (LN), the feature tensor is transformed by changing its size. and Adjusted to a format that better suits the attention of bulls: ， in , d m =C / m is the dimension of each feature head, where C is the dimension of the input features, m is the number of attention heads, and n=1,2,3…,m. Considered Input low-light image As The representations of Q, K, and V are as follows: ， in The signal-to-noise ratio attention map is a learnable parameter matrix constructed through convolutional layers, and therefore its representation is as follows: ， in This is a learnable parameter that acts as a scaling factor to adjust the attention score. It adaptively scales matrix multiplication. Through the above operations, an output feature with noisy information is obtained. Then, the original image is used to optimize the noisy output image, resulting in a relatively clean image. After that, continue to The process is refined and is described below: ， SS2D is the 2D selective scanning mechanism in the VMamba model, and CSF is composed of the CBAM attention mechanism and the ShuffleAttention attention mechanism connected in series. Step S23: Design a short-range branch consisting of basic feature extraction and local detail enhancement to locally model the low-light image. Specifically, in the basic feature extraction stage, two consecutive 3×3 convolutional layers combined with the ReLU activation function are used to extract the input features. Skip connections are used to preserve the original illumination distribution information, and a He initialization strategy (weights initialized to a Gaussian distribution with 0.1 variance) is adopted to ensure training stability. In the local detail enhancement stage, the intermediate features F obtained in the basic feature extraction stage are used to extract spatial details independently in each channel using a 3×3 depthwise separable convolution. These details are then further fused using a 1×1 point convolution, and a non-linear mapping is constructed using the GELU activation function. Finally, a 1×1 convolution is used to adjust the enhanced features back to their original dimensions to obtain the final output features of the short-range branch. ; Step S24: Design an adaptive signal-to-noise ratio (SNR) fusion module to dynamically fuse the outputs of long-range and short-range branches. Specifically, adjust the size of the SNR map to H / 4 × W / 4, normalize its value to the range of [0,1], and then normalize the SNR map... As a feature of weight fusion and Its mathematical expression is: ， in To provide output features for the decoder to generate the final output image, the signal-to-noise ratio (SNR) map dynamically quantizes the noise intensity distribution in each region, driving the feature fusion network to construct an adaptive weight allocation mechanism. This achieves the synergistic integration of short-range correlation features and long-range dependency features, ultimately generating fused features. ; Step S25: Apply the decoder network to perform feature decoding on the fused result from S24 to obtain a preliminary enhanced image.

3. The low-light image enhancement method according to claim 1, characterized in that... Step S3 includes the following steps: Step S31: Obtain the intensity map of the input low-light image. Specifically, the intensity map of the image is represented by the maximum value in the RGB channels. The mathematical formula for calculating the image intensity map is as follows: ； Step S32: Convert the low-light image to HSV space and then perform HV transformation; Step S33: Apply a convolutional neural network to extract features from the outputs of S31 and S32 respectively; Step S34: Design a bidirectional cross-attention module for Vmamba fusion to perform feature fusion on the intensity and chromaticity components output from S33. Specifically, firstly, the intensity map is used as Q, and the HV features are used as K and V. Cross-channel attention weights are calculated through the cross-attention mechanism to obtain the attention map of HV. Then, according to Retinex theory, the attention map of HV is decomposed into reflectance and illuminance components. After processing with separable convolution and Tanh activation function, the two components are multiplied. To further enhance the representation ability of chromaticity features, a 2D selective scanning mechanism is added to effectively capture the long-range dependencies in the HV chromaticity components, making the image colors smoother and more fluid. Then, the HV features are used as Q, and the intensity map is used as K and V to complete the reverse propagation of luminance information to the chromaticity domain. Step S35: Convert the output of S34 to the RGB color space via the HSV color space to obtain the color consistency characteristics.

Images